Simon Bolivar, liberator of large parts of South America, rode throughout the mountains of Colombia while defeating the Spanish Conquistadores hundreds of years ago. You too, can travel throughout large parts of Colombia on a horseback riding tour. Horseriding is possible throughout almost the entire country, though some areas are better known for it than others. Broadly speaking, horseback riding is a possibility in any of the Andean highland regions that are frequented by tourists.
The most famous region for horseriding in Colombia is the Llanos, but sadly the area is not particulalrly safe for travel. You're better off heading to the countryside surrounding Popayan which has some very scenic horsebacking riding tours available. The Circuito Ancestral is an historic 15km trail through beautiful countryside and small villages east of Popayan. Further east, the ancient stone sculptures of San Agustin are best visited by horse, and the countryside offers a spectacular backdrop for horseback riding tours.
If you're a fan of horses, make an effort to go to some of Colombia's local festivals and ferias. Each town has it's own festival, and horseback riding processions (Cabalgatas) are commonly a major feature of the celebrations. Most famous is the Cali Festival (Feria de Cali) from 25 December to 31 December which used to be the best option for such horse processions, but sadly the horses were so badly treated by the drunken locals that the Cabalgatas were banned as of 2007. An alternative are the horse shows offered by certain large out of town rumba-style-restaurants, that are complete with dancing horses (they actually dance to local music).
Travel note - it's traditional among Colombians to drink a few shots of Aguardiente whilst horseback riding.