Due to the rural exodus, San Felices de los Gallegos now only has around 400 inhabitants. After the reconquest (reconquista) from the Moors and the repopulation (repoblación) of the almost deserted mountain region in the 11th and 12th centuries, the area was disputed for centuries between the Kingdom of Portugal and the Kingdom of León (later the Kingdom of Castile).

The Castillo de San Felices de los Gallegos is a remnant of the disputes between the kingdoms over control of the area. It was built in the 13th century by the Portuguese king Don Dinis, who also erected the town's defence walls. However, it only took on its final form in the 15th century.In 1374, Beatriz of Portugal and Leonor, widow and daughter of Sancho of Castile respectively, settled in the castle of San Felices with their entire court. The proximity of San Felices to the border enabled the princesses to stay in contact with both sides of their family.