There was already a fortified settlement in Beja in Celtic-Roman times, which was renamed Pax Julia after the peace treaty between Julius Caesar and the Lusitanians in 48 BC; from then on it belonged to the Roman province of Lusitania. The town remained an important economic and strategic centre under the Suebi, Visigoths and Moors. In 1159, King Alfonso I of Portugal reconquered the town from the hands of the Muslims, but abandoned it just a few months later. In 1191, the Almohad sultan and general Almansor conquered the area. It only reverted back to the Christians in the 1230s. Kings Alfonso III (+ 1279) and especially Dinis I (+ 1325) restored and enlarged the castle.

The castle remained inhabited until the 17th century, when the Jesuits established a college here. The two-storey keep (Torre de Menagem), which is around 40 metres high, stands at the highest point of the complex.