The castle was built about 1500 by the 9th Earl of Desmond. It was used as a Customs House until 1641 when it was converted into a naval prison, following the construction of a new Customs House. The prisoners kept in the Castle were in the main French and Spanish, and the building became known locally as the "French Prison" as a result.

In the 1990s it was restored and hosts today the International Museum of Wine which details the links between Ireland and the wine trade.