Kasteel Biljoen, originally called Broekerhof, was a place were taxes were collected. It was first mentioned in 1076 as it was a large farm, which was donated to the chapter of St. Peter in Utrecht by Emperor Heinrich IV. Around the year of 1530 Duke Karel van Gelder built the first Noble House, partly with materials from the nearby Kasteel Overhagen.

In 1535, the Duke of Gelre sold the castle due to a lack of money. It had various owners until it came into the possession of Alexander van Spaen in 1661. He rebuilt the house into the present day castle with four equal corner towers, each crowned by a helmet-shaped roof. The family Van Spaen rebuilt the castle again in the the 18th century.

In 1795 French troops were quartered in the castle. After they left the castle was plundered and the interior largely destroyed. In 1849 baron Van Hardenbroek allowed Anna Pavlovna of Russia to use it temporarily. In 1872 it was inherited by the family Lüps, until 2008. Today (2021) Kasteel Biljoen is owned by an American family and the castle itself is closed off to public visits.