Graft - then still called Greft - is first mentioned around the year 1100. At that time it was located much more to the west, possibly along the Vuile Graft, a small lake in the neighbourhood. Later the village was built on its present location. Originally the inhabitants lived from arable and livestock farming. Later the women worked mostly on small farms and the men on herring boats. On the town hall of Graft both means of livelihood can still be found: on the street side the coat of arms of Graft with the cow and on the south side a herring boat.

In the 17th century Graft was a prosperous village. This is easily visible from the old town hall, built in 1613: a real eye catcher with three stepped gables built in Dutch Renaissance style. It was not only used as a meeting room for the village council, but also served to give the village more prestige. Although the name of the master builder is not known, because of its style it is thought it must have been a pupil of the famous Dutch architect Hendrick de Keyser.

At the end of the 19th century the building was in poor condition; around 1900 the top facade on the street side had disappeared. In 1909/1910 it was extensively restored, followed by a major facelift in 1982/1983. It now serves as an exhibition space (annex to the museum In 't Houten Huis in De Rijp), wedding location and meeting room.