Inverness was one of the main strongholds of the Picts and was visited in 565 by St Columban of Iona, intending to convert the Pictish king Brude I. A church with a small monastery complex had already been founded by early Celtic monks on St Michael's Mount.

The castle above the city was built by King Malcolm III, eldest son of Duncan I. The reign of King Duncan I was marked by the conflict with his cousin Macbeth. All that is known is that Duncan I was killed by Macbeth in the battle near Elgin. In Shakespeare's account, however, Macbeth killed Duncan I in Inverness Castle and Malcolm III is said to have avenged his father's death by destroying the castle in order to build a new castle on its site.

The city of Inverness was attacked several times by residents of the Hebrides. In 1187, Donald Bane led the enemy islanders in the Battle of Torvean against the men from Inverness Castle led by Duncan Mackintosh. Both were killed in the battle. Further battles against the MacDonald clan are documented in 1340, 1411 and 1427. In 1554, the Munro clan defeated the Mackintosh clan at the Battle of Clachnaharry west of the city. The MacDonald clan and their allies stormed Inverness Castle in 1491.

In 1562, the Queen of Scotland Mary Stuart was denied access to the city by the English governor during the Huntly Rebellion. The Munro and Fraser clans then took the castle for her.

The current structure was built on the site of the original castle. The red sandstone structure is the work of a few 19th-century architects.