In the 13C, this building was the residence of the bishops of Moray. It remained the residence of the prelates after the transfer of the seat to Elgin. Among the different constructions, the so-called Davie's Tower is a six-storey keep with walls almost 2.7m thick.