Kobo Daishi founded the temple in 822, and it was later designated a prayer temple by Emperor Goshirakawa to hold memorial services for Emperor Sutoku.

Emperor Sutoku was exiled to Sanuki Province after his defeat in the Hogen Rebellion of 1156 and died there in 1164 in profound despair. Emperor Goshirakawa (1127–1192) designated Shiromine-ji as an imperially vowed temple (chokuganjo) to pray for the repose of Sutoku's soul. The imperial mausoleum (Shirominegoryo) was established adjacent to the temple precinct, and from this point onward Shiromine-ji became a sacred site closely connected to the imperial house. Among Goshirakawa's many temple dedications during his long reign, this particular vow carried special weight as a rite of reconciliation for the brother whose fate was bound to his own by the tragedy of the Hogen Rebellion.