Yoritomo's eldest son and second shogun, Yoriie, was confined at Shuzenji by the Hojo clan in 1203 and assassinated the same year. The shogunate Yoritomo had built upon his marriage to Masako thus lost its direct heir at this site, and real power passed into Hojo hands.

Shuzenji is where Minamoto no Yoriie, the second shogun of the Kamakura shogunate, was confined after being stripped of his office. Following the Hiki Yoshikazu Rebellion of 1203, Yoriie was sent to Shuzenji in Izu and is said to have been assassinated by the Hojo side the next year, in 1204. The temple symbolizes the tragedy of the Kamakura shogunal house.