When Masako became pregnant, Yoritomo himself traveled to Gandenji — a temple tucked away in the hills southeast of Kamakura — to pray for her safe delivery (Azuma Kagami, Juei 1). Rather than relying on official shogunal prayer sites such as Tsurugaoka Hachimangu or Mishima Taisha, he deliberately chose Gandenji as a Kannon pilgrimage site, said to be because of the Eleven-faced Kannon's reputation for protecting women. Within Masako's turbulent life, the record that her husband prayed at this old mountain temple before the birth of their heir Manju (Yoriie) survives as an intimate trace of the private devotion of the still-newlywed couple.

In 1182 (Juei 1), Yoritomo visited Gandenji to pray for the safe delivery of his pregnant wife Hojo Masako. Recorded in the Azuma Kagami, this is one of the few attested visits by Yoritomo to Gandenji, made on the eve of the birth of his heir Manju (the future second shogun Yoriie). The temple was already widely known as a Kannon pilgrimage site; Yoritomo is said to have traveled south from Kamakura up the Tagoe River into the Hisagi hills and stood before the Eleven-faced Kannon. The visit preserves a glimpse of the young Yoritomo as a private believer praying for the birth of his heir, in the year before the Kamakura shogunate was formally founded.

In 1209 (Kenpo 6), the third shogun Minamoto no Sanetomo made consecutive pilgrimages to Jimmuji and Gandenji (Azuma Kagami). His visit to the temple where his father Yoritomo had prayed for Masako's safe delivery is an act that reflects the young warrior-poet shogun's devotion to Buddhism. He would be assassinated by his nephew Kugyo at Tsurugaoka Hachimangu just ten years later, in 1219. The delicate sensibility recorded in his poetry collection Kinkai Wakashu may well have been deepened in the stillness of mountain temples like Gandenji.