The temple is famous as the birthplace of Minamoto no Yoritomo: tradition holds that in late-Heian times, Yura Gozen—wife of Minamoto no Yoshitomo and daughter of Atsuta High Priest Fujiwara no Suenori—returned to her family's estate here and gave birth to Yoritomo in 1147 (though a Kyoto birth theory also exists). A "Birthplace of Minamoto no Yoritomo" stele and a "Yoritomo's First-Bath Pond" marker stand beside the main gate, drawing pilgrims as the founding site of the first Kamakura shogun.

The founding priest Zenko Shonin, who tutored Tokugawa Ieyasu (Takechiyo) during his years as a hostage, established this temple in 1529.

In 1590, Ōmandokoro, the mother of Toyotomi Hideyoshi, made a pilgrimage to Seiganji Temple — the same year Hideyoshi completed the unification of Japan. After Hideyoshi's death, the temple burned down in 1600 and was rebuilt by order of his son Toyotomi Hideyori, cementing the temple's lasting ties with the Toyotomi clan.