Minamoto no Noriyori, half-brother of Yoritomo and former Mikawa-no-kami, is said to have maintained a villa here at Segasaki where he enshrined a Yakushi Nyorai as his personal devotional image. In 1193, after the Soga brothers' revenge incident drew him under Yoritomo's suspicion, he was confined at Shuzen-ji in Izu and killed in an attack led by Kajiwara Kagetoki. The temple is traditionally said to have been founded on the grounds of his former villa to console his spirit. A memorial tablet bearing his Buddhist name Taneiji-den Dogo Daizenmon, dated 24 August Kenkyu 4 (1193), is preserved within the precincts, and the Mikawa-ki memorial service is observed there every year on the anniversary of his death — keeping alive the memory of one of the early Kamakura period's great Genji tragedies.

Yakuoji stands on the former villa site of Minamoto no Noriyori, Yoritomo's half-brother, who was confined and killed in 1193 after Yoritomo grew suspicious following the Soga brothers' revenge incident. The temple was founded to honor Noriyori's memory, enshrining his personal image of Yakushi Nyorai, and the annual "Mikawa-ki" memorial is still observed on the anniversary of his death.

In the 8th month of 1193 (Kenkyu 4), false rumors of Yoritomo's death during the Soga brothers' revenge incident at Mount Fuji reached Kamakura. Seeking to console Yoritomo's wife Masako, Minamoto no Noriyori is said to have remarked, 'So long as I, Noriyori, stand by you, the realm is secure' — words Yoritomo, upon his return, took as evidence of treasonous ambition. Noriyori was confined at Shuzen-ji in Izu (one tradition holds it was this very Segasaki villa), and on the 17th of that month Kajiwara Kagetoki was dispatched at Yoritomo's order to attack him. The Azuma Kagami and the Horyaku-kanki record Kagetoki as the leader of the assault, and Noriyori is said to have taken his own life on the 24th. Once Kagetoki had fought alongside the Noriyori-Yoshitsune brothers at Ichi-no-Tani, Yashima, and Dan-no-Ura to destroy the Heike — yet, backed by Yoritomo's trust and serving as Samurai-dokoro shoshi (deputy commissioner), he became the agent who pursued first Yoshitsune and then Noriyori to their deaths, cutting down the Genji clan from within. The temple is said to have been founded on the grounds of Noriyori's villa, enshrining his personal image of Yakushi Nyorai, so that the site itself testifies to the tragedy of the Genji lineage extinguished by Kagetoki's hand. Kagetoki himself, only six years after Noriyori's death, was driven from Kamakura by a joint indictment signed by 66 gokenin in 1199 (Shoji 1), and was killed at Kitsunezaki in Suruga in 1200 — bringing the Kajiwara clan, too, to extinction.