Yagyu Munenori served as sword instructor to three generations of Tokugawa shoguns — Ieyasu, Hidetada, and Iemitsu. He built this jinya in 1636, establishing the Yagyu domain. He authored the Heiho Kadensho, elevating swordsmanship to a political philosophy through his concept of the life-giving sword.

Yagyu Jubei Mitsuyoshi was born as the eldest son of Yagyu Munenori and raised at this jinya. He served Shogun Iemitsu but was suddenly dismissed in 1626, spending the next twelve years as a ronin traveling and training in swordsmanship. After his return, he was also involved in governing the Yagyu domain.

Yagyu Muneyoshi (Sekishusai) was the founder of the Yagyu Shinkage-ryu school of swordsmanship, born and raised in this Yagyu village. In 1563, he received full transmission of Shinkage-ryu from Kamiizumi Nobutsuna and founded the Yagyu branch. In his later years, he lived in seclusion in Yagyu village, training his grandson Jubei and other disciples. The jinya built by Munenori stands on the land nurtured by Sekishusai, from which the Yagyu swordsmanship tradition spread across Japan.