Between 1561 and 1563, Matsudaira Motoyasu (Ieyasu) launched the final phase of his unification of Mikawa with attacks on the Tōjō-Kira. In autumn 1563, when Kira Yoshiaki rose again in concert with the Mikawa Ikkō-ikki uprising, Ieyasu dispatched Matsui Tadatsugu, Matsudaira Kamechiyo, and Honda Hirotaka to besiege the castle. Yoshiaki was forced to surrender and the castle fell, completing Ieyasu's subjugation of the last independent power in western Mikawa and establishing his unified rule of the province.

When Matsudaira Iemori, third lord of the Tōjō Matsudaira branch, died heirless in 1581 (Tenshō 9), Tokugawa Ieyasu installed his four-year-old fourth son Matsudaira Tadayoshi as Iemori's adopted heir, granting him the lordship of Tōjō Castle with a 10,000-koku fief — a measure that both preserved Iemori's holdings and secured the lands within the Tokugawa kin network. The following year (1582), in the territorial reorganization following the Takeda's collapse, Tadayoshi was transferred to Numazu Castle in Suruga with a 40,000-koku domain, and Tōjō Castle was formally abandoned. Tadayoshi later distinguished himself at the Battle of Sekigahara (1600), where he was gravely wounded pursuing Shimazu Yoshihiro, and eventually held Owari Kiyosu Castle with a 520,000-koku domain.