Said to be a son of Saito Dosan, with his mother a sister of Inaba Ittetsu. He was the third lord of Kajita Castle and served as chief retainer to Oda Nobutaka. In 1582 he is said to have built Moriyama Castle (also called Ushigahana Fort) on Mt. Atago to counter suspicious movements by Mori Nagayoshi, lord of Kanayama Castle. After the Honnoji Incident in June of that year, he seized control of Gifu Castle as its deputy castellan, issuing protective edicts to temples and shrines, and governed Mino during the chaotic aftermath while maintaining a neutral stance between Oda Nobukatsu and the Akechi faction.

A commander who displayed careful political judgment by maintaining a neutral stance amid the internal strife of the Oda house, and who showed prudence in fortifying against the pressure of his powerful neighbor, Mori Nagayoshi, by building a new castle.

Toshitaka's brief seizure of Gifu Castle amid the chaos immediately following the Honnoji Incident is recorded as an episode symbolizing the power vacuum in Mino at the time. Some remains of Moriyama Castle survive today on Mt. Atago in Minokamo City, Gifu Prefecture.