Born in 1546 as the second son of a powerful Owari lord. After his father's death he served various masters before joining Toyotomi Hideyoshi, accumulating merit to become lord of Kakegawa Castle. His wife Chiyo is legendary for saving money to buy her husband a fine horse—the archetype of wifely support in Japanese culture. At Sekigahara in 1600, when Ieyasu was departing to fight the Uesugi, Kazutoyo offered to hand over his own castle of Kakegawa—read as a declaration of loyalty that Ieyasu greatly valued. As a reward he was granted Tosa at 200,000 koku and built Kochi Castle, founding the Tosa domain. He died in 1605 at age 60.

Sincere and steady. He valued loyalty and integrity over grand ambition, and had the ability to make the right decision at the right moment. He was also known for valuing the counsel of his wife Chiyo.

Kochi Castle retains its original donjon and is a famous historic site; statues of Kazutoyo and Chiyo stand at the foot of the castle. The tale of Chiyo's "wifely support" is deeply embedded in Japanese culture as a symbol of the ideal partnership.