A cousin of Yamana Munetoyo, Toyokazu inherited the headship of Inaba in 1561 when Munetoyo died young at 18. His chief retainer Takeda Takanobu turned against him; Toyokazu's attack on Tottori Castle failed in 1563, and that winter Takeda's forces drove him from Fuse Tenjinyama Castle to Shikano Castle. In 1564 Takeda's forces pressed as far as Shikano's foot, after which Toyokazu vanishes from the record. His younger brother Yamana Toyokuni later became lord of Tottori Castle and surrendered to Toyotomi Hideyoshi.

A shugo daimyo who inherited leadership of Inaba while still young, only to be overwhelmed by his own retainer's rebellion and driven from his castle.

His expulsion from Fuse Tenjinyama Castle symbolizes the process by which the authority of the Yamana shugo house was eroded by the gekokujo of its own retainers.