Hojo Soun (Ise Shinkuro) is credited with building Nirayama Castle around 1493, using it as his base to consolidate control of Izu Province. Having risen as a guest general of the Imagawa, Soun drove out Ashikaga Chachamaru to seize Izu, making Nirayama Castle the first stronghold of what would become a century of Later Hojo dominance.

In the 1590 Odawara Campaign, Toyotomi Hideyoshi sent a large army — under Oda Nobukatsu and others, said to number 40,000–50,000 — against Nirayama Castle. Its lord Hojo Ujinori held out for about 100 days with only some 3,000 men. After the main castle at Odawara surrendered, Ujinori opened the gates following persuasion by Tokugawa Ieyasu, with whom he had once been a fellow hostage of the Imagawa. In contrast to Yamanaka, which fell in half a day, Nirayama is remembered as a siege that showed the Later Hojo's resolve.

Long before Hojo Soun raised Nirayama Castle in the Sengoku period, this land of Nirayama was where Minamoto no Yoritomo spent his youth as an exile. Banished to Izu after the Heiji Rebellion, Yoritomo passed nearly twenty years at nearby Hirugashima, during which he was united with Masako, daughter of Hojo Tokimasa, before raising his army in 1180. The castle that later became the stronghold of the Later Hojo thus stands on ground layered with two histories — Yoritomo's years of endurance and the rise of the Hojo — in a region tied twice over to the birth of warrior rule.