Imagawa Yoshimoto wasn't stupid. In the early Warring States Era of Japan, he controlled the Three Kingdoms region. The Imagawa family, able to muster tens of thousands of troops and launch expeditions, was by no means weak. If any daimyo of the same era really went head-to-head with him, no one could guarantee they'd withstand Imagawa Yoshimoto's full-force assault. Yet, in the end, Imagawa Yoshimoto just died like that.
He probably died from arrogance. He never expected that Oda Nobunaga would dare take the initiative to attack, would actually scheme against him, and so he was completely unprepared.
And Oda Nobunaga didn't win by luck, either. He had prepared in advance, plotted for a long time, and on top of that, he was stubborn enough, willful enough, desperate enough, and with a little bit of luck, this is how he created the "miracle of the Battle of Okehazama."