Caught by off guard by Xuan Hao's reaction to him wanting to retreat from the battle, the injured Celestial Immortal Realm cultivator could not stop Xuan Hao from seizing the advantage to push him back slightly.
Although it wasn't that major of an advantage to onlookers, to the already injured Celestial Immortal Realm cultivator, this sudden shift was of critical importance, as the equal stalemate he had relied on to fight back against Xuan Hao without using too much of his qi so far, had also allowed him to preserve much of his remaining strength and focus it on recovering his injuries while defending against Xuan Hao's attacks.
However, now that this stalemate had been broken by Xuan Hao, he could no longer dedicate most of his strength to recovering his injuries and instead had to focus most of that strength towards fighting Xuan Hao.