The clock could be reset backwards to a point long before when the timeout was set.
If we don't check for it, we could wait indefinitely.