Ohh, thank you, that would also fix my issue probably:

I currently fetch the timestamp when the invoice will expire once and then I let the browser count down the seconds.

But instead, I should let the browser calculate the difference between now and the expiration timestamp every second. Then there would be no clock drift, for example if the browser doesn't run the JS interval every second for whatever—probably good—reason.