Multiplayer boardgame with timers

I am creating a multiplayer boardgame (similar to chess.com but for another game) where the users can create a game with a certain game time and then another player can join that game and play.

The flow right now is sort of like this:

1. On load I fetch all game data including a move list which includes time left in milliseconds for player1 and player2 respectively. If no move was made I pick the game time since it is the start of the game.

2. On client side I start a timer which ticks down time for the player who is starting the game.

3. When the player makes a move I send this to the server which calculates the elapsed time for the move like this:

   let now = Date.now();
   let newRemainingTime = game.gameTime;
   if(game.moveList.length > 0){                
       const elapsedTime = now - game.lastMoveTimestamp;
       const playerRemainingTime = game.moveList.length > 0 ? game.moveList[game.moveList.length - 1].timeLeft[player] : game.gameTime;
       newRemainingTime = Math.max(0, playerRemainingTime - elapsedTime + game.timeIncrement);
   } 
   

4. Then server (socketio) emits to both players that the game state is update and I update front end timers with the values I get from the new gamestate.

Under normal circumstances this works fine. The timers update in sync and server handles the time between moves. However I am experiences some issues with handling of connection issues, e.g.:

• If a player reloads the page during his turn the front end timers will show he has time left as per the latest move (instead of latest move minus his thinking time minus the time he has been away from the page).

• There might be cases where player1 leaves during opponents turn so the opponents time should tick down, then opponent makes a move during player1 abscence and then player1 time should tick down. When player1 returns the timings for both players are off.

Quesitons

1. How should I handle timers for the users if something happens between moves without using too much performance on the server?

2. Are there any other considerations that I need to account for with the timers?