Pointer lock doesn’t work on Safari during fullscreen

I’m developing a fullscreen game, where I need to programmatically enable and disable the pointer lock to let the user click on UI elements without exiting fullscreen. This seems to work fine on Chrome and Firefox, but fails in Safari.

I managed to reproduce the problem using a minimal plain HTML example:

const elem = document.getElementById("elem");
elem.addEventListener("click", (e) => {
  if (!document.fullscreenElement)
    elem.requestFullscreen();
  else
    elem.requestPointerLock();
});

document.addEventListener("pointerlockchange", (e) => {
  if (document.pointerLockElement)
    elem.innerText = "Pointer lock is now active";
  else
    elem.innerText = "Pointer lock is now inactive";
});
document.addEventListener("pointerlockerror", (e) => {
  elem.innerText = "Pointer lock has failed";
});

<div id="elem" style="width: 100px; height: 100px; background-color: red;">
  This element will go fullscreen
</div>

The first click on the element should activate the fullscreen mode, the second one should lock the pointer. On Chrome and Firefox this works properly, and “Pointer lock is now active” is displayed. On Safari, “Pointer lock has failed” is displayed instead, and neither the event or the JS console show any reason why.

However, if you modify the code to first request the pointer lock, and only then switch to fullscreen mode, everything seems to work fine in all browsers:

elem.addEventListener("click", (e) => {
  elem.requestPointerLock();
});
document.addEventListener("keydown", (e) => {
  elem.requestFullscreen();
});
// click the element to lock the cursor first, then press space to enter fullscreen

But this doesn’t achieve what I’m trying to do for my use case, as if you then release the pointer lock, it can’t be reacquired without exiting fullscreen.

I could not find any reference to what could be causing this behavior in the docs.