Apply constraints of Pan Tilt Zoom to Camera without the stream pausing for a little while

So i am in a bit of a pickle I am using WebRTC to control the PTZ functions of acamera using buttons for movement and zooming.

const constraints = {
    video: {
      pan: true,
      tilt: true,
      zoom: true
    }
  };

const stream = await navigator.mediaDevices.getUserMedia(constraints);
const [videoTrack] = stream.getVideoTracks();
const capabilities = videoTrack.getCapabilities();

After setting the current state where the camera should be I use

const constraints = { advanced: [{ [command]: adjustedValue }] };
await track.applyConstraints(constraints);

the problem is that this pauses my stream as the camera moves, in around 300-600ms the stream runs again in case of a longer movements its restored while the camera is still moving.

I kinda dont want to use OnVif or any HTTP based solution for this. Is there any way for PTZ controls in browser using the WebRTC but without pausing the stream