my panning handler resets the offset state after the second time

Im trying to make a panning handler for my canvas element using a state for the offset and a state to hold the mouse start position. The first pan works as expected and the offset is set to the correct values. On the secont pan when clicked it worked as expected but then when i move it it resets the offset to 0 asif its the first pan. Im very new to react and will appreciate any help.

function handlePan(event) {
        switch (event.type) {
            case "mouseleave":
                setIsPanning(false);
                break;

            case "mousedown":
                setIsPanning(true);
                setPanStartX(event.clientX);
                setPanStartY(event.clientY);
                break;

            case "mouseup":
                setIsPanning(false);
                console.log("UP");
                break;

            case "mousemove":
                if (isPanning) {
                    // Calculate the panning offset based on the mouse position and the pan start position
                    const offsetX = event.clientX - panStartX;
                    const offsetY = event.clientY - panStartY;
                    setPanOffsetX(offsetX);
                    setPanOffsetY(offsetY);
                    console.log("PANNING");
                }
                break;

            default:
                break;
        }
    }

I tried making a second state to store values to be substracted with the offset as the transform of the canvas which i thought would fix it but it made the canvas’s offset stack when moved consecutively.