Mesh follow cursor breaks on scroll

I’m struggling with a problem and still can’t figure out what’s going on and why.

Code here:

export const Example = () => {
    const ref = useRef();

    useFrame((state) => {
        const targetX = (state.mouse.x * state.viewport.width) / 2;
        const targetY = (state.mouse.y * state.viewport.height) / 2;

        ref.current.position.x +=
            (targetX - ref.current.position.x) * 0.2;
        ref.current.position.y +=
            (targetY - ref.current.position.y) * 0.2;
    });

    return (
        <mesh ref={ref}>
            <sphereGeometry />
            <meshNormalMaterial />
        </mesh>
    );
};

Mesh is placed inside a group. I’m trying to make it follow my cursor but if I’m “scrolling” and moving camera on y axis it breaks right in the end of state.viewport.height (bottom of visible page, idk how to describe it). Then it jumps back to the top.