React video restarts on every re-render in functional components with additional return values

This one stumped me for a bit, but I have a wrapper around an html tag which handles the reference to the video element and provides some functions (play/pause, skip forward/back, etc). I wanted to create a hook that could send back those functions and the way to actually render the wrapped video component:

function useVideo() {
  const videoRef = useRef(null);

  function playPause() {
    if (videoRef.current.paused) {
      videoRef.current.play();
    } else {
      videoRef.current.pause();
    }
  }

  function Video() {
    return <video ref={ videoRef } ...etc />
  }

  return { playPause, Video };
}

And then use that hook like so:

function App() {
  const { playPause, Video } = useVideo();

  ...

  return <Video key="key" />;
}

But every time I did anything that would cause App to re-render, the html video would restart. I believe the issue is that the Video function changes every time useVideo is run, so even though I have a key on the component, react treats it as a different component.