the clean up function of the Ref callback is not executed

Here in my React the clean up function is not executed when re-rendering:

import React, { useRef, useState } from 'react';

export default function App() {
  const [x, setX] = useState(2);
  const d = useRef(null);

  return (
    <>
      <p ref={(node) => {
        console.log('start');
        return () => {
          console.log('finish');
        };
      }}>{x}</p>
      <button onClick={() => setX(x + 1)}>hello</button>
    </>
  );
}

i want my clean up function to be executed when the ref callback is updated hence here the callback is created every render then the clean up must be executed every render!