Event listener not being added to conditionally rendered element on load even when initial state is true

I’m having a problem (NextJS 12) attaching an event listener to an element that is conditionally rendered on load even though the initial state is true.

Basically, I have some buttons on a page that I’m adding event listeners to with useRef, but one of the buttons is in a component that is conditionally rendered via a state value. I have the state value set to true on load so that the button is initially rendered and exposed.

const { status: authStatus, data: session } = useSession();
const [emailExists, setEmailExists] = React.useState(true);
const buttonRefs = React.useRef([]);

React.useEffect(() => {
  if (authStatus !== 'unauthenticated') return;

  // Save initial ref
  const buttons = buttonRefs.current;
  console.log(buttons);

  buttons.forEach(button => {
    if (button) {
      console.log(button);
      button.addEventListener('keydown', myCallback);
    }
  });

  setEmailExists(false);

  // Cleanup
  return () => {
    buttons.forEach(button => {
      if (button) {
        button.removeEventListener('keydown', myCallback);
      }
    });
  };
}, [authStatus]);

// Handle ref assignment
const setButtonRef = (ref) => {
  return (el) => {
    if (el && !ref.current.includes(el)) {
      ref.current.push(el);
    }
  };
};

The component I’m conditionally rendering:

{emailExists && (
  <Box>
    <Button ref={setButtonRef(buttonRefs)}>
      Email exists
    </Button>
  </Box>
)}

When I console log buttonRefs.current, it correctly shows all buttons in the array, but only the one inside the conditionally rendered component is not getting the event listener attached. I suspect it has something to do with how state is handled on load because if I change emailExists to just true, then everything works as it should. Any help would be appreciated.