Event Listeners not being removed [duplicate]

This is my function which gets called when a delete button is clicked. The problem here is, inside the if and else part I am adding an event listener and also removing an event listener but the events only get added and not removed and when I click on the button again, it calls those function more than once and that count keeps increasing as the events dont get removed.

export const deleteButtonHandler = () => {
  const deleteButton = document.getElementById('deleteButton');
    const editButtons = document.querySelectorAll('.editIcon');
    const isDeleteMode = deleteButton.dataset.deleteMode === 'true';

    if (isDeleteMode) {
      editButtons.forEach((editButton) => {
        editButton.innerHTML = '<img src="some link to a picture" width="15" height="15">';
        const tagContainer = editButton.parentNode.parentNode;
        const tagText = tagContainer.querySelector('.tagText');

        editButton.removeEventListener('click', () => removeTag(tagText.textContent, tagContainer));
        editButton.addEventListener('click', () => handleTagEdit(tagText));
      });

      deleteButton.textContent = 'Delete Tag';
      deleteButton.dataset.deleteMode = 'false'

    } else {
      editButtons.forEach((editButton) => {
        const editIcon = document.createElement('img');
        editIcon.src = "some link to a picture";
        editIcon.width = "15";
        editIcon.height = "15";
        
        editButton.innerHTML = '';
        editButton.appendChild(editIcon);
        
        const tagContainer = editButton.parentNode.parentNode;
        const tagText = tagContainer.querySelector('.tagText');
      
        editButton.removeEventListener('click', () => handleTagEdit(tagText));
        editButton.addEventListener('click', () => removeTag(tagText.textContent, tagContainer));
      });

      deleteButton.textContent = 'Cancel';
      deleteButton.dataset.deleteMode = 'true';
    }
  };

and the if-else part works perfectly I checked it, with each click it alternates with if and else block of code.