Fabricjs Handling events when items positions are recalculated

i need to have two “buttons” on a canvas with a mousedown event for each button.

   // Add buttons for adding/removing neurons
    const addButton = new fabric.Text("+", {
      left: layerLeft - 40,
      top: y,
      fontSize: 20,
      fill: "green",
      selectable: false,
      hoverCursor: "pointer",
    });

    const removeButton = new fabric.Text("-", {
      left: layerLeft - 60,
      top: y + 40,
      fontSize: 20,
      fill: "red",
      selectable: false,
      hoverCursor: "pointer",
    });

    addButton.on("mousedown", () => this.addNodeToLayer(layerIndex));
    removeButton.on("mousedown", () => this.removeNodeFromLayer(layerIndex));
    this.canvas.add(addButton);
    this.canvas.add(removeButton);

    // Save buttons to the layer for future positioning
    layer.addButton = addButton;
    layer.removeButton = removeButton;

and after adding them to the canvas, it will work.
But then i have two functions to move the top and left coordinates of those buttons according to other items i have.

The position of the items will move correctly, but the mousedown event will not, it stays at the initial coordinates.

I tried to remove and reassing the event but nothing

   // Remove old event handlers before reassigning
      layer.addButton.off("mousedown");
      layer.removeButton.off("mousedown");

      // Assign new event handlers with updated layerIndex
      layer.addButton.on("mousedown", () => this.addNodeToLayer(layerIndex));
      layer.removeButton.on("mousedown", () => this.removeNodeFromLayer(layerIndex));
enter code here

Have you got any ideas h