In NVDA, Button receives focus upon pressing Enter/space in the inner icon when aria-label/aria-labelled is used

Steps to Reproduce the Issue:

Open the link: https://nvda-test.w3spaces.com/button-accessibility-label.html
Two buttons: This page contains two buttons, each with a focusable icon inside.
ARIA Attributes: The first button has an aria-label attribute or an aria-labelledby attribute, while the second button does not have either.
NVDA Behavior: When using a screen reader like NVDA, focusing the icon within the first button (the one with aria-label or aria-labelledby) and pressing Enter or Space will cause the focus to return back to the button itself. This issue doesn’t occur with the second button (without ARIA attributes).
Project Use Case:

In my project, I have a button with a close icon inside. Ideally, focusing the close icon and pressing Enter/Space should remove the button from the page. This works as expected without NVDA, but the issue mentioned above arises when NVDA is enabled. While removing the aria-labelledby attribute from the button fixes this issue, but it’s necessary for other accessibility concerns and hence cannot be removed.

I want to prevent the focus from returning to the button when pressing Enter/Space on the icon inside the button, while still maintaining the aria-label or aria-labelledby attribute on the button for accessibility.