ReactJS – Going back in nested routes with Reacter Router Dom V6

My initial page is

http://localhost:3000/home

With a navigation bar I go to a list of products in:

http://localhost:3000/products/all/

When I click a product in the list I go to:

 http://localhost:3000/products/banana/

Now if I click Back on the browser or if I make a button with the new useNavigate and try to do navigate(-1) I go to /home in both examples.

How can I make it so I go to /products/all/ with useNavigate and hopefully I can do change the browser interaction too?