Go Back Functionality in Remix and React

I wanted make a reusable back functionality for my app. I’m using Remix.
I wanted to achieve that if you click the “Back” button, it should redirect back to the previous page. However there is a fallBackRoute provided if there is no way to go back.
The problem right now is that, sometimes, when you click “Back”, it redirects you to the page outside of your website. How should we best fix that?

BackButton.tsx

import { useLocation, useNavigate } from "@remix-run/react";
import { ArrowLeft } from "lucide-react";
import { Button } from "~/components/ui/button";

export default function BackButton({ fallbackRoute}: { fallbackRoute: string }) {
    const navigate = useNavigate();
    const location = useLocation();

    return (
        <Button
            type="button"
            onClick={() => {
                if (location.key !== "default") {
                    navigate(-1);
                } else {
                    navigate(fallbackRoute);
                }
            }}
        >
            <ArrowLeft className="mr-2" /> Back
        </Button>
    );
}

Sample component

<BackButton fallbackRoute={`/orders`} />