How can i control error status in React router loader?

I am using react router dom 6.22.3

I have created browser router like this

{
    path: "/",
    element: <MainLayout />,
    errorElement: <Error />,
    children: [
      {
        path: "/",
        element: <Dashboard />,
        loader: DashbaordLoader,
        errorElement: <Error />,
      },
    ]
}

Inside the Dashboard Component I am fetching data.

export const DashbaordLoader = async () => {
  try {
    const data = await fetcher("http://localhost:3000/users", "GET");
    return data;
  } catch (err) {
    throw err;
  }
};

The fetcher is just the javascript fetch function

export const fetcher = async (url: string, method: string) => {
  try {
    const response = await fetch(url, {
      method, 
      headers: {
        "Content-Type": "application/json",
        Authorization: "",
      },
    });
    return response.json()
  } catch (err) {
    throw err;
  }
};

As you can see there is no Authorization token so the server is responding like this

With status code 401

In the body {error: 'Need to login'}

But the thing is in the dashboardLoader. even though the data is with statusCode 401

It doesn’t goes into the catch block. How can i make it show the error element when the response is 401

Here is the error response in postman