How to sort the dates from different objects with multiples dates inside of an array of object

I have this array of objects that I gather from an API:

  const schedules = [
    {
        "_id": "6436b48b875967d0bea245b4",
        "service": "64246dc9a2d61593d103c749",
        "scheduledDates": [
            {
                "date": "2023-04-17T18:00:00.000Z",
                "user": "643701f7e5f61e6760f4d1f3"
            },
            {
                "date": "2023-04-12T18:00:00.000Z",
                "user": "643701f7e5f61e6760f4d1f3"
            }
        ]
    },
    {
        "_id": "6436b48b875967d0bea245b5",
        "service": "64246dc9a2d61593d103c749",
        "scheduledDates": [
            {
                "date": "2023-04-19T10:30:00.000Z",
                "user": "64217a8dcc69c5fa48a5b484"
            },
            {
                "date": "2023-04-12T18:00:00.000Z",
                "user": "6414be936b0bbf2bd8fa964f"
            }
        ]
    },
]

How can I sort this array by the date inside schedulesDates array, keep the same array of objects structure, looking like this array:

const newSchedules = [
  {
      "_id": "6436b48b875967d0bea245b4",
      "service": "64246dc9a2d61593d103c749",
      "scheduledDates": [
          {
              "date": "2023-04-12T18:00:00.000Z",
              "user": "643701f7e5f61e6760f4d1f3"
          }
      ]
  },
  {
    "_id": "6436b48b875967d0bea245b5",
    "service": "64246dc9a2d61593d103c749",
    "scheduledDates": [
        {
            "date": "2023-04-12T18:00:00.000Z",
            "user": "6414be936b0bbf2bd8fa964f"
        }
    ]
},
  {
      "_id": "6436b48b875967d0bea245b4",
      "service": "64246dc9a2d61593d103c749",
      "scheduledDates": [
          {
              "date": "2023-04-17T18:00:00.000Z",
              "user": "643701f7e5f61e6760f4d1f3"
          }
      ]
  },
  {
      "_id": "6436b48b875967d0bea245b5",
      "service": "64246dc9a2d61593d103c749",
      "scheduledDates": [
          {
              "date": "2023-04-19T10:30:00.000Z",
              "user": "64217a8dcc69c5fa48a5b484"
          }
      ]
  },
]

I have this function to handle the sort:

  const handleSort = () => {
    if (sortOrder === "asc") {
      setSortOrder("desc");
      return schedules.map((schedule) =>
      schedule.scheduledDates.sort((a, b) => new Date(a.date) - new Date(b.date))
      );
    } else {
      setSortOrder("asc");
      return schedules.map((schedule) =>
      schedule.scheduledDates.sort((a, b) => new Date(b.date) - new Date(a.date))
    }
  };

But it sorts only within each array of scheduledDates. Thanks for helping.