React Google Maps and unnecessary re-render

I am working on a simple app that shows a list of places in Europe.
I am using @react-google-maps/api and apollo client to get the data which is then passed to the markers.
Every time the user zooms or moves the map, the app will call a graphQL endpoint and get the new data to render, based on the map bounds.

My issue is that when I zoom on the map, it works fine and smoothly, but when I drag (move) the map then the map flashes and basically goes to the initial state. This happens also when I simply set the state inside the handleBounds function, but what I don’t understand is why it works fine with zooming but not with dragging, I tried also using the drag_end event but is always the same situation. I am stuck with this issue for some time now.
Here is the whole code:

export const Map = () => {
  const [map, setMap] = useState<google.maps.Map | null>(null)

  const {
    data = {
      map: [],
    },
    loading,
    error,
    refetch,
  } = useQuery(GET_PROPERTIES_ON_MAP, {
    variables: {
      bounds: map?.getBounds()?.toJSON(),
    },
  })

  const onLoad = useCallback(
    (map: google.maps.Map) => {
      map.fitBounds(EUROPE_BOUNDS)
      setMap(map)
    },
    [setMap]
  )

  const options: GoogleMapProps["options"] = {
    restriction: {
      latLngBounds: EUROPE_BOUNDS,
      strictBounds: false,
    },
  }

  const handleBounds = useCallback(() => {
    const bounds = map?.getBounds()?.toJSON()
    if (bounds) {
      refetch({ bounds })
    }
  }, [map, refetch])

  if (loading) {
    return <p>Loading ...</p>
  }

  return (
    <div
      style={{
        height: 400,
        borderRadius: 8,
        overflow: "hidden",
        marginTop: 32,
      }}
    >
      <GoogleMap
        mapContainerStyle={{ height: "100%" }}
        zoom={4}
        center={DEFAULT_CENTER}
        onLoad={onLoad}
        onBoundsChanged={handleBounds}
        clickableIcons
        options={options}
      >
        {data.map.map((property: Property) => (
          <Marker key={property.id} lat={property.lat} lng={property.lng} />
        ))}
      </GoogleMap>
    </div>
  )
}

this code

const handleBounds = useCallback(() => {
    const bounds = map?.getBounds()?.toJSON()
    if (bounds) {
      refetch({ bounds })
    }
  }, [map, refetch])

is what causes the issue, basically I tried all the GoogleMaps options including onDrag** events, but nothing each time the app re-renders, but weirdly it works when the user is zooming. Any idea what am I doing wrong?
Thanks