504 gateway timeout error when bigquery is performing slowly

Context: I am naive developer. My team has an array of services deployed in k8lens. 2 services are there, service_1 : self-service, service_2: user-journey. self-service is a fullstack jhipster spring-boot and react app for internal purpose and user-journey is an apache-camel spring boot based microservice. self-service communicates with user-journey using grpc call.

Issue: Everyday from 9-11 am IST, bigquery is down for us. During this time when we query from self-service, then request flows till user-journey, job gets submitted to BQ and then after around 2-3 mins response is available with user-journey which it sends back (via the grpc response) to self-service which then should render on frontend (response of the http get call).
But it is not rendering it on frontend and before that only gateway timeout error 504 is comming.

(please ignore the time-stamps in the pictures above, and focus on the issue.)

On the frontend code, i have set the timeout of 1400 sec (verified from the request object sent via axios), on the backend it sends a grpc call to the user-journey and gets response after 2-3 mins (verified from the logs)

front-end code:

// user-journey

const onSubmit = async () => {
    setLoading(true);
    setResponse([]);
    setSummary({});
    setFilterList([]);
    const requestUrl = `api/user-journey/range?startTs=${
      Math.round(startDate.valueOf()) / 1000
    }&field=user_id&fieldValue=${fieldValue}&endTs=${Math.round(endDate.valueOf()) / 1000}`;
    
    console.log('Request URL from user-journey:', requestUrl);
    
    try {
      const payload = await axios.get(requestUrl);
      
      console.log('Payload from user-journey:', payload);
      
      const { filterData, res } = _calFilterData(payload.data.activity);
      setResponse(res);
      setSummary(filterData);
      setFilterList(Object.keys(filterData));
    } catch (e) {
      console.error('Error:', e);
      toast.error(e.response?.data?.customErrorMessage || e.response?.data?.detail);
    }
    setLoading(false);
  };

//axios-global-interceptor code

const TIMEOUT = 1 * 1400 * 1000;
axios.defaults.timeout = TIMEOUT;
axios.defaults.baseURL = SERVER_API_URL;

const setupAxiosInterceptors = onUnauthenticated => {
  const onRequestSuccess = config => {
    const token = Storage.local.get('jhi-authenticationToken') || Storage.session.get('jhi-authenticationToken');
    if (token) {
      config.headers.Authorization = `Bearer ${token}`;
    }
    return config;
  };
  const onResponseSuccess = response => response;
  const onResponseError = err => {
    const status = err.status || (err.response ? err.response.status : 0);
    if (status === 403 || status === 401) {
      onUnauthenticated();
    }
    return Promise.reject(err);
  };
  axios.interceptors.request.use(onRequestSuccess);
  axios.interceptors.response.use(onResponseSuccess, onResponseError);
};

export default setupAxiosInterceptors;

Only during the mentioned hours, when BQ is slow for us, the frontend gives 504 error. but the response is still comming as we can see on the logs of self-service and user-journey.

I tried to reproduce this error on my local, by putting thread.sleep(300_000) in backend code as well as multiple breakpoints on frontend. Frontend still renders the response even when the response comes at a delay of more than 30 mins. but on production it gives timeout 504 issue.

What could be the error ? how to resolve this ?

Things I have verified:

1. Frontend does set the timeout of around 1400 secs.
2. backend gets the response from the user-journey.
3. but frontend gives 504 error only in prod during the mentioned hours.

more context: 2 instances of both the service is deployed on k8lens. jhipster uses undertow as its server.