I am using puppeteer to scrape a website for real time data in nodejs. Instead of scraping the page, I am watching the backend requests and capturing the JSON/TEXT responses so I have more structured data. (and seeing more data than what’s being displayed in the browser) Everything is working except some of the data is updated with a Firestore request. I can capture the response to that request, but only get the data when the request is complete.
If I monitor the request/response in the browser, I can see that there are several numbered “messages” in the response packet:
16
[[732,[“noop”]]]
16
[[733,[“noop”]]]
123
[[734,[{
“targetChange”: {
“resumeToken”: “CgkIgqb+n7TFiwM=”,
“readTime”: “2025-02-15T09:53:39.558146Z”
}
}
]]]
16
[[735,[“noop”]]]
with each message coming in over several seconds. At some point the request completes and a new firebase request is issued. The problem is I only see all these messages in my app once the response is complete and not in real time as each message comes in. (which my app requires in order to display changes in real time)
Is there a way to see the response data as it is received and not just when the request is completed?
page.on('response', async (response) => {
if (response.request().resourceType() === 'xhr') {
console.log('Firestore Response URL:', response.url());
const theResponse = await response.text();
console.log('response.text: ', theResponse);
}
}