I wish to use puppeteer to simulate clicking on a link which posts data to the server and returns a csv file designed to be downloaded by the user. When that response arrives, I would like to read and process the response payload.
The response header includes:
content-disposition: attachment; filename=GVG-SearchResults-20231205121846.csv
content-type: text/csv
the code I am currently using throws an error because there is no body detected in the response – I am assuming because of the content-disposition above:
const url = 'https://www.greenvehicleguide.gov.au/Vehicle/Search';
browser = await puppeteer.launch({
headless: false,
});
let page = await browser.newPage();
// select Toyota then Corolla and click submit
await page.goto(url, {timeout: 180000, waitUntil: 'networkidle0'});
await page.waitForSelector('#VS_4_SelectedManufacturer option[value="3"]');
await page.select('#VS_4_SelectedManufacturer', '3');
const waitForSelectPop = 'https://www.greenvehicleguide.gov.au/Vehicle/GetNamesForSelectList';
await page.waitForResponse(waitForSelectPop);
await page.waitForSelector('#VS_4_VehicleModel option[value="Corolla"]');
await page.select('#VS_4_VehicleModel', 'Corolla');
await page.click('#submitType4');
await page.waitForNavigation();
// wait until the csv download link is available, click it and intercept response
const csvSelector = 'a.csv';
await page.waitForSelector(csvSelector);
const responsePromise = page.waitForResponse(
response => response.headers()['content-disposition']
&& response.headers()['content-disposition'].startsWith('attachment')
);
await page.click(csvSelector);
const response = await responsePromise;
if (response.status() === 200) {
const text = await response.text(); // throws
// const buffer = await response.buffer(); // also throws
// ... do stuff here which needs to run in a node context - processing and saving to fileSystem or database
}
this throws on the response.buffer() (or response.text()) with:
...myProjectnode_modulespuppeteer-corelibcjspuppeteercdpHTTPResponse.js:103
return Buffer.from(response.body, response.base64Encoded ? 'base64' : 'utf8');
^
TypeError: Cannot read properties of undefined (reading 'body')
In summary – how do I obtain the body of the response and use it within the NodeJS context running pupeteer, overcoming the content-disposition of the response being attachment?
