I am creating a React site that retrieves web-scraping data from a third-party site. My puppeteer script runs fine as a .js file, but when I add it to my server using Express & Axios, it seemingly does not go to the page and cannot execute the following lines of code. (My code is a lot longer so I put a shorter example here:)
puppeteer.js // works fine
const puppeter = require('puppeteer');
(async () => {
const browser = await puppeter.launch({headless: true});
const page = await browser.newPage();
await page.goto(`https://www.scrapethissite.com/lessons/`);
let header = "";
header = await page.$eval( "#header", el => el.textContent.trim());
console.log(header)
await browser.close();
})();
index.js // does not work
const puppeteer = require('puppeteer');
app.get('/', async (req, res) => {
const browser = await puppeter.launch({headless: true});
const page = await browser.newPage();
await page.goto(`https://www.scrapethissite.com/lessons/`);
let header = "";
header = await page.$eval( "#header", el => el.textContent.trim());
console.log(header)
await browser.close();
});
The server code either outputs the header in the console as ” ” or when I modify it slightly to remove the initialization of the variable “header” and just have the line: let header = await page.$eval( "#header", el => el.textContent.trim());
I get an “UnhandledPromiseRejectionWarning.” The script works fine by itself, but when I use it in the server I start to get errors. Is this a problem with using Puppeteer in Axios or do I need to modify my code so it can handle the promises better?