I am trying to make a scraper that gets the reviews for hotel on tripadvisor.com. I was just working with pagination and testing if the browser would go all the way to the end, where there is no more pages.
Here is my code so far:
const puppeteer = require("puppeteer");
const cheerio = require("cheerio");
async function main() {
const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();
await page.goto('https://www.tripadvisor.com/Hotels-g298656-Ankara-Hotels.html');
while(true) {
await page.click('a[class="nav next ui_button primary"]');
await page.waitForNavigation({waitUntil: 'networkidle0'});
}
}
main();
However, this stops when the ‘accept cookies’ popup appears. How can I handle this?