I need to scrape a page where:
- [Loop:
- There’s a long list with rows of items
- Final col in each row is a button which I must click
- [Loop:
- Clicking the button displays another control with additional data
- I scrape data within that new control
- I close the control
- :End Loop]
- I click ‘next page’
- :End Loop]
The buttons on each row seem indistinguishable, so the xpath to every button is:
//div[@class=”btn generic-btn”]
But that means every time I click a button, I’m just getting data for the first row. It is not recursing through rows.
Any ideas how to approach this? I need to click on each button as I work down the page, but Octoparse doesn’t seem happy to iterate through them and just keeps clicking the first one.
This is the page, if helpful (it takes 20 secs to load then please click the first blue button)
https://agent-counsellor-ukhub.britishcouncil.org/gal
Thanks in advance
Dan