I am trying to scrap data from a webpage that shows a limited amount of data, and requires the user to click a button to navigate to the next set of records. The webpage achieves that by sending GET requests to itself.
I tried to write a code in Python that would send a GET request to the page hoping to get the next set of results, and write a for loop to retrieve subsequent results, but I am always getting the initial sets (apparently the website is ignoring my params)
This is the website I am targeting:
https://portaltransparencia.procempa.com.br/portalTransparencia/despesaLancamentoPesquisa.do?viaMenu=true&entidade=PROCEMPA
This is my code:
url = "https://portaltransparencia.procempa.com.br/portalTransparencia/despesaLancamentoPesquisa.do?viaMenu=true&entidade=PROCEMPA"
r_params = {
"perform": "view",
"actionForward": "success",
"validate": True,
"pesquisar": True,
"defaultSearch.pageSize":23,
"defaultSearch.currentPage": 2
}
page = requests.get(url, params=r_params)
I expected that this generated a response with data from the 2nd page, but it is responding that from the first page.