I’ll try to explain everything the way I understand it as clearly as possible, so correct me if I’m confused with something.
I was trying to scrape the users from a member list on a website, I used Python and the first thing I did was making a post request to the Request URL with the required headers so I get a response that contains the data I need but this didn’t work, so I tried to find out the reason.
From what I understand now the website uses AJAX to make XHR and JavaScript calls which respond with the content (users).
The JS code is stored on a static website from what Chrome’s developer tool request initiators
tell me (Here is an image for reference), which responds with the HTML that contains the users
The idea is to create a script that runs this static JS script that’s stored online and fetch the data about the users from it. (Image for clarification)
How do I achieve this, I’m using python. What libraries do I need etc.? Any help/advice is greatly appreciated!