Webharvest / Beautiful Soup
Hi there,
I am looking for a developer experienced with BeautifulSoup software (http://www.crummy.com/software/BeautifulSoup/).
I need to automate the checking of several webpages :
I am a professional mail router and i use many IP adresses to perform my routing job.
Each of them have a reputation that evolves with time, depending on how many mails i send, the deviation from the average number of mails sent, how many complaints, etc.
Several websites offer free services to check the IP reputation (for example : http://www.trustedsource.org/query/188.165.126.78).
Of course, checking each IP reputation “by hand” is impossible when you deal with several hundreds IPs…
Therefore i need to create an “IPPanel” kind of page that would interact with a MySQL table (in which i can list the IPs and domain names to check, the URLs of the web pages where to check, and of course the results of the checks).
To achieve this, I need to “web-harvest” the pages.
I came across Beautiful Soup, which is apparently the software of choice to perform such task.
Can anybody help ?