I need to crawl some data from a certain analytics site. I’d like to seed some data in a table (in the form of domain names), and for the crawler to get the number of unique visitors for each site, and populate that back into the table. The crawler should wait X seconds between each call to avoid any blocks. If you private message me, I’ll tell you which site this is.
If anyone knows how to do this through a proxy server, to guarantee it won’t get shutdown, that would be even better.