I need to scrape a predefined data set from 3 or 4 websites. I dont really care what technology is actually used only that the content scraped is usable. The item to scrape is quite simple i.e 1 x digital file with 10 meta attributes with approx 100,000 item listed over four site. Being able to de-dupe content is a real plus.
