I need a web crawler that will eventually crawl all web sites in the world and index the home page. There should only be one page per url and then it should go to the next URL. The data must be stored in a SQL Server 2008 database so that it can be searched.
I am not sure where to get the starting point for the web crawler, but I would imagine that there would be many places to get urls that can act as the seeds. If you are bidding on this project, please know where you will get the seed w…
