we have a script that spiders our website but since we added some new pages it doesnt complete the spidering.
its like it times out at some point.
this is the website that is crawled
http://www.solesurvivorleather.com
but it never completes the crawl on this site anymore
it works fine on my other sites though.
i have attatched crawler file so you can test on your own server.
