Script To Download Web Pages
Write script to download html files to local directory for research project.
Site: http://interactive.web.insurance.ca.gov/survey/index.jsp
approximately 150,000 to 200,000 unique webpages
Script should:
1. submit form with first set of variables from drop down menus
2. load web page with 2009 HOMEOWNERS PREMIUM SURVEY RESULTS page (html table)
3. save html file locally
4. repeat with all combinations of unique variables that can be selected in form
Script should be python, perl, or php. Can be run from mac desktop (in terminal or automator shell script) or from remote Linux server.