I’m currently attempting to scrape a wiki for some image files. I have determined that every image I want is hosted at a URL with the following structure:
https://static.wikia.nocookie.net/<game name>/images/X/XY/<file name.png>
In all cases I know the exact file name corresponding to the image I’m searching for. However, here’s my issue: X is always a one-digit hexadecimal number e.g. 3, and XY is always a two-digit hexadecimal number whose first digit is the same as X, e.g. 3c. But as far as I can tell these numbers are completely arbitrary and there is no way to reliably predict them in advance for a specific image I want to retrieve.
My plan moving forward is to search through the entire web directory until I find the files I want, check the exact URL they are stored at, and write them to a local file for instantaneous subsequent lookup. To accomplish this, I see two options:
- For a given
XandXY, I could somehow retrieve the entire directory at.../images/X/XY/, check what files are stored there, and write all of the URLs to a local file. - For a given file name, I iterate through all possible combinations of
XandXYuntil I find where the file is stored, and write its URL to a local file.
In total I have several thousand images I want to find the URLs for. Given that, option 1 would appear to be an astronomically faster approach, but I’m not sure if retrieving an entire directory of files from the web at once is possible. Can it be done with HTTPS requests (I’m using Node.js for reference)? If not, are there any other tools I could potentially use, or will I have to resort to option 2?