I’m currently working on a big scapping project so i use a main big php script to run all of my scrapping (over 150 websites) the script take around 5-8h to run.
So in my main script i have a foreach who run a exec(‘php -f …’) who run the scraping for one website.
I would prevent some crash of this php exec because when i have multiples timeout the script crashing and don’t continue so maybe i can replace :
echo exec('php -f ...');
by :
try{
echo exec('php -f ...');
}catch(Exception $e){
// i will put a log here
continue;
}
But i think it’s not working with timeout (fatal error), so what is the best option to run all of my scripts without crash stop it ?
Thanks !