I have this 5 steps witch ares pretty clear how to handle it, but because the fetched data is valid only few seconds, I have to repeat all this steps every 20 seconds to keep it fresh and here come my problem witch I don’t know the direction to resolve it: How to keep track of all these collection names in NodeJs (which run in a single thread) in an atomic way (one completed operation at a time) to ensure data consistency?
1. Fetch 5 different API’s from some API providers (~120KB size loading time ~400ms each)
2. Save all on MongoDB (this is necessary because I want to keep the api calls under the rate limit and also to aggregate data)
3. Make some data filtering, sorting and aggregation operations on MongoDB (left join) and save the result in a new collection’s name
4. Call the new saved collections from MongoDB into NodeJs
5. Send it to my own API endpoint.
How I said, all this are no problem if all this steps would be once, but because is a repetitive process, I have no idea if that is even possible