I am building a project. The part I am working on right now is supposed to do the following:
- log in to the platform using Id and Password (done)
- Take the array of data (hardcoded for now)
- perform an API call using each element of the array, get the result for each and put it to another array
- In a nutshell By passing an array of ids as an Input I should get an array of its results as output
To simplify the problem even more, I have decided to use one id and pass it 1000 times instead of array of 1000 different values.
LIMITATIONS:
I can call the API 300 times per minute or else I’d get the famous “429 problems” ->
My application does not use express. I was looking for alternatives and came across bottleNeck package Here
It does exactly what I need and is supposed to limit API calls as well as perform things as retries in case of failure or events management (see the documentation in the link above).
Here is a what I have so far:
const BottleNeck = require('bottleneck');
const logger = require('./functions/logger');
/**
* this reservoir is allowed to execute 300 requests/ minute
*/
const limiter = new BottleNeck({
reservoir: 300, // 300 requests per minute
reservoirRefreshAmount: 249,
reservoirRefreshInterval: 60 * 1000, // must be divisible by 250
trackDoneStatus: true,
minTime: 300//5 requests per second
});
limiter.on("debug", function (message, data) {
// Useful to figure out what the limiter is doing in real time
// and to help debug your application
});
limiter.on("depleted", function (empty) {
logger.info(`BottleNeck() - reservoir is empty. Waiting... ${limiter.options}`);
// This will be called every time the reservoir drops to 0.
// The `empty` (boolean) argument indicates whether `limiter.empty()` is currently true.
});
limiter.on("dropped", function (dropped) {
// This will be called when a strategy was triggered.
// The dropped request is passed to this event listener.
});
limiter.on("empty", function () {
// This will be called when `limiter.empty()` becomes true.
});
limiter.on("error", function (error) {
/* handle errors here */
logger.error(`Limiter : fetched error ${error.message}`);
});
// Listen to the "failed" event
limiter.on("failed", async (error, jobInfo) => {
const id = jobInfo.options.id;
console.warn(`Job ${id} failed: ${error}`);
if (jobInfo.retryCount === 0) { // Here we only retry once
console.log(`Retrying job ${id} in 25ms!`);
return 25;
}
});
limiter.on("idle", function () {
// This will be called when `limiter.empty()` is `true` and `limiter.running()` is `0`.
});
limiter.on("queued", function (info) {
// This event is triggered when a job transitions from one Lifecycle stage to another
});
// Listen to the "retry" event
limiter.on("retry", (error, jobInfo) => console.log(`Now retrying ${jobInfo.options.id}`));
module.exports = { bottleNeck: limiter };
(assume that logger is defined using logs4js or any other.
The main piece of code :
const { logger } = require("./functions/logger");
const { moment } = require("moment");
const { toCloudLogin, youOnlyHadOneJob } = require("./functions/toCloud");
const { bottleNeck } = require("./apiBottleNeck");
const go = async () => {
try {
logger.info(`hello`);
await toCloudLogin();
logger.info(`starting the bottleneck`);
let counter = 0;
for (let x = 0; x < 310; x++) { //calling same job through bottleneck many times
counter+=1;
let reservoir = await bottleNeck.currentReservoir(); // returns integer of the current
// reservoir api calls left
const result = await bottleNeck.schedule(async () => youOnlyHadOneJob(reservoir));
}
logger.info(`finished the bottleneck` + counter)
} catch (e) {
console.log(e);
logger.error(`error: ${e.message} stack :${e.stack}`);
}
}
go();
Here are my questions:
-
If bottleNeck is supposed to filter the amount of API calls , why do I still get Maximum call stack size exceeded knowing that I launch with 300 storage (which is actually 301 with 0) and 300 recharge
-
how should I implement all these events ( for example depleted:
limiter.on("depleted", function (empty) {
logger.info(`BottleNeck() - reservoir is empty. Waiting... ${limiter.options}`);
// This will be called every time the reservoir drops to 0.
// The `empty` (boolean) argument indicates whether `limiter.empty()` is currently true.
});
) In my main function so that it sends me anything when reservoir is empty. I have read on event Emitters and i assume that i should limiter.emit(depleted) somewhere but I do not understand how to properly integrate it.
