I have a small app that runs several requests(node-fetch) with in a queue with bullMQ.
And i have 3 worker threads (if i dont use this main event loop gets blocked and my whole app freezes.) (This is an another issue why main loop gets blocked with a few fetch requests ?)
queue A , queue B and queue C
all queues have same configuration
new Queue('a-queue', {
connection: {
url: process.env.REDIS_URL,
enableReadyCheck: true,
},
defaultJobOptions: {
attempts: 5,
removeOnComplete: true,
removeOnFail: false,
},
worker for the queue (a,b,c all same):
const processor = pathToFileURL(__dirname + '/a_processor.js')
new Worker('a-queue', processor, {
connection: {
url: process.env.REDIS_URL,
enableReadyCheck: true,
},
concurrency: 1,
useWorkerThreads: true,
})
And a_processor.js is sending some requests with fetch and returns the values.
Also i have a setting for the global concurrency limit in index.js (start of the express app):
await queue.setGlobalConcurrency(5)
Upto this points it seems like it works as i expected (bullmq dashboard blue ones are active yellow ones are waiting):
But after like a minute redis instance is crashing due high memory consumption:
Memory usage is like skyrocket when i run this app.And what is doing is just sending fetch requests (10-15 in paralel) and these requests are not like responsing mbs of data its only few kb of json each time.
Here’s the request method:
const headerGenerator = new HeaderGenerator()
const headers = headerGenerator.getHeaders()
const response = await fetch(`${domain}`, {
headers: headers,
agent: agentDcIp,
signal: AbortSignal.timeout(15000) as any,
})
const body = await response.text()
return {
body: body,
ok: response.ok,
status: response.status,
}
Any idea why this is happening and how to prevent it.