I’m building a NestJS application where users upload multiple files via an API. These files are uploaded to an S3 bucket, but the upload process takes time, so I decided to use BullMQ to process the uploads in the background. This way, the client doesn’t have to wait for the S3 upload. Problem: The files are uploaded as Buffer objects, and I add them to the queue as part of a job. However, when passing these Buffer objects through Redis (via BullMQ), they are automatically converted into JSON, which results in the following format:
{
type: 'Buffer',
data: [137, 80, 78, 71, 13, ...]
}
In the worker, I have to manually recreate the buffer from this JSON object using:
const buffer = Buffer.from(job.data.file.data);
This works, but I’m wondering if it’s the best practice for handling binary data in BullMQ. I’m especially concerned about the efficiency of storing and retrieving large file buffers in Redis. My Requirements: I don’t want the client to wait for the file to be uploaded to S3. The client uploads multiple files, and I want the API to respond quickly after receiving the files, while the actual file upload happens in the background.
My Requirements:
- I don’t want the client to wait for the file to be uploaded to S3.
- The client uploads multiple files, and I want the API to respond quickly after receiving the files, while the actual file upload happens in the background.
Current Implementation:
- Client uploads files: The client sends multiple files to the API.
- Files are received as Buffers: The server temporarily stores the Buffer and adds it to a BullMQ job.
- Recreate Buffers in Worker: On the worker side, I manually recreate the Buffer using Buffer.from() from the JSON data before uploading to S3.
My Questions:
- Is recreating the Buffer from JSON data the most efficient way to handle binary data in BullMQ?
- Are there any performance concerns with storing large buffers in Redis as JSON (e.g., increased memory usage or serialization overhead)?
- Is there a better approach that keeps the client from waiting while efficiently handling the file upload to S3 in the background?
Any insights, recommendations, or best practices would be appreciated!