I am using nodejs and input code as below:
async batchInsert(
productList: ProductList[]
} {
const productListFilter =
productList.map((product) => {
product.createdAt = new Date();
return product;
});
const batchSize = 100;
const batches = Math.ceil(productList.length / batchSize);
for (let i = 0; i < batches; i++) {
const startIndex = i * batchSize;
const endIndex = (i + 1) * batchSize;
const batch = productListFilter.slice(startIndex, endIndex);
}
return true;
}
Chatgpt suggest me as follow:
Key Strategies for Memory Management
Slice Responsibly:
Use the slice method carefully, as it creates a shallow copy of the array segment. If the array is large, ensure that you release references after processing.
Clear References:
After processing each batch, clear any references to large objects if they are no longer needed.
Batch Length Reset:
After processing each batch with await queryRunner.manager.insert(ProductList, batch);, we reset batch.length = 0;. This explicitly clears the references in the array, making them eligible for garbage collection.
My question is as follow:
-
Why do I need to explicitly setting batch.length = 0?? Wouldn’t it be clear off and garbage collect on each loop?
-
Why do i need to explicitly set productListFiltered.length = 0 in finally block? Is it because productListFiltered will not be garbage collect if it failed at catch block?
async batchInsert(productList: ProductList[]): Promise {
if (productList.length === 0) {
return true; // Nothing to insert
}// Create a filtered list without mutating the original products const productListFiltered = productList.map((product) => ({ ...product, createdAt: new Date(), })); const batchSize = 100; const batches = Math.ceil(productListFiltered.length / batchSize); const queryRunner = this.dataSource.createQueryRunner(); // Connect and start a transaction await queryRunner.connect(); await queryRunner.startTransaction(); try { for (let i = 0; i < batches; i++) { const startIndex = i * batchSize; const endIndex = Math.min(startIndex + batchSize, productListFiltered.length); const batch = productListFiltered.slice(startIndex, endIndex); // Perform the batch insert await queryRunner.manager.insert(ProductList, batch); // Clear reference to the batch after processing batch.length = 0; // Helps garbage collection } // Commit the transaction await queryRunner.commitTransaction(); return true; } catch (error) { await queryRunner.rollbackTransaction(); return false; } finally { // Release the query runner await queryRunner.release(); // Clear references to the filtered list productListFiltered.length = 0; // Helps garbage collection } }