I’m working on a project that requires fetching large datasets from Firestore to display patient data. The data is structured as follows: patients/daily(collection)//document, where each document contains summary and samples fields. My goal is to read this data efficiently to display it in my application.
Problem:
Regardless of the approach I take, fetching data for significant intervals (e.g., the last 2 years, 3 months intervals going back 2 years, or combinations like 3m, 3m, 6m, 1y), the operations consistently take over 30 seconds, which is far from optimal. I’m seeking a method to significantly reduce this latency.
What I’ve Tried:
- Fetching the Last 2 Years Directly: Attempted to fetch the entire dataset for the last two years in one go.
- Interval Fetching: Tried fetching data in intervals (3m, 3m, 6m, 1y) sequentially.
- Batch Processing: Considered batch processing but unsure how to implement it effectively in Firestore.
Questions:
- Are there any Firestore-specific strategies or optimizations for reading large datasets that I might be overlooking?
- Is there a more efficient way to structure my database or queries to improve read performance?
- Could Firestore’s local caching capabilities be leveraged to improve subsequent read operations’ latency?
const collectionRef = collection(db, "patients", id, "daily");
const startDateStr = startDate.toISOString().split("T")[0];
const q = query(
collectionRef,
where("start_time", ">=", startDateStr),
where("start_time", "<", endDateStr)
);
const querySnapshot = await getDocs(q);
After this i go over every snapshot to retrieve the summary and samples into 2 different arrays one for the summary and one for the samples.
I expected that by breaking down the fetch operations into smaller intervals or by batching, I could reduce the overall latency. However, none of these methods have reduced the fetch time to under 30 seconds.