As it says I have a large list of records up to 10M and I need to query them very quickly within the browser
My initial naive attempt was simply to use a filter as below:
this.data.filter(row => row.age === 30);
This is incredibly slow once getting to these large sizes of records see stats below:
Record Count | Time MS (AVG) |
---|---|
10,000 | 7.199999999254942ms |
100,000 | 134.29999999701977ms |
1,000,000 | 6808.099999997765ms |
10,000,000 | DNF |
As you can see it gets silly long pretty fast so I then though well what if there were indexes of the values I know it’s some extra ram but you can get some speedup from this so I make a quick and dirty IndexedList Class (gist here) and while this does give me a huge performance boost I was wondering if there are any libraries around which have completed this all other questions I have seen have been talking about databases.
Record Count | Time MS (AVG) |
---|---|
10,000 | 0.09999999776482582ms |
100,000 | 0.8000000007450581ms |
1,000,000 | 7.799999997019768ms |
10,000,000 | 70.19999999925494ms |