perf: split MaterializeIndex stream into batches

[wjones127]

Fixes #2768

After these changes, we can do the query using <2GB of RAM instead of 33GB! 🚀

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 83.33333% with 1 line in your changes missing coverage. Please review.

Project coverage is 77.94%. Comparing base (2838a87) to head (c9a7a3c).
Report is 11 commits behind head on main.

Files with missing lines	Patch %	Lines
rust/lance/src/io/exec/scalar_index.rs	83.33%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2770      +/-   ##
==========================================
- Coverage   78.47%   77.94%   -0.53%     
==========================================
  Files         228      229       +1     
  Lines       69341    70147     +806     
  Branches    69341    70147     +806     
==========================================
+ Hits        54417    54679     +262     
- Misses      11859    12389     +530     
- Partials     3065     3079      +14     

Flag	Coverage Δ
unittests	77.94% <83.33%> (-0.53%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jacketsj]

Wow nice this doesn't just fix my memory issue, it also seems to make my job progress 2.5x faster!

[jacketsj]

Unless there's a chance this might break something else, is there any chance you can un-draft this, and save a better solution for the backlog? It's probably not the best solution, but I've been actively using this branch, since it works for brute-force queries. In addition to fixing filter queries, it also seems to fix unfiltered queries on billion-scale datasets.

[westonpace]

Awesome, thanks. Can you delete the TODO on line 413?