feat: make the I/O buffer size configurable

[westonpace]

fix: fix a potential deadlock when working with extremely large batches

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 62.71186% with 22 lines in your changes missing coverage. Please review.

Project coverage is 79.15%. Comparing base (1fd649f) to head (36e90b5).

Files	Patch %	Lines
...ust/lance-encoding/src/encodings/logical/struct.rs	67.34%	15 Missing and 1 partial ⚠️
rust/lance/src/dataset/scanner.rs	42.85%	4 Missing ⚠️
rust/lance-io/src/scheduler.rs	33.33%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2736      +/-   ##
==========================================
- Coverage   79.16%   79.15%   -0.02%     
==========================================
  Files         227      227              
  Lines       67356    67398      +42     
  Branches    67356    67398      +42     
==========================================
+ Hits        53322    53347      +25     
- Misses      10927    10940      +13     
- Partials     3107     3111       +4     

Flag	Coverage Δ
unittests	79.15% <62.71%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.