v0.18.1-beta.4

What's Changed

New Features 🎉

• feat: allow users to specify late/early materialization, adjust default threshold by @westonpace in #2916
• feat: add an optional process-wide limit for IOPS by @westonpace in #2928

Bug Fixes 🐛

• fix: support negative values in SQL array literals by @wjones127 in #2925
• fix: v2 deadlock when there are multiple data files by @westonpace in #2923

Full Changelog: v0.18.1-beta.3...v0.18.1-beta.4

v0.18.1-beta.3

What's Changed

New Features 🎉

• feat: cuvs acceleration for gpu k-means by @jacketsj in #2816

Bug Fixes 🐛

• fix: fix dot distance for cuda by @westonpace in #2914
• fix: use existing storage version on overwrite, if none specified by @westonpace in #2924

Full Changelog: v0.18.1-beta.2...v0.18.1-beta.3

v0.18.1-beta.2

What's Changed

New Features 🎉

• feat: make Debug for useful Session by @wjones127 in #2903
• feat: log the execution plan in a more compact fashion by @westonpace in #2906
• feat(java): add dataset open storage options by @LuQQiu in #2871
• feat: reconnect block compression by @westonpace in #2878
• feat: handle nulls when creating indices with cuda by @westonpace in #2910

Bug Fixes 🐛

• fix: restore ability to write fixed size list of nullable by @westonpace in #2907
• fix: use retry loop when loading deletion files by @westonpace in #2899

Full Changelog: v0.18.1-beta.1...v0.18.1-beta.2

v0.18.1-beta.1

What's Changed

Bug Fixes 🐛

• fix: enlarge cargo release timeout and add workflow dispatch by @LuQQiu in #2901
• fix: fix cargo release tag checkout by @LuQQiu in #2902

Full Changelog: v0.18.0...v0.18.1-beta.1

v0.18.0

What's Changed

Breaking Changes 🛠

• feat!: change the default data storage version to "stable" (e.g. v2.0) by @westonpace in #2829
• feat!: update_table returns stats by @QianZhu in #2867
• Revert "chore!: remove manylinux 2014 build (#2891)" by @eddyxu in #2892
• feat!: add batch_size option to merge_columns by @westonpace in #2896

New Features 🎉

• feat: propagate storage_options to LanceFileWriter and LanceFileReader by @ankitvij-db in #2840
• feat(java): support Lance Spark Batch Write by @LuQQiu in #2500
• feat: add a separate retry loop on object contents download by @westonpace in #2874
• feat: support decimal 128 and 256 json types by @LuQQiu in #2880
• feat: add CreateIndex commit type to python API by @jiachengdb in #2883

Bug Fixes 🐛

• fix: ensure the v2 file writer does not mix up the order of a column by @westonpace in #2836
• fix: calculate decode priority correctly by @westonpace in #2841
• fix: vector column mismatch should not be a IO Error by @LuQQiu in #2845
• fix: phrase query may return incorrect results if there's unknown word by @BubbleCal in #2850
• fix: check label list list column type earlier by @LuQQiu in #2846
• fix: can't filter on string column which has FTS index by @BubbleCal in #2875
• fix: limit/offset pushdown was not working correctly in the presence of deletions by @westonpace in #2895
• fix: fix release versions by @LuQQiu in #2897
• fix: revert Bump version f3eef61 by @LuQQiu in #2900

Performance Improvements 🚀

• perf: parallelize remapping to improve FTS compaction by @BubbleCal in #2834
• perf: concurrent loading FTS index files by @BubbleCal in #2787

Other Changes

• refactor: convert encoders to use data blocks by @westonpace in #2817

Full Changelog: v0.17.0...v0.18.0

v0.18.0-beta.2

Full Changelog: v0.18.0-beta.1...v0.18.0-beta.2

v0.18.0-beta.1

What's Changed

Breaking Changes 🛠

• feat!: change the default data storage version to "stable" (e.g. v2.0) by @westonpace in #2829

New Features 🎉

• feat: propagate storage_options to LanceFileWriter and LanceFileReader by @ankitvij-db in #2840
• feat(java): support Lance Spark Batch Write by @LuQQiu in #2500
• feat: add a separate retry loop on object contents download by @westonpace in #2874

Bug Fixes 🐛

• fix: vector column mismatch should not be a IO Error by @LuQQiu in #2845
• fix: phrase query may return incorrect results if there's unknown word by @BubbleCal in #2850
• fix: check label list list column type earlier by @LuQQiu in #2846
• fix: can't filter on string column which has FTS index by @BubbleCal in #2875

Other Changes

• refactor: convert encoders to use data blocks by @westonpace in #2817

Full Changelog: v0.17.1-beta.2...v0.18.0-beta.1

v0.17.1-beta.2

What's Changed

Bug Fixes 🐛

• fix: calculate decode priority correctly by @westonpace in #2841

Performance Improvements 🚀

• perf: parallelize remapping to improve FTS compaction by @BubbleCal in #2834
• perf: concurrent loading FTS index files by @BubbleCal in #2787

Full Changelog: v0.17.1-beta.1...v0.17.1-beta.2

v0.17.1-beta.1

What's Changed

Bug Fixes 🐛

• fix: ensure the v2 file writer does not mix up the order of a column by @westonpace in #2836

Full Changelog: v0.17.0...v0.17.1-beta.1

v0.17.0

What's Changed

Breaking Changes 🛠

• feat!: index_statistics returns the concrete index type by @BubbleCal in #2716
• feat: stop writing _latest.manifest by @wjones127 in #2776
• refactor: build & search vector index with new indexer & IVF impl by @BubbleCal in #2552

New Features 🎉

• feat: add max_bytes_per_file and batch_size to CompactionOptions by @westonpace in #2728
• feat: add fixed size binary encoding to lance by @raunaks13 in #2707
• feat: make the I/O buffer size configurable by @westonpace in #2736
• feat: pushdown limit & offset into file reader if there is no filter by @westonpace in #2747
• feat: support phrase query for full text search by @BubbleCal in #2751
• feat: make the # of CPU threads configurable and document cpu/memory patterns by @westonpace in #2773
• feat: add logging to compaction by @westonpace in #2791
• feat(python): account for both multi-process and distributed torch workers by @tonyf in #2761
• feat: allow duckdb / polars pushdown to operate with non-substrait types in schema but not filter by @westonpace in #2796
• feat: add an environment variable LANCE_INITIAL_UPLOAD_SIZE by @westonpace in #2806
• feat: extend python bindings for the v2 reader/writer by @ankitvij-db in #2800
• feat: provide param to control whether to build FTS index with positions by @BubbleCal in #2795
• feat: allow creating fragments from v2 files, expose rewrite operation to python by @westonpace in #2811
• feat: use a considerably higher default for frag readahead in v2 by @westonpace in #2797
• feat: support update tags by @broccoliSpicy in #2813
• feat(java): add create vector search index, list index names, and vector search scan by @LuQQiu in #2782
• feat: make default I/O buffer size configurable via env var by @westonpace in #2826
• feat: constant-time manifest lookup on object stores by @wjones127 in #2798

Bug Fixes 🐛

• fix: drop indices if all fragment ids are missing by @wjones127 in #2720
• fix: ignore pages with zero rows by @westonpace in #2724
• fix: various changes to fix backpressure by @westonpace in #2721
• fix: raise exception if user passes unsorted indices to take_rows() of lance file api by @raunaks13 in #2729
• fix: flaky test caused by PQ distortion by @BubbleCal in #2732
• fix: don't panic when v2 scans end early (reader is dropped) by @westonpace in #2690
• fix: fix a crash that could sometimes happen reading largish string/binary data in a list by @westonpace in #2731
• fix: fix two cases in the v2 decoder where the decode order didn't match the scheduling priority by @westonpace in #2754
• fix: fix several situations where we were incorrectly inferring the storage version by @westonpace in #2756
• fix: rework priority handling for lists. We now properly schedule list item pages in priority order by @westonpace in #2769
• fix: try and fix an invalid data storage version automatically by @westonpace in #2759
• fix: add not-linux version of benches to avoid bench compiler error on mac by @westonpace in #2794
• fix: fix v2 error that can happen when writing list<struct<...>> with many empty lists by @westonpace in #2762
• fix: torch to_tensor for FixedShapeTensorType data by @jacketsj in #2824
• fix: cleanup external staging manifests by @wjones127 in #2792
• fix: the DataFile version doesn't respect the writer's version by @BubbleCal in #2825
• fix: merge-insert and update can't work on v2 format by @BubbleCal in #2833

Documentation 📚

• docs: clean up quickstart notebook and add tags example usage by @dsgibbons in #2735
• docs: fix version table so it renders by @wjones127 in #2745
• docs: document newly added compaction options in optimize.py by @westonpace in #2746
• docs: fix transaction conflict table by @wjones127 in #2744
• docs: fix #2748, "Merge Insert" doc error by @broccoliSpicy in #2788

Performance Improvements 🚀

• perf: coalesce ids before executing take by @westonpace in #2680
• perf: cache default session contexts by @dsgibbons in #2709
• perf: various fixes to improve shuffling performance at high scales by @westonpace in #2710
• perf: calculate the max scores for posting lists then have a tighter upper bound by @BubbleCal in #2763
• perf: stable row id prefilter by @wjones127 in #2706
• perf: add prefetching to hnsw greedy_search by @jacketsj in #2783
• perf: parallelize FTS indexing by @BubbleCal in #2807
• perf: split MaterializeIndex stream into batches by @wjones127 in #2770

Other Changes

• refactor: convert DataBlock to an enum, add conversion from arrow, normalize dictionaries by @westonpace in #2789

New Contributors

• @dentiny made their first contribution in #2740
• @jacketsj made their first contribution in #2783
• @tonyf made their first contribution in #2761
• @ankitvij-db made their first contribution in #2800

Full Changelog: v0.16.1...v0.17.0