Releases: lancedb/lance
Releases · lancedb/lance
v0.15.0: Experimental FTS, packed struct encoding
What's Changed
Breaking Changes 🛠
- feat: upgrade DataFusion, Arrow, PyO3, ObjectStore by @wjones127 in #2594
New Features 🎉
- feat: add fast search to
to_table
method by @chebbyChefNEQ in #2591 - feat: loosen pyarrow pin by @wjones127 in #2597
- feat: add FSST string compression by @broccoliSpicy in #2470
- feat: integrate inverted index into lance index APIs by @BubbleCal in #2577
- You can now create full text search indices and pass text search queries into the
full_text_query
argument
- You can now create full text search indices and pass text search queries into the
- feat: add projection to take and take_rows by @westonpace in #2623
- feat: add storage_options to _BaseLanceDatasink, LanceDatasink, LanceCommitter by @BitPhinix in #2619
- feat: add bitpack encoding for LanceV2 by @albertlockett in #2333
- feat: add a packed struct encoding to lance by @raunaks13 in #2593
- feat: support deletion with stable row id and scalar indices by @wjones127 in #2609
Bug Fixes 🐛
- fix: avoid making empty range requests as these will err on S3 by @westonpace in #2592
- fix: upgrade AWS SDK by @wjones127 in #2616
- fix: specify behavior version when creating dynamodb config by @westonpace in #2624
- fix: pin numpy version to avoid numpy >= 2.0 until pyarrow supports it by @westonpace in #2637
Performance Improvements 🚀
- perf: improve inverted index performance by @BubbleCal in #2574
- perf: improve v2 scan performance by @westonpace in #2604
- perf: allow fragment scan for nearest query if there is a prefilter by @jiachengdb in #2631
- perf: speed up fsst decompression by @broccoliSpicy in #2626
- perf: skip documents with WAND by @BubbleCal in #2632
Other Changes
- refactor: move planner.rs, logical_expr.rs, and sql.rs from lance to lance_datafusion by @westonpace in #2622
- refactor: remove custom ProjectionExec by @wjones127 in #2608
New Contributors
- @BitPhinix made their first contribution in #2619
Full Changelog: v0.14.1...v0.15.0
v0.15.0-beta.2
What's Changed
New Features 🎉
- feat: add FSST string compression by @broccoliSpicy in #2470
- feat: integrate inverted index into lance index APIs by @BubbleCal in #2577
- feat: add projection to take and take_rows by @westonpace in #2623
Bug Fixes 🐛
- fix: upgrade AWS SDK by @wjones127 in #2616
- fix: specify behavior version when creating dynamodb config by @westonpace in #2624
Other Changes
- refactor: move planner.rs, logical_expr.rs, and sql.rs from lance to lance_datafusion by @westonpace in #2622
Full Changelog: v0.15.0-beta.1...v0.15.0-beta.2
v0.15.0-beta.1
What's Changed
Breaking Changes 🛠
- feat: upgrade DataFusion, Arrow, PyO3, ObjectStore by @wjones127 in #2594
New Features 🎉
- feat: loosen pyarrow pin by @wjones127 in #2597
Performance Improvements 🚀
- perf: improve inverted index performance by @BubbleCal in #2574
- perf: improve v2 scan performance by @westonpace in #2604
Full Changelog: v0.14.2-beta.1...v0.15.0-beta.1
v0.14.2-beta.1
What's Changed
New Features 🎉
- feat: add fast search to
to_table
method by @chebbyChefNEQ in #2591
Bug Fixes 🐛
- fix: avoid making empty range requests as these will err on S3 by @westonpace in #2592
Full Changelog: v0.14.1...v0.14.2-beta.1
v0.14.1
What's Changed
New Features 🎉
- feat: add inverted index by @BubbleCal in #2526
- feat(python): expose fast search flag in python by @eddyxu in #2573
- feat: use hyperloglog for cardinality estimation for dictionary encoding by @niyue in #2555
- feat: add scalar bitmap index to lance by @raunaks13 in #2560
- feat: add a standalone method for calculating the PQ model by @westonpace in #2562
- feat: integrate bitmap index into top-level lance APIs by @westonpace in #2575
- feat: support to set ef for search from python by @BubbleCal in #2569
- feat: dataset update accepts binary value by @eddyxu in #2579
- feat: do flat search if too many rows are filtered out by @BubbleCal in #2583
- feat: add label list index by @westonpace in #2581
- feat: allow passing in precomputed centroids to lance.util.KMeans by @jiachengdb in #2586
- feat: add standalone partition assignment operation by @westonpace in #2556
Bug Fixes 🐛
- fix: add support for list fields with names other than 'item' by @westonpace in #2580
- fix: propagate storage_options to v2 writer by @jiachengdb in #2578
- fix: be compatible with v1 index format by @BubbleCal in #2582
- fix: search panic with prefilter and HNSW by @BubbleCal in #2584
Performance Improvements 🚀
- perf: change scalar index to return RowIdTreeMap instead of u64 array by @westonpace in #2587
Full Changelog: v0.14.0...v0.14.1
v0.14.1-beta.2
What's Changed
New Features 🎉
- feat: support to set ef for search from python by @BubbleCal in #2569
Full Changelog: v0.14.1-beta.1...v0.14.1-beta.2
v0.14.1-beta.1
What's Changed
New Features 🎉
- feat: add inverted index by @BubbleCal in #2526
- feat(python): expose fast search flag in python by @eddyxu in #2573
- feat: use hyperloglog for cardinality estimation for dictionary encoding by @niyue in #2555
- feat: add scalar bitmap index to lance by @raunaks13 in #2560
- feat: add a standalone method for calculating the PQ model by @westonpace in #2562
- feat: integrate bitmap index into top-level lance APIs by @westonpace in #2575
Full Changelog: v0.14.0...v0.14.1-beta.1
v0.14.0
What's Changed
Breaking Changes 🛠
- fix: return correct sub index statistics by @BubbleCal in #2532
- feat: change to snapshot isolation by @wjones127 in #2537
New Features 🎉
- feat: enhance binary array encoding, make it the default by @westonpace in #2521
- feat: stable row id support in queries by @wjones127 in #2452
- feat: make it possible to create v2 fragments using the fragment API by @westonpace in #2534
- feat: add dictionary encoding to lance by @raunaks13 in #2409
- feat: provide a fast ANN search plan by @eddyxu in #2554
- feat: create standalone IVF training API in python by @westonpace in #2553
- feat: support u8 for existing distance types by @BubbleCal in #2501
- feat: reserve cores for IO by @chebbyChefNEQ in #2496
- feat: allow users to create a file writer without a schema by @westonpace in #2558
Bug Fixes 🐛
- fix: pytorch dot inference pass unwanted parameters by @eddyxu in #2539
- fix: show underlying cause of DynamoDB service error by @wjones127 in #2541
- fix: build HNSW with wrong distance type by @BubbleCal in #2533
- fix: propagate storage_options to LanceFragment.create and LanceDataset.commit by @jiachengdb in #2547
- fix: propagate shuffle_partition_concurrency from Python SDK by @jiachengdb in #2564
- fix: avoid multiple threads loading same index partition by @albertlockett in #2559
- fix: failed to return index stats if the first partition is empty by @BubbleCal in #2565
- fix: propagate storage_options to create_index for reading precomputed partition file by @jiachengdb in #2568
Documentation 📚
Performance Improvements 🚀
- perf: avoid allocating bitvec for each insertion while indexing HNSW by @BubbleCal in #2527
Other Changes
- refactor: move IVF_HNSW_SQ & IVF_FLAT to new buliding & search path by @BubbleCal in #2469
- refactor: flat search to use datafusion top k by @eddyxu in #2535
- refactor: remove generic PQ impl and trait by @BubbleCal in #2529
New Contributors
- @jiachengdb made their first contribution in #2547
- @walterddr made their first contribution in #2382
Full Changelog: v0.13.0...v0.14.0
v0.14.0-beta.2
What's Changed
New Features 🎉
- feat: add dictionary encoding to lance by @raunaks13 in #2409
- feat: provide a fast ANN search plan by @eddyxu in #2554
- feat: create standalone IVF training API in python by @westonpace in #2553
- feat: support u8 for existing distance types by @BubbleCal in #2501
- feat: reserve cores for IO by @chebbyChefNEQ in #2496
- feat: allow users to create a file writer without a schema by @westonpace in #2558
Bug Fixes 🐛
- fix: propagate shuffle_partition_concurrency from Python SDK by @jiachengdb in #2564
- fix: avoid multiple threads loading same index partition by @albertlockett in #2559
Other Changes
- refactor: remove generic PQ impl and trait by @BubbleCal in #2529
New Contributors
- @walterddr made their first contribution in #2382
Full Changelog: v0.14.0-beta.1...v0.14.0-beta.2
v0.14.0-beta.1
What's Changed
Breaking Changes 🛠
- fix: return correct sub index statistics by @BubbleCal in #2532
- feat: change to snapshot isolation by @wjones127 in #2537
New Features 🎉
- feat: stable row id support in queries by @wjones127 in #2452
- feat: make it possible to create v2 fragments using the fragment API by @westonpace in #2534
Bug Fixes 🐛
- fix: pytorch dot inference pass unwanted parameters by @eddyxu in #2539
- fix: show underlying cause of DynamoDB service error by @wjones127 in #2541
- fix: build HNSW with wrong distance type by @BubbleCal in #2533
- fix: propagate storage_options to LanceFragment.create and LanceDataset.commit by @jiachengdb in #2547
Performance Improvements 🚀
- perf: avoid allocating bitvec for each insertion while indexing HNSW by @BubbleCal in #2527
Other Changes
New Contributors
- @jiachengdb made their first contribution in #2547
Full Changelog: v0.13.1-beta.1...v0.14.0-beta.1