-
Notifications
You must be signed in to change notification settings - Fork 22
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[builder] ongoing refactoring for Dask (#1040)
* first cut at fixed budget anndata handling * memory * refactor consolidate * checkpoint refactoring for memory budget * always have at least one worker * smaller strides * improve memory diagnostics * autoupdate precommit modules * fix bug in no-consolidate * update test to match new manifest field requirements * remove unused code * further memory budget refinement and tuning * add missing __len__ to AnnDataProxy * further memory usage reduction * preserve column ordering in dataframe loading * comments and cleanup * add extra verbose logging level * back out parallel consolidation for now * added a todo reminder * a few more memory tuning tweaks * simplify open_anndata interface * pr review * clean up logger * lint * snapshot initial dask explorations * pr feedback * additional dask refactoring * fix empty slice bug * additional refactoring to use dask * refine async consolidator * checkpoint progress * additional X layer processing refinement * fix pytest * fix mocks in test * update package deps for builder * comment * improve dataset shuffle * tuning * update to latest tiledb * update to latest tiledb * cleanup * additional scale updates * fix numpy cast error * shorten step count for async consolidator * additional cleanup * update to latest cellxgene_census * update tiledbsoma dep * lint * tune thread count cap * update to latest tiledbsoma * lint * remove debugging code * checkpoint partial refactoring * second checkpoint * clean up logging * add docstring * third checkpoint * further refinement of validation refactoring * dep update and cleanup * lint * fix builder test * additional cleanup of heartbeat and exit hang * cleanup minor detritus from refactoring * additional cleanup * fix tests and missing parameterization * remove dead code * remove dead code
- Loading branch information
Bruce Martin
authored
Mar 12, 2024
1 parent
b72dac1
commit 44c01f9
Showing
17 changed files
with
944 additions
and
1,114 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.