Celltype annotation automl #423
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
xingzhongyu
commented
Feb 27, 2024
xingzhongyu
commented
Feb 27, 2024
xingzhongyu
commented
Feb 27, 2024
dance/pipeline.py
Outdated
| @@ -835,6 +849,22 @@ def save_summary_data(entity, project, sweep_id, summary_file_path): | |||
| result.update({"id": run.id}) | |||
| summary_data.append(flatten_dict(result)) # get result and config | |||
| ans = pd.DataFrame(summary_data).set_index(["id"]) | |||
| conf = OmegaConf.load(conf_load_path) | |||
There was a problem hiding this comment.
Change the column names of the summary table
xingzhongyu
commented
Feb 27, 2024
| for x in row: | ||
| for k in conf.pipeline: | ||
| if k["target"] == x: | ||
| pipeline.append(k) | ||
| for i, f in zip(required_indexes, required_funs): |
There was a problem hiding this comment.
Some necessary functions, such as setConfig, can be obtained from the yaml of step 2 instead of the default yaml of step 3.
xingzhongyu
commented
Feb 27, 2024
dance/pipeline.py
Outdated
| count += 1 | ||
|
|
||
|
|
||
| def run_step3(MAINDIR, evaluate_pipeline, step2_pipeline_planer: PipelinePlaner, tune_mode="params", sweep_id=None): |
There was a problem hiding this comment.
The step3 function is run by default
xingzhongyu
commented
Feb 27, 2024
xingzhongyu
commented
Feb 27, 2024
| parser = argparse.ArgumentParser() | ||
| parser.add_argument("--batch_size", type=int, default=500) | ||
| parser.add_argument("--cache", action="store_true", help="Cache processed data.") | ||
| # parser.add_argument("--dense_dim", type=int, default=400, help="number of hidden gcn units") |
There was a problem hiding this comment.
There are hyperparameters that both the preprocessing function and the model depend on. Delete them and read them through the configuration file.
xingzhongyu
commented
Feb 28, 2024
| parser.add_argument("--valid_dataset", nargs="+", default=[1970], help="List of valid dataset ids.") | ||
| parser.add_argument("--seed", type=int, default=0, help="Initial seed random, offset for each repeatition") | ||
|
|
||
| parser.add_argument("--tune_mode", default="pipeline_params", choices=["pipeline", "params", "pipeline_params"]) |
There was a problem hiding this comment.
Need to add some parameters
xingzhongyu
commented
Feb 28, 2024
| parser.add_argument("--log_level", type=str, default="INFO", choices=get_args(LogLevel)) | ||
| parser.add_argument("--species", default="mouse") | ||
| parser.add_argument("--test_dataset", nargs="+", default=[2695], type=int, help="list of dataset id") | ||
| parser.add_argument("--tissue", default="Brain") # TODO: Add option for different tissue name for train/test | ||
| parser.add_argument("--train_dataset", nargs="+", default=[753], type=int, help="list of dataset id") | ||
| parser.add_argument("--valid_dataset", nargs="+", default=[3285], type=int, help="list of dataset id") | ||
| parser.add_argument("--tune_mode", default="pipeline", choices=["pipeline", "params"]) | ||
| parser.add_argument("--tune_mode", default="pipeline_params", choices=["pipeline", "params", "pipeline_params"]) |
There was a problem hiding this comment.
Add new tune_mode
xingzhongyu
commented
Mar 2, 2024
| self.train2valid() | ||
|
|
||
| def train2valid(self): | ||
| logger.info("Copy train_dataset and use it as valid_dataset") |
There was a problem hiding this comment.
train_dataset can be used as valid_dataset when needed
xingzhongyu
commented
Mar 2, 2024
xingzhongyu
commented
Mar 2, 2024
dance/transforms/normalize.py
Outdated
| n_cells: Optional[int] = None, | ||
| bin_size: int = 500, | ||
| bw_adjust: float = 3, | ||
| processes_num=os.cpu_count(), |
There was a problem hiding this comment.
The number of processes needs to be parameterized
xingzhongyu
commented
Mar 3, 2024
| args = parser.parse_args() | ||
| logger.setLevel(args.log_level) | ||
| logger.info(f"\n{pprint.pformat(vars(args))}") | ||
| MAINDIR = Path(__file__).resolve().parent | ||
| pipeline_planer = PipelinePlaner.from_config_file(f"{MAINDIR}/{args.config_dir}{args.tune_mode}_tuning_config.yaml") | ||
| file_root_path = Path( |
There was a problem hiding this comment.
Currently, the name of the data set is used as the folder, and the name of the biological tissue can also be used.
xingzhongyu
commented
Mar 3, 2024
|
|
||
|
|
||
| def run_step3(root_path, evaluate_pipeline, step2_pipeline_planer: PipelinePlaner, tune_mode="params"): | ||
| """Run step 3 by default. |
There was a problem hiding this comment.
Run step 3 by default.
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
… celltype_annotation_automl
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
… celltype_annotation_automl
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
… celltype_annotation_automl
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
RemyLau
approved these changes
Jun 14, 2024
| @@ -117,6 +117,27 @@ def __call__(self, data): | |||
| data.data.raw = data.data.copy() | |||
|
|
|||
|
|
|||
| @register_preprocessor("misc") | |||
| class UpdateRaw(BaseTransform): | |||
| """Update raw data. | |||
There was a problem hiding this comment.
UpdateRaw is confusing. Change to something like AlignRaw to more accurately reflect its function.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Complete the automl task about celltype_annotation. Currently, for each celltype annotation algorithm task, a data set is arbitrarily selected for a full set run. Due to the merge time problem, it does not end completely, but it is close to the complete run. I will wait until this merge is completed before resuming operation.