HPO with Comet backend does not work

[ashwinvaidya17]

Describe the bug

• HPO fails with key error when used with comet.

To Reproduce

Steps to reproduce the behavior:

python tools/hpo/sweep.py --model stfpm --sweep_config tools/hpo/configs/comet.yaml 

Stack Trace

COMET INFO: COMET_OPTIMIZER_ID=ce90bba8fae04d5fa9875ee119b080ed
COMET INFO: Using optimizer config: {'algorithm': 'bayes', 'configSpaceSize': 4, 'endTime': None, 'id': 'ce90bba8fae04d5fa9875ee119b080ed', 'lastUpdateTime': None, 'maxCombo': 20, 'name': 'ce90bba8fae04d5fa9875ee119b080ed', 'parameters': {'dataset.image_size': {'type': 'discrete', 'values': [128, 256]}, 'model.backbone': {'type': 'categorical', 'values': ['resnet18', 'wide_resnet50_2']}}, 'predictor': None, 'spec': {'gridSize': 10, 'maxCombo': 20, 'metric': 'pixel_AUROC', 'minSampleSize': 100, 'objective': 'maximize', 'retryAssignLimit': 0, 'retryLimit': 1000}, 'startTime': 7711165905, 'state': {'mode': None, 'seed': None, 'sequence': [], 'sequence_i': 0, 'sequence_pid': None, 'sequence_retry': 0, 'sequence_retry_count': 0}, 'status': 'running', 'suggestion_count': 0, 'trials': 1, 'version': '2.0.1'}
COMET WARNING: Comet has disabled auto-logging functionality as it has been imported after the following ML modules: tensorboard, torch. Metrics and hyperparameters can still be logged using comet_ml.log_metrics() and comet_ml.log_parameters()
COMET ERROR: Failed to calculate active processors count. Fall back to default CPU count 1
COMET INFO: Experiment is live on comet.com https://www.comet.com/ashwin-vaidya/stfpm-hazelnut/194c327a17854bcaa634fb971144a26b

CometLogger will be initialized in online mode
Traceback (most recent call last):
  File "./tools/hpo/sweep.py", line 144, in <module>
    sweep.run()
  File "./tools/hpo/sweep.py", line 106, in run
    set_in_nested_config(self.config, param.split("."), run_params[param])
  File "/usr/local/lib/python3.7/dist-packages/anomalib/utils/sweep/config.py", line 133, in set_in_nested_config
    get_from_nested_config(config, keymap[:-1])[keymap[-1]] = value
TypeError: 'str' object does not support item assignment
COMET INFO: Optimizer metrics is 'pixel_AUROC' but no logged values found. Experiment ignored in sweep.
COMET INFO: ---------------------------
COMET INFO: Comet.ml Experiment Summary
COMET INFO: ---------------------------
COMET INFO:   Data:
COMET INFO:     display_summary_level : 1
COMET INFO:     url                   : https://www.comet.com/ashwin-vaidya/stfpm-hazelnut/194c327a17854bcaa634fb971144a26b
COMET INFO:   Others:
COMET INFO:     optimizer_count        : 1
COMET INFO:     optimizer_id           : ce90bba8fae04d5fa9875ee119b080ed
COMET INFO:     optimizer_metric       : pixel_AUROC
COMET INFO:     optimizer_metric_value : 1
COMET INFO:     optimizer_name         : ce90bba8fae04d5fa9875ee119b080ed
COMET INFO:     optimizer_objective    : minimum
COMET INFO:     optimizer_parameters   : {"dataset.image_size": 256, "model.backbone": "resnet18"}
COMET INFO:     optimizer_pid          : b0a87dc3b62fb6abe9ac181c7b00bdd69a1adc44
COMET INFO:     optimizer_process      : 981
COMET INFO:     optimizer_trial        : 1
COMET INFO:     optimizer_version      : 2.0.1
COMET INFO:   Parameters:
COMET INFO:     dataset.image_size : 256
COMET INFO:     model.backbone     : resnet18
COMET INFO:   Uploads:
COMET INFO:     environment details      : 1
COMET INFO:     filename                 : 1
COMET INFO:     git metadata             : 1
COMET INFO:     git-patch (uncompressed) : 1 (253 bytes)
COMET INFO:     installed packages       : 1
COMET INFO:     os packages              : 1
COMET INFO:     source_code              : 1 (5.35 KB)
COMET INFO: ---------------------------
COMET WARNING: Comet has disabled auto-logging functionality as it has been imported after the following ML modules: tensorboard, torch. Metrics and hyperparameters can still be logged using comet_ml.log_metrics() and comet_ml.log_parameters()
COMET INFO: Uploading metrics, params, and assets to Comet before program termination (may take several seconds)
COMET INFO: The Python SDK has 3600 seconds to finish before aborting...

• OS: Ubuntu 20.1

Possible Solution

1. The replace config method in sweep might be wandb specific. We will have to modify it to match comet keys.