Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hydra MLFlow Clash #2058

Closed
ssakhavi opened this issue Jun 3, 2020 · 2 comments · Fixed by #2216
Closed

Hydra MLFlow Clash #2058

ssakhavi opened this issue Jun 3, 2020 · 2 comments · Fixed by #2216
Labels
bug Something isn't working good first issue Good for newcomers help wanted Open to be worked on logger Related to the Loggers
Milestone

Comments

@ssakhavi
Copy link
Contributor

ssakhavi commented Jun 3, 2020

🐛 Bug

When using the MLFlow logger with Hydra, because the parameters passed to the LightningModule is a DictConfig, the condition in the logger/base.py is not met.

https://github.com/PyTorchLightning/pytorch-lightning/blob/8211256c46430e43e0c27e4f078c72085bb4ea34/pytorch_lightning/loggers/base.py#L177

To Reproduce

Use Hydra and MLFlow together.

Traceback (most recent call last):
  File "/home/siavash/KroniKare/kwae2/kwae_ma/models/pl_train_segmentation_model.py", line 115, in <module>
    main()
  File "/home/siavash/anaconda3/envs/kwae-ma/lib/python3.7/site-packages/hydra/main.py", line 24, in decorated_main
    strict=strict,
  File "/home/siavash/anaconda3/envs/kwae-ma/lib/python3.7/site-packages/hydra/_internal/utils.py", line 174, in run_hydra
    overrides=args.overrides,
  File "/home/siavash/anaconda3/envs/kwae-ma/lib/python3.7/site-packages/hydra/_internal/hydra.py", line 86, in run
    job_subdir_key=None,
  File "/home/siavash/anaconda3/envs/kwae-ma/lib/python3.7/site-packages/hydra/plugins/common/utils.py", line 109, in run_job
    ret.return_value = task_function(task_cfg)
  File "/home/siavash/KroniKare/kwae2/kwae_ma/models/pl_train_segmentation_model.py", line 111, in main
    trainer.fit(wound_seg_pl)
  File "/home/siavash/anaconda3/envs/kwae-ma/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 765, in fit
    self.single_gpu_train(model)
  File "/home/siavash/anaconda3/envs/kwae-ma/lib/python3.7/site-packages/pytorch_lightning/trainer/distrib_parts.py", line 492, in single_gpu_train
    self.run_pretrain_routine(model)
  File "/home/siavash/anaconda3/envs/kwae-ma/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 843, in run_pretrain_routine
    self.logger.log_hyperparams(ref_model.hparams)
  File "/home/siavash/anaconda3/envs/kwae-ma/lib/python3.7/site-packages/pytorch_lightning/loggers/base.py", line 275, in log_hyperparams
    [logger.log_hyperparams(params) for logger in self._logger_iterable]
  File "/home/siavash/anaconda3/envs/kwae-ma/lib/python3.7/site-packages/pytorch_lightning/loggers/base.py", line 275, in <listcomp>
    [logger.log_hyperparams(params) for logger in self._logger_iterable]
  File "/home/siavash/anaconda3/envs/kwae-ma/lib/python3.7/site-packages/pytorch_lightning/utilities/distributed.py", line 10, in wrapped_fn
    return fn(*args, **kwargs)
  File "/home/siavash/anaconda3/envs/kwae-ma/lib/python3.7/site-packages/pytorch_lightning/loggers/mlflow.py", line 105, in log_hyperparams
    self.experiment.log_param(self.run_id, k, v)
  File "/home/siavash/anaconda3/envs/kwae-ma/lib/python3.7/site-packages/mlflow/tracking/client.py", line 206, in log_param
    self._tracking_client.log_param(run_id, key, value)
  File "/home/siavash/anaconda3/envs/kwae-ma/lib/python3.7/site-packages/mlflow/tracking/_tracking_service/client.py", line 177, in log_param
    _validate_param_name(key)
  File "/home/siavash/anaconda3/envs/kwae-ma/lib/python3.7/site-packages/mlflow/utils/validation.py", line 120, in _validate_param_name
    INVALID_PARAMETER_VALUE)
mlflow.exceptions.MlflowException: Invalid parameter name: ''. Names may be treated as files in certain cases, and must not resolve to other names when treated as such. This name would resolve to '.'

Expected behavior

Check whether the instance if dict or DictConfig in the given line.

@ssakhavi ssakhavi added the help wanted Open to be worked on label Jun 3, 2020
@github-actions
Copy link
Contributor

github-actions bot commented Jun 3, 2020

Hi! thanks for your contribution!, great first issue!

@Borda Borda added the bug Something isn't working label Jun 3, 2020
@Borda
Copy link
Member

Borda commented Jun 3, 2020

Check whether the instance if dict or DictConfig in the given line.

@ssakhavi that sounds reasonable solution, mind sending a PR - fix and its test?

@Borda Borda added the good first issue Good for newcomers label Jun 3, 2020
@Borda Borda added this to the 0.8.0 milestone Jun 3, 2020
@Borda Borda modified the milestones: 0.8.0, 0.9.0 Jun 9, 2020
ssakhavi added a commit to ssakhavi/pytorch-lightning that referenced this issue Jun 17, 2020
This is in reference to Lightning-AI#2058 . 

To be honest, I have no idea how I should go about writing a test for this.
Borda pushed a commit to ssakhavi/pytorch-lightning that referenced this issue Jun 19, 2020
This is in reference to Lightning-AI#2058 . 

To be honest, I have no idea how I should go about writing a test for this.
Borda added a commit that referenced this issue Jun 23, 2020
* Checking if the parameters are a DictConfig Object

This is in reference to #2058 . 

To be honest, I have no idea how I should go about writing a test for this.

* Update pytorch_lightning/loggers/base.py

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>

* fix ...

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Co-authored-by: Jirka <jirka@pytorchlightning.ai>
@Borda Borda added the logger Related to the Loggers label Aug 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working good first issue Good for newcomers help wanted Open to be worked on logger Related to the Loggers
Projects
None yet
2 participants