Add: mlflow for experiment tracking #1059
Conversation
JohanWork
changed the title
| @@ -694,6 +694,10 @@ wandb_name: # Set the name of your wandb run | |||
| wandb_run_id: # Set the ID of your wandb run | |||
| wandb_log_model: # "checkpoint" to log model to wandb Artifacts every `save_steps` or "end" to log only at the end of training | |||
|
|
|||
| # mlflow configuration if you're using it | |||
| # Make sure your `MLFLOW_TRACKING_URI` is set. | |||
| mlflow_experiment_name: # Your experiment name | |||
There was a problem hiding this comment.
It might be worth logging a warning when this is set but the uri env isnt.
There was a problem hiding this comment.
Wouldn't it be better to make a config mlflow_tracking_uri: ? The parsing code should be able to set that if needed.
There was a problem hiding this comment.
I agree with what you propose @NanoCode012 . Have update the code.
Update mlflow_tracking_uri
|
Is there any need to pass this config to the HF Trainer to enable log? I think some validation test would also be needed to make sure both aren't active at same time. In addition, would you be able to provide an example run output + image on mlflow? |
|
I disagree, isn't the tracking uri equivalent to a secret token? It can get inadvertently exposed if you share your YAML (say integration with wandb also) |
To my understanding it is only an url actually. In Mlflow auth seams to be handled with username and password. But I might miss something. For reference https://mlflow.org/docs/latest/auth/index.html I have no strong opinion if it should be or not be in the config file. |
|
Oh, I wasn't aware it was a secret. In this case, I would not recommend it being in the yaml anymore due to reasons wing mentioned. |
update trainer building
|
looks like the URI isn't considered a secret, but could be sensitive if someone starts a server and doesn't enable auth on it. |
Adding MLFOW to Axolotl for experiment tracking, looked into how Weight and Bias has been setup and tried to follow the same pattern. Have tested the changes and everything looks good to me.
Happy for any feedback or comments.