-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Evaluate Hydra for composing config and command line arguments #807
Comments
In lightning, Using hydra could probably solve this issue. |
@hadim @sai-prasanna this sounds awesome. I'd love to support hydra. Mind looking into it and submit a proposal here? cc: @Borda |
My current ML project is using Hydra. It is pretty awesome ! Have a look: https://github.com/nicolas-chaulet/deeppointcloud-benchmarks |
I think that having parameters organised in groups would make it easier to navigate, on the other hand, passing
@PyTorchLightning/core-contributors ^^ |
I like the idea of
That being said I think it's important to keep allowing |
YAML is great and compares to JSON you can write a comment inside... lol @williamFalcon ^^ ? |
I would look into the current configuration flow and try to evaluate the pros and cons of hydra over it this weekend. |
Not a current lightning user, but our team is evaluating switching over. When you look at hudra, you might want to take a look at jssonet too. From what I can tell, hydra has a pretty mediocre use of variables, compared to jsonnet where you can cleanly define variables and do simple math and boolean logic on them. Jsonnet also implements composing multiple configs, just like hydra. And it can implement command line overrides via something like jsonargparse. Anyway, its useful for allennlp so might be worth taking a look to see if it is the right thing for pytorch-lightning |
I looked at hydra and this pretty high level and also opinionated the way they handle the configuration. That being said the configuration object they use is based on So I would propose to use/support |
I am a Allennlp contributor. Allennlp provides abstractions to make models generic. i.e. Say you have a text classifier, instead of putting an LSTM inside it, you would rather recieve a abstract I agree with @hadim 's assessment that being too opinionated is wrong choice for lightning. Instead it would be better to remove impediments for users to use their own config system, while maybe providing a sane default maybe as an example. I have two questions for regular users of lightning.
|
from allennlp import lstm class MyS2S(pl.LightningModule): def init(self, hparams): |
@williamFalcon In allennlp it is done that way to search registry using with type signature of objects in the constructor. @Model.register("seq2seq")
class Seq2Seq(Model):
def __init__(self, encoder: Seq2SeqEncoder):
...
@Seq2SeqEncoder.register("lstm")
class LSTM(Seq2SeqEncoder):
def __init__(self, h: int):
... Now in a configuration {
"model": {
"type": "seq2seq"
"encoder": {
"type": "lstm"
"h": 1000
}
} This allows multiple experiments to be made by configuration changes, without having to write a mapping between the configuration and the actual class. @williamFalcon I used plt few months back, so was a bit hazy. Since in the current work flow, creating models is explicit there is no need for anything more. Anyone wishing to use hydra can use that instead of argparse to construct the models easily. We can have an example of using hydra for some complex configuration case instead of argparse and be done with that. |
In AllenNLP there is effort now to get pytorch-lightning work as its trainer abstraction. I think AllenNLP's jsonnet configuration works well for it. Would be interesting to see how this synthesis goes. AllenNLP though focused on NLP, has very good dependency injection + configuration management abstractions that make writing models that are easily ammenable to experimentation via simple configuration changes. With this integration, one can use the powerful config system in allennlp for many different types of tasks (maybe with few changes here and there). |
Hi all!
Please correct me if I'm wrong and suggest the other features to add! |
Yeah, the way I'm currently using Hydra is:
and then in training
|
I've tried a few different approaches in using hydra with PL, and I believe this is the cleanest approach:
Example command:
Pros:
Cons:
|
I tried mixing This way Hydra doesn't own the logging directory structure. |
Friends, I did something like @lkhphuc:
#configs/config.yaml
dataset:
train_path: "resources/dataset/train.jsonl"
test_path: "resources/dataset/train.jsonl"
val_path: "resources/dataset/train.jsonl"
train:
batch_size: 32
test:
batch_size: 64
val:
batch_size: 32
preprocessing:
max_length: 64
@hydra.main(config_path="configs/config.yaml", strict=False)
def dev_run(cfg: DictConfig):
print(cfg.pretty())
model = JointEncoder(hparams=cfg, ... )
trainer = Trainer(fast_dev_run=True)
trainer.fit(model)
if __name__ == "__main__":
dev_run()
class JointEncoder(LightningModule):
"""Encodes the code and docstring into an same space of embeddings."""
def __init__(self,
hparams: DictConfig,
code_encoder,
docstring_encoder
):
super(JointEncoder, self).__init__()
self.hparams = hparams
...
self.loss_fn = NPairsLoss() But I'm getting the following error: ValueError: Unsupported config type of <class 'omegaconf.dictconfig.DictConfig'>. |
awesome this is great! mind adding a tutorial to the docs on this? (under hyperparameters) |
the problem here is that you always have to list out trainer args.
the above will enable all arguments in argparse |
@ceceu |
(just passing by) Note: |
@vict0rsch thanks for passing by! I actually no longer use that approach, as it is not the recommended approach. Please check out my blog post for more details! https://link.medium.com/KaROmhBvz9 |
This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team! |
I would like to propose that this issue be re-opened. Facebook Fairseq has now standardized on Hydra |
@turian The conclusion was that PL and Hydra are orthogonal, and there isn't really a good reason for PL to integrate Hydra directly as its default config/cli system. Hydra can be viewed as a fancier |
@yukw777 @williamFalcon I am still interested in an official tutorial / doc, as was suggested in #2322 |
@turian ah i see. in that case, yes PL definitely supports Hydra. As you can tell from the age of this issue, Hydra's been on our radar for a while, and we've fixed a number of bugs related to its integration into PL. For example, non-flat Hydra OmegaConf is supported, so no need to flatten it. I guess what we're missing is an official tutorial as we just have a disjoint set of seeder projects and blog posts. It is quite easy to start using Hydra with PL though. Do you have anything in particular you'd like to see in an official tutorial? |
@yukw777 At the bare minimum, I would like to see a skeleton PL with best practices for using Hydra. So that, for example, my Hydra configs appear in Tensorboard. I would also be interested in common pitfalls and gotchas to avoid. Lastly, some guidance on best practices for what happens if you are loading an old checkpoint, and how to play nicely with Hydra. For example, I might have non-PL-model config (URL for the dataset or whatever). If I load an old checkpoint, what are the pros and cons of:
So I'm mostly interested in patterns, best practices, and pitfalls to avoid. |
Also, I'd just really like to see what are best-practice for structuring your config, particularly if you are also using tools like ray.io and DVC or replicate.ai. Another example: Let's say I have two different neural networks and I want to switch between them to see what works better. Should I have the optim configs separately in each? Perhaps because each neural network will train better with different optimization learning rates etc. Or should I reduce duplication and just have one high-level optim config? Talking about hyperparameter optimization (e.g. optuna or ray), I might want in the config file to specify both:
|
I would also be interested in having some guidance / best practice about how to best integrate Hydra with PL |
+1, There doesnt seem to be any clear guidelines for integrating Hydra to PL. If we had an official best practice example as a starting point it would really help us greatly. |
I already have this simple example that can be expanded and generalized to something more expressive. |
I've come to this implementation. I think it's quite flexible
|
🚀 Feature
We can evaluate hydra for configuration.
Motivation
Hydra provides config file composition with overrides, command line completions and parameter sweep.
https://medium.com/pytorch/hydra-a-fresh-look-at-configuration-for-machine-learning-projects-50583186b710
Pitch
Evaluate whether hydra is more powerful than argparse. Check whether the overhead of it doesn't make stuff too complex.
The text was updated successfully, but these errors were encountered: