Log pretrained model validation before fine-tuning

[hbredin]

To make sure fine-tuning actually improves performance, it would be nice to run (and log) a full validation loop before training.

Related Lightning-AI/pytorch-lightning#1715

[hbredin]

I haven't tried this yet but the linked issue above contains a "solution" that advises to run trainer.validate(model) before calling trainer.fit(model). Did you try this? Does this work?

[dapello]

yes, it seems like in v 1.5.1, calling trainer.validate(model) before trainer.fit(model) adds the initial validation score to the logs!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[hbredin]

Not perfect, but it does the trick...

cc @FrenchKrab @juanmc2005

[odedbd]

When running trainer.validate(model) before trainer.fit(model), the val dataloader is initialized twice (workers are initialized twice). In my use case, workers initialization is a costly process, so it would be beneficial if the val dataloader workers created when running trainer.validate would be reused when running trainer.fit.

Is there a way to set this up while using a DataModule as the datasource for trainer.validate and trainer.fit?

[juanmc2005]

@odedbd have you tried setting persistent_workers=True when instantiating the validation DataLoader? I think this could solve your problem.

[odedbd]

@juanmc2005 Thank you for the suggestion, I am already using persistent_workers=True and it doesn't seem to effect the creation of workers when using DataModule and running trainer.validate before trainer.fit.

I just realized I posted my comment on an issue of the pyannote-audio repository rather then that related Pytorch Lightning issue, where it probably belongs. Sorry for the confusion.