5-Fold with PyTorchLightning + Wandb seems to log to the same experiment

[tchaton]

I am training 5-fold CV with PyTorch Lightning in a for loop. I am also logging all the results to wandb. I want wanbd to reinitalize the run after each fold, but it seems to continue with the same run and it logs all the results to the same run. I also tried passing kwargs in the WandbLogger as mentioned in the docs here, with no luck.
Here's a pseudo code of it:

def run(fold):
    kwargs = {
        "reinit": True,
        "group": f"{CFG['exp_name']}"
    }
    wandb_logger = WandbLogger(project='<name>', 
                        entity='<entity>', 
                        config = CFG,
                        name=f"fold_{fold}",
                        **kwargs
            )
    trainer = Trainer(
        precision=16,
        gpus=1,
        fast_dev_run=False,
        callbacks = [checkpoint_callback],
        logger=wandb_logger,
        progress_bar_refresh_rate=1,
        max_epochs=2,
        log_every_n_steps=1

    )

    trainer.fit(
        lit_model,
        data_module
    )

if __name__ == "__main__":
    for fold in range(5):
        run(fold)

Originally posted by @Gladiator07 in #8572

[tchaton]

Dear @Gladiator07,

I converted this to an issue as it should be created several runs.

Best,
T.C

[awaelchli]

@Gladiator07 I think I have a workaround for you. Put this

import wandb
wandb.finish()

before instantiating WandbLogger.

This will make sure that the experiment from the previous "fold" gets finished. For context, our WandbLogger is simply wrapping the wandb.Run object which is sort of a global variable in wandb according to my understanding. I will try to make this into a real fix for our WandbLogger. Any feedback appreciated. Maybe @borisdayma has another idea :)

[awaelchli]

Actually, just found this PR which enforced the current behavior and also defines tests for this. #4648
As written on the PR users are advised to call wandb.finish() when they want to manually create multiple distinct experiments as OP wants. So what I presented as workaround is meant to be the real usage.

[borisdayma]

Yes this is perfectly correct @awaelchli
In some cases the users could want to use the same run, for example when training in multiple stages.

[justusschock]

@borisdayma @awaelchli could we say that one run corresponds to one logger object? So when training multiple stages you just reuse the logger object and if recreating that, you just get a new run?

[borisdayma]

So the main issue is that users could have a run already created before, even without the logger: using sweeps or for example if they use an artifact (like a previous checkpoint logged).

What do you think about adding a warning when a run is existing saying that we will be using the same one and that they can manually call wandb.finish()

[awaelchli]

@borisdayma I like that.

Perhaps my PR #8617 adding the finish() method should be closed as it is confusing to have finish() alongside finalize() and close() methods.

[borisdayma]

Yes I think I would just close it.
Basically we just need to change this line to add a warning.

[awaelchli]

See the new PR attached. Let me know what you think of this

[borisdayma]

I like it!