Trainer.from_argparse_args with additional kwargs causes model to not be saved #1714
Labels
Comments
|
Hi! thanks for your contribution!, great first issue! |
|
I had the same issue. I fixed it using the following "import copy; self.hparams=copy.deepcopy(hparams)". This is a quick fix, but maybe it can help you. |
|
I usually have some lines like mparams = Namespace(**vars(params))
del mparams.loggerfor this after I've created the trainer object with |
I try your solution, but it makes trainer can't save model. Here is my code: Also I have tried Is there any solutions to this problem? |
This solved an issue I had that occurred only when returning val_loss from Show traceback
|
Changing just this small part of the source code worked for me Another workaround would be to use for eg. Trainer(gpus = hparams.gpus ...), which is not suggested but will definitely work |
|
Just confirmed that #2029 fixes the issue for me. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
🐛 Bug
When using Trainer.from_argparse_args() to initialize the trainer, there will be some specific arguments that we would like to keep constant and not send as part of hparams. If the extra arguments turn out to be an object, such as a TensorBoardLogger or a ModelCheckpoint object, the model will not be saved because these objects get added to hparams
To Reproduce
Code sample
import os from argparse import ArgumentParser import pytorch_lightning as pl import torch from pytorch_lightning import Trainer from pytorch_lightning.loggers import TensorBoardLogger from torch.nn import functional as F from torch.utils.data import DataLoader from torchvision import transforms from torchvision.datasets import MNIST class LitModel(pl.LightningModule): def __init__(self, hparams): super().__init__() self.hparams = hparams self.l1 = torch.nn.Linear(28 * 28, 10) def forward(self, x): return torch.relu(self.l1(x.view(x.size(0), -1))) def training_step(self, batch, batch_idx): x, y = batch y_hat = self(x) loss = F.cross_entropy(y_hat, y) tensorboard_logs = {"train_loss": loss} return {"loss": loss, "log": tensorboard_logs} def configure_optimizers(self): return torch.optim.Adam(self.parameters(), lr=0.001) def train_dataloader(self): dataset = MNIST( os.getcwd(), train=True, download=True, transform=transforms.ToTensor() ) loader = DataLoader(dataset, batch_size=32, num_workers=4, shuffle=True) return loader def main(hparams): logger = TensorBoardLogger(save_dir=os.getenv("HOME"), name="logs") net = LitModel(hparams) trainer = Trainer.from_argparse_args( hparams, logger=logger, checkpoint_callback=True, overfit_pct=0.01, ) trainer.fit(net) if __name__ == "__main__": parser = ArgumentParser() parser.add_argument("--sample", type=int, default=42, help="Sample Argument") hparams = parser.parse_args() main(hparams)Error
TypeError: cannot pickle '_thread.lock' objectExpected behavior
The model to be saved without considering the extra arguments sent as hparams
Environment
Additional context
Possible Fix
I believe that a possible fix is to change the from_argparse_args method
from
@classmethod def from_argparse_args(cls, args, **kwargs): params = vars(args) params.update(**kwargs) return cls(**params)to
@classmethod def from_argparse_args(cls, args, **kwargs): params = vars(args) return cls(**params, **kwargs)This ensures that the **kwargs are not added to the hparams of the model and the model gets saved successfully, but I'm not sure of the impact of this change
The text was updated successfully, but these errors were encountered: