dataframes passed outside hparams causing issues

[s-rog]

🐛 Bug

In 0.8.0 (updating from 0.7.5), using hyperparameters parsed by test-tube causes an exception.

In 0.8.0 (updating from 0.7.5), dataframes passed outside hparams errors out

To Reproduce

import argparse
import pandas as pd
import pytorch_lightning as pl

parser = argparse.ArgumentParser()
parser.add_argument('--hparam1', default=8, type=int)
parser.add_argument('--hparam2', default="test", type=str)
parser.add_argument('--hparam3', default=True, type=bool)
hparams = parser.parse_args()

some_data = pd.DataFrame([1, 2, 3])

class module(pl.LightningModule):
    def __init__(self, hparams, some_data):
        super(module, self).__init__()
        self.hparams = hparams
        self.data = some_data

    def train_dataloader(self):
        return

    def val_dataloader(self):
        return

    def configure_optimizers(self):
        return
    
    def forward(self, x):
        return x
    
    def training_step(self, batch, batch_idx):
        return batch

    def validation_step(self, batch, batch_idx):
        return batch

model = module(hparams, some_data)

Traceback (most recent call last):
  File "train_test.py", line 37, in <module>
    model = module(hparams, some_data)
  File "train_test.py", line 16, in __init__
    self.hparams = hparams
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 638, in __setattr__
    object.__setattr__(self, name, value)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/core/lightning.py", line 1695, in hparams
    self.save_hyperparameters(hp, frame=inspect.currentframe().f_back.f_back)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/core/lightning.py", line 1662, in save_hyperparameters
    cand_names = [k for k, v in init_args.items() if v == hp]
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/core/lightning.py", line 1662, in <listcomp>
    cand_names = [k for k, v in init_args.items() if v == hp]
  File "/opt/conda/lib/python3.6/site-packages/pandas/core/generic.py", line 1479, in __nonzero__
    f"The truth value of a {type(self).__name__} is ambiguous. "
ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

Expected behavior

hparams is saved correctly as previous versions, disregarding the dataframes

[williamFalcon]

ummm wait... TTNamespace is a Namespace... this is weird.

can you post minimal code to reproduce?

[s-rog]

Yeah, I swapped over to pure argparse to test and my hparams is <class 'argparse.Namespace'> but it doesn't seem to evaluate correctly... let me compile it

[williamFalcon]

argparse is 100% supported. we have explicit tests for this

[s-rog]

import argparse
import pandas as pd
import pytorch_lightning as pl

parser = argparse.ArgumentParser()
parser.add_argument('--hparam1', default=8, type=int)
parser.add_argument('--hparam2', default="test", type=str)
parser.add_argument('--hparam3', default=True, type=bool)
hparams = parser.parse_args()

some_data = pd.DataFrame([1, 2, 3])

class module(pl.LightningModule):
    def __init__(self, hparams, some_data):
        super(module, self).__init__()
        self.hparams = hparams
        self.data = some_data

    def train_dataloader(self):
        return

    def val_dataloader(self):
        return

    def configure_optimizers(self):
        return
    
    def forward(self, x):
        return x
    
    def training_step(self, batch, batch_idx):
        return batch

    def validation_step(self, batch, batch_idx):
        return batch

model = module(hparams, some_data)

ok I just tested it, it's caused by dataframes passed outside hparams

Edit: updated the original post to reflect the true source of the bug

[williamFalcon]

got it.
Found the bug!

@Borda

         self.save_hyperparameters(hp, frame=inspect.currentframe().f_back.f_back)

we shouldn't be doing frame inspection here...

[s-rog]

thanks for the quick response, I'll pull from the fork and give it a shot

[williamFalcon]

fixed!

https://colab.research.google.com/drive/1jO8iyCyfC1BiJbjAdqcQxc7Qclil5hok#scrollTo=kppSZVsHRPtJ