AveragePrecision Broken on Master

[peblair]

🐛 Bug

On master (currently 4857546), the implementation of __hash__ on PyTorch Lightning metrics causes an exception to be raised for metric implementations which make use of non-hashable state. One such exception is AveragePrecision, meaning that any model which uses it is currently broken on master.

Please reproduce using the BoringModel

https://colab.research.google.com/drive/1sSMuMoLyeOEtNupTOPC7YuLQonVxBt4H?usp=sharing

To Reproduce

See above BoringModel. All that's needed to reproduce the bug is installing master (currently 4857546) and adding an AveragePrecision metric to a model. This exception is raised (copied from above Colab link):

/usr/local/lib/python3.6/dist-packages/pytorch_lightning/metrics/metric.py in __hash__(self)
    336             hash_vals.append(getattr(self, key))
    337 
--> 338         return hash(tuple(hash_vals))
    339 
    340     def __add__(self, other: Any):

TypeError: unhashable type: 'list'

Expected behavior

Not crashing 🙂

Environment

• CUDA:
  • GPU:
    • Tesla T4
  • available: True
  • version: 10.1
• Packages:
  • numpy: 1.19.5
  • pyTorch_debug: True
  • pyTorch_version: 1.7.0+cu101
  • pytorch-lightning: 1.2.0dev
  • tqdm: 4.41.1
• System:
  • OS: Linux
  • architecture:
    • 64bit
  • processor: x86_64
  • python: 3.6.9
  • version: Proposal for help #1 SMP Thu Jul 23 08:00:38 PDT 2020

Additional context

[peblair]

Now, the "correct" solution here would be to migrate all metrics using non-hashable state to use hashable alternatives. I have an alternative solution on this branch, which circumvents the issue by allowing lists to be used as state, so long as the contents of those lists are hashable. I leave the decision whether to merge this or go down another route up to the maintainers, but let me know if you would like me to open a Pull Request with this branch's change.

[SkafteNicki]

@peblair thanks for catching this bug. Please feel free to open a PR with your suggested solution. As metric states that are list should only contain torch.Tensor they should always be hasable.