Perplexity with nltk and TorchMetrics

[d-kleine]

With reference to the given example in this notebook, would it be possible to demonstrate how perplexity can be calculated with

• nltk

There is also perplexity implemented in TorchMetrics, but it seems to take the log probabilities and ground truth values, different to the example you have provided.

[rasbt]

Good suggestions! As far as I know, NLTK doesn't implement perplexity functions ... Torchmetrics does though. I added an example and the results are pretty close. I wonder if the discrepancy is due to rounding error.

[d-kleine]

Thank you, got it! 👍

One thing I am not happy about is that the values for "fast" add up to 108%:

[0.0, 0.0, 0.0, 0.0, 0.0, 0.2, 0.02, 0.01, 0.00, 0.85, 0.00, 0.00]

I think it would be better to set one value to 0.15 to the one of 0.85, like

[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.15, 0.0, 0.85, 0.0, 0.0]

or set the values more likely to a normal distribution like in reality, like

[0.01, 0.02, 0.0, 0.0, 0.0, 0.02, 0.00, 0.1, 0.0, 0.85, 0.0, 0.0]

just to make it more clearer that probabilities need to add up to 100%.

Furthermore, what do you think about the torch.log2() instead of torch.log() question (please see above)?

[d-kleine]

@rasbt Could you please take second look into the code?

[rasbt]

Thanks for the reminder @d-kleine , I totally missed your follow-up! I went over the code examples again and fixed all the input inconsistencies (also added a check this time to ensure everything sums to 1).

The Torchmetrics example should now agree to the from scratch implementation and the PyTorch implementation (added this one in addition to the previous ones).

At the bottom, there is also a natural log and log-2 example. They should give the same results.

[d-kleine]

Thank you very much - great work, looks fine now! 👍 😃

From my point of view, the TorchMetrics example has significantly improved the content, I now have a much better understanding how perplexity can be calculated when implementing in practice! 🏆

[rasbt]

Awesome, glad to hear! And thanks so much for the feedback suggesting these fixes and additions, I really appreciate it!