Perplexity

[assaftibm]

No description provided.

[codecov]

Codecov Report

Attention: 4 lines in your changes are missing coverage. Please review.

Comparison is base (12cac48) 93.96% compared to head (56f33f9) 94.01%.

Files	Patch %	Lines
src/unitxt/metrics.py	94.93%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #442      +/-   ##
==========================================
+ Coverage   93.96%   94.01%   +0.04%     
==========================================
  Files         179      180       +1     
  Lines        7574     7699     +125     
==========================================
+ Hits         7117     7238     +121     
- Misses        457      461       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[assaftibm]

The metric computes the likelihood of generating the text Y after the text X (i.e. P(Y|X))
There is no clear alignment between X and Y, and the notions of reference and prediction as used in Unitxt.
After consulting with Yosi Mass, we think it makes sense to treat them this way:

• X = prediction
• Y = reference.

[yoavkatz]

I think you also need to commit the json files.

[assaftibm]

I think you also need to commit the json files.

I just pushed them now

[yoavkatz]

Perhaps use this naming convention:

instead of
src/unitxt/catalog/metrics/perplexity/flan/t5/small.json

use
src/unitxt/catalog/metrics/perplexity/flan_t5_small.json

So there will be less directories.

[assaftibm]

fixed