attempting to let meteor handle multiple references per prediction

[sashavor]

I'm not sure I'm doing the word_tokenize() correctly in line 27 -- @lvwerra can you please help?

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[sashavor]

Thank you @lvwerra !
One thing I can't seem to replicate is the example from NLTK:

    >>> reference1 = ['It', 'is', 'a', 'guide', 'to', 'action', 'that', 'ensures', 'that', 'the', 'military', 'will', 'forever', 'heed', 'Party', 'commands']
    >>> reference2 = ['It', 'is', 'the', 'guiding', 'principle', 'which', 'guarantees', 'the', 'military', 'forces', 'always', 'being', 'under', 'the', 'command', 'of', 'the', 'Party']
    >>> reference3 = ['It', 'is', 'the', 'practical', 'guide', 'for', 'the', 'army', 'always', 'to', 'heed', 'the', 'directions', 'of', 'the', 'party']

    >>> round(meteor_score([reference1, reference2, reference3], hypothesis1),4)
    0.7398

When I do:

hypothesis1=  ['It is a guide to action which ensures that the military always obeys the commands of the party']
>>> references = [['It is a guide to action that ensures that the military will forever heed Party commands', 'It is the guiding principle which guarantees the military forces always being under the command of the Party', 'It is the practical guide for the army always to heed the directions of the party']]
>>> results = meteor.compute(predictions=hypothesis1, references=references)
>>> results
{'meteor': 0.6944444444444445}

The alpha, beta and gamma are all the same as NLTK.

Do you have any ideas? do you think it's the tokenizer?...

[lvwerra]

If you execute the first NLTK example yourself you get the same results? Or are they also different?

[sashavor]

Nope 😕

>>> hypothesis1 = ['It is a guide to action which ensures that the military always obeys the commands of the party']
>>> reference1 = ['It is a guide to action that ensures that the military will forever heed Party commands']
>>> results = meteor.compute(predictions=hypothesis1, references=reference1)
>>> results
{'meteor': 0.6944444444444445}

Whereas the result given by NLTK is 0.7398

This is weird, right? since we're using it under the hood?

[lvwerra]

I just checked the NLTK example. It seems to be an issue with the version on their side: They switched from sentences to word tokens between 3.6 and 3.7 which is where the change must have happened:

with nltk==3.6.0 (note we need to join the examples):

hypothesis1 = ['It', 'is', 'a', 'guide', 'to', 'action', 'which', 'ensures', 'that', 'the', 'military', 'always', 'obeys', 'the', 'commands', 'of', 'the', 'party']

reference1 = ['It', 'is', 'a', 'guide', 'to', 'action', 'that', 'ensures', 'that', 'the', 'military', 'will', 'forever', 'heed', 'Party', 'commands']
reference2 = ['It', 'is', 'the', 'guiding', 'principle', 'which', 'guarantees', 'the', 'military', 'forces', 'always', 'being', 'under', 'the', 'command', 'of', 'the', 'Party']
reference3 = ['It', 'is', 'the', 'practical', 'guide', 'for', 'the', 'army', 'always', 'to', 'heed', 'the', 'directions', 'of', 'the', 'party']

round(meteor_score([" ".join(reference1), " ".join(reference2), " ".join(reference3)], " ".join(hypothesis1)),4)

>>> 0.7298

in nltk==3.7.0:

hypothesis1 = ['It', 'is', 'a', 'guide', 'to', 'action', 'which', 'ensures', 'that', 'the', 'military', 'always', 'obeys', 'the', 'commands', 'of', 'the', 'party']

reference1 = ['It', 'is', 'a', 'guide', 'to', 'action', 'that', 'ensures', 'that', 'the', 'military', 'will', 'forever', 'heed', 'Party', 'commands']
reference2 = ['It', 'is', 'the', 'guiding', 'principle', 'which', 'guarantees', 'the', 'military', 'forces', 'always', 'being', 'under', 'the', 'command', 'of', 'the', 'Party']
reference3 = ['It', 'is', 'the', 'practical', 'guide', 'for', 'the', 'army', 'always', 'to', 'heed', 'the', 'directions', 'of', 'the', 'party']

round(meteor_score([reference1, reference2, reference3], hypothesis1),4)

>>> 0.6944

So I would not worry about it in this PR but maybe open an issue on NLTK.

[sashavor]

Ok great! I'll open a PR with NLTK then 😄

[lvwerra]

Thanks a lot for working on this! A few minor comments. In addition, I think we should also document this in the README, right?

[lvwerra]

same as above. maybe you can also just check at the beginning once:

multiple_refs = isinstance(references[0], list)

[sashavor]

I made the change (I think) -- is this what you had in mind?

[sashavor]

Also updated the README!

[lvwerra]

Just a minor comment about reusing multiple_refs. LGTM 🚀