Test for valid YAML files

[mathemakitten]

Closes #296, which hopefully results in fewer broken Spaces. Nothing fancy about this implementation and it's pretty specific to Hub metric card formats but works just fine for what we need, let me know if you think it should do anything else!

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[lvwerra]

Thanks @mathemakitten!

My main suggestion: What do you think about moving the validation code to something like src/evaluate/utils/file_utils.py. This could be just a function that takes a string as input and raises an error if it's not valid yaml. Then we could test if that function works with a few examples as well as testing all the readmes. What do you think?

[lvwerra]

Is it not necessary to split off the YAML part of the README?

[mathemakitten]

The README is loaded in delimited by ---, so calling next() on the generator automatically gets the YAML (the first section)

[mathemakitten]

@lvwerra The bit which actually does the validation is next(x), which gets the first element from the loaded YAML (fine because for our metric cards the YAML is always at the top of the README, so it's the first element). If it isn't valid YAML then calling next() on the generator actually fails.

If we wanted to parse out the first bit of the file and pass it around to validate it'd look like this:

import re
REGEX_YAML_BLOCK = re.compile(r"---[\n\r]+([\S\s]*?)[\n\r]+---[\n\r]")
with open('metrics/perplexity/README.md', 'r') as f:
    x = f.read()
    match = REGEX_YAML_BLOCK.search(x)
y = yaml.safe_load_all(match.group())
assert type(y) == dict

However, this isn't any cleaner and also doesn't really generalize, since it depends on loading from the file in a specific way. We could have a general function which wraps yaml.safe_load_all for some piece of text, but I don't actually think that helps with code reuse much. Thoughts?

[lvwerra]

Fair enough, let's leave it like that then!

To solve the issue on windows, maybe you need to load with utf-8?

[lvwerra]

Awesome, thanks @mathemakitten 🚀