resolve #379 audio classification evaluator + docs #405
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. |
|
Looks awesome. Would you mind running |
|
Thanks a lot @Plutone11011 the PR looks very clean! I added @lewtun as a reviewer for the Audio specific part since he suggested this evaluator. |
There was a problem hiding this comment.
Thanks a lot for adding this very clean implementation @Plutone11011 ! Overall it looks great - my only question is whether we should enable support for raw audio data since that would provide parity with the audio-classification pipeline
Would you like to extend your PR to support that?
| >>> model_or_pipeline=""superb/wav2vec2-base-superb-ks"", | ||
| >>> data=data, | ||
| >>> label_column="label", | ||
| >>> input_column="file", |
There was a problem hiding this comment.
Since the audio-classification pipeline supports inference on both raw waveforms and audio files, would it make sense to enable support for both in evaluation?
In other words, input_column could be either an audio column of type np.ndarray or a file column of type str.
Also, if I'm not mistaken, using audio files requires ffmpeg to be installed - perhaps we should add a note somewhere in the example?
There was a problem hiding this comment.
Ok, I'll add support for this.
Do you think it makes sense to also add a test for the dataset with audio column?
| ```python | ||
| >>> from evaluate import evaluator | ||
| >>> from datasets import load_dataset | ||
| >>> task_evaluator = evaluator("audio-classification") |
There was a problem hiding this comment.
Nit: let's add a space between the imports and code
| >>> task_evaluator = evaluator("audio-classification") | |
| >>> task_evaluator = evaluator("audio-classification") |
| Audio classification evaluator. | ||
| This audio classification evaluator can currently be loaded from [`evaluator`] using the default task name | ||
| `audio-classification`. | ||
| Methods in this class assume a data format compatible with the [`AudioClassificationPipeline`]. |
There was a problem hiding this comment.
Nit:
| Methods in this class assume a data format compatible with the [`AudioClassificationPipeline`]. | |
| Methods in this class assume a data format compatible with the [`transformers.AudioClassificationPipeline`]. |
| n_resamples: int = 9999, | ||
| device: int = None, | ||
| random_state: Optional[int] = None, | ||
| input_column: str = "file", |
There was a problem hiding this comment.
See comment about about extending this to support raw waveforms in an audio column
fixed typo in berscore readme
…ace#411) Added max_length kwarg to docstring of Perplexity measurement
|
@lewtun I made the changes. In the docs example I added a map operation to rearrange the dataset to be consistent with what the evaluator and pipeline expect, since usually audio datasets have an Audio column that looks like this EDIT: |
…gface#405) * audio classification evaluator + docs * fix styling issue * [Docs] fixed a typo in bertscore readme (huggingface#386) fixed typo in berscore readme * Add max_length kwarg to docstring of Perplexity measurement (huggingface#411) Added max_length kwarg to docstring of Perplexity measurement * adding support raw audio input audio classification evaluator --------- Co-authored-by: Hazrul Akmal <hazrulakmal121@gmail.com> Co-authored-by: Kalyan Dutia <kalyan.dutia@gmail.com>
This is the PR addressing #379.
For testing, I used the audio file highlighted in the Pipelines API example, mainly because the superb dataset requires manual download, which I felt was more cumbersome and less aligned with the other evaluators' tests.