Added metric Brier Score

[kadirnar]

I added Brier Score metric. #235

[kadirnar]

Hi @lvwerra,

Can you share the reformatter code?

[lvwerra]

Hi @kadirnar, thanks for that awesome contribution! If you are inside the repo (main folder) you can run make style && make quality. Let me know if you run into any issue!

[kadirnar]

make style && make quality

Thank you for your support. I am running vscode application as administrator.
Run Code:

make style && make quality

151 files left unchanged.
isort tests src metrics measurements
WARNING: Unable to parse file tests due to [Errno 13] Permission denied: 'C:\\Users\\kadir\\Desktop\\github\\evaluate\\tests'
WARNING: Unable to parse file src due to [Errno 13] Permission denied: 'C:\\Users\\kadir\\Desktop\\github\\evaluate\\src'
WARNING: Unable to parse file metrics due to [Errno 13] Permission denied: 'C:\\Users\\kadir\\Desktop\\github\\evaluate\\metrics'
WARNING: Unable to parse file measurements due to [Errno 13] Permission denied: 'C:\\Users\\kadir\\Desktop\\github\\evaluate\\measurements'
black --check --line-length 119 --target-version py36 tests src metrics comparisons measurements
All done! ✨ 🍰 ✨
151 files would be left unchanged.
isort --check-only tests src metrics measurements
WARNING: Unable to parse file tests due to [Errno 13] Permission denied: 'C:\\Users\\kadir\\Desktop\\github\\evaluate\\tests'
WARNING: Unable to parse file src due to [Errno 13] Permission denied: 'C:\\Users\\kadir\\Desktop\\github\\evaluate\\src'
WARNING: Unable to parse file metrics due to [Errno 13] Permission denied: 'C:\\Users\\kadir\\Desktop\\github\\evaluate\\metrics'
WARNING: Unable to parse file measurements due to [Errno 13] Permission denied: 'C:\\Users\\kadir\\Desktop\\github\\evaluate\\measurements'
flake8 tests src metrics
metrics\brier_score\brier_score.py:47:1: W293 blank line contains whitespace
metrics\brier_score\brier_score.py:50:1: W293 blank line contains whitespace
metrics\brier_score\brier_score.py:53:1: W293 blank line contains whitespace
metrics\brier_score\brier_score.py:61:1: W293 blank line contains whitespace
metrics\brier_score\brier_score.py:63:1: W293 blank line contains whitespace
make: *** [Makefile:8: quality] Error 1

The codes in the tokenizer_13a.py file have also changed

[lvwerra]

Seems to be an issue of isort and permissions on Windows (which I don't have access to): PyCQA/isort#835

Which version are you using? Can you run:

pip install .[dev]

to make sure all the right versions for black and isort are installed?

[lvwerra]

I added a few fixes that should hopefully make the tests pass.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[kadirnar]

Hi @lvwerra ,
I updated the codes. It works when I test it with the sklearn library. Sample codes of the sklearn library:

    >>> import numpy as np
    >>> from sklearn.metrics import brier_score_loss
    >>> y_true = np.array([0, 1, 1, 0])
    >>> y_prob = np.array([0.1, 0.9, 0.8, 0.3])
    >>> brier_score_loss(y_true, y_prob)

Why doesn't it pass the tests?

[lvwerra]

Added two suggestions that should fix the issues.

[lvwerra]

Side comment: if there are code suggestions you can just click on "Commit suggestion" so you don't need to do an extra commit locally.

[kadirnar]

Side comment: if there are code suggestions you can just click on "Commit suggestion" so you don't need to do an extra commit locally.

"Commit suggestion

Side comment: if there are code suggestions you can just click on "Commit suggestion" so you don't need to do an extra commit locally.

I didn't know that. Thank you for your advice. I will try.

[lvwerra]

Thanks again for working on this! Left a few minor nits and then I think we can merge!

[lvwerra]

Left one more suggestion to fix the YAML in the README.

[kadirnar]

Left one more suggestion to fix the YAML in the README.

I didn't know how to fix this error. Thank you very much for your help.

[lvwerra]

Thanks a lot, looks good now! I'll approve and merge it after we merge another bug fix.