Docs for EvaluationSuite #340
Conversation
|
The documentation is not available anymore as the PR was closed or merged. |
mathemakitten
force-pushed
the
hn-evaluation-suite-v2
branch
3 times, most recently
from
e64cd83
to
f8da2a6
Compare
docs/source/evaluation_suite.mdx
Outdated
|
|
||
| ```python | ||
| {'glue/cola': {'accuracy': 0.0, 'total_time_in_seconds': 0.9766696180449799, 'samples_per_second': 10.238876909079256, 'latency_in_seconds': 0.09766696180449798}, 'glue/sst2': {'accuracy': 0.5, 'total_time_in_seconds': 1.1422595420153812, 'samples_per_second': 8.754577775166744, 'latency_in_seconds': 0.11422595420153811}, 'glue/qqp': {'accuracy': 0.6, 'total_time_in_seconds': 1.3553926559980027, 'samples_per_second': 7.377935800188323, 'latency_in_seconds': 0.13553926559980026}, 'glue/mrpc': {'accuracy': 0.6, 'total_time_in_seconds': 2.021696529001929, 'samples_per_second': 4.946340786832532, 'latency_in_seconds': 0.2021696529001929}, 'glue/mnli': {'accuracy': 0.2, 'total_time_in_seconds': 2.0380110969999805, 'samples_per_second': 4.9067446270142145, 'latency_in_seconds': 0.20380110969999807}, 'glue/qnli': {'accuracy': 0.3, 'total_time_in_seconds': 2.082032073987648, 'samples_per_second': 4.802999975330509, 'latency_in_seconds': 0.20820320739876477}, 'glue/rte': {'accuracy': 0.7, 'total_time_in_seconds': 2.8592985830036923, 'samples_per_second': 3.4973612267855576, 'latency_in_seconds': 0.2859298583003692}, 'glue/wnli': {'accuracy': 0.5, 'total_time_in_seconds': 1.5406486629508436, 'samples_per_second': 6.490772517107661, 'latency_in_seconds': 0.15406486629508437}} | ||
| ``` |
There was a problem hiding this comment.
(nit) Would be nice to show it as a pandas DataFrame for readability
There was a problem hiding this comment.
Good call, the result is now a list of dicts so it can be easily transformed into a dataframe. I've added that to the example 😄
| self.preprocessor = lambda x: {"text": x["text"].lower()} | ||
| self.suite = [ | ||
| SubTask( | ||
| task_type="text-classification", |
There was a problem hiding this comment.
Can you list the available task types maybe ? Or redirect to their docs ?
There was a problem hiding this comment.
I've added a link to the supported tasks on the Evaluator docs so we don't have to maintain the list in two places!
There was a problem hiding this comment.
Hi @mathemakitten this great, thanks for working on this. I left a few comments, happy to discuss further if you want.
Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com>
Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com>
Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com>
mathemakitten
force-pushed
the
hn-docs-evalsuite
branch
from
0527ff4
to
7da77ed
Compare
| >>> suite = EvaluationSuite.load('mathemakitten/glue-evaluation-suite') | ||
| >>> results = suite.run("gpt2") | ||
|
|
||
| | accuracy | total_time_in_seconds | samples_per_second | latency_in_seconds | task_name | |
There was a problem hiding this comment.
Would do the same here and remove the table from the codeblock so it's actually rendered as a nice table.
Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com>
Adding docs for
EvaluationSuite.