add trainer integration docs #325
Conversation
|
The documentation is not available anymore as the PR was closed or merged. |
There was a problem hiding this comment.
Overall looks good! I left a couple of suggestions. I would also love @NimaBoscarino's feedback on this
docs/source/metric_integrations.mdx
Outdated
|
|
||
| ### Seq2SeqTrainer | ||
|
|
||
| For sequence-to-sequence tasks such as translation or summarization the [`~transformers.Seq2SeqTrainer`] is useful. It allows for the use of the generate method when setting `predict_with_generate=True`. That means we evaluate generated text within the `compute_metric` function, we just need to decode the predictions and labels first. |
There was a problem hiding this comment.
It was a bit hard to understand this paragraph with the eyes of a new reader. Why is the predict_with_generate needed? What would be used if not?
There was a problem hiding this comment.
I tried to elaborate a bit more.
docs/source/metric_integrations.mdx
Outdated
|
|
||
| This guide outlines how `evaluate` metrics can be used with popular machine learning frameworks. | ||
|
|
||
| ## 🤗 Transformers |
There was a problem hiding this comment.
WDYT Of having separate pages for each library + a general one? Similar to https://huggingface.co/docs/hub/transformers. Then anyone could come and create a new page for their own libraries.
docs/source/metric_integrations.mdx
Outdated
| compute_metrics=compute_metrics, | ||
| ) | ||
|
|
||
| trainer.train() |
There was a problem hiding this comment.
FYI training a model with this setup in Colab + GPU seems to take some time, maybe a lighter version could be useful here as this is just to showcase
There was a problem hiding this comment.
When I tried it, it took about 5min or so. You think that's too long?
There was a problem hiding this comment.
5 min is ok, although maybe even faster is nice, otherwise a % of your audience will just go away for a thing not vital for the guide
There was a problem hiding this comment.
should now be ~1min to download datasets and ~1min to train for both examples.
docs/source/metric_integrations.mdx
Outdated
| trainer.train() | ||
| ``` | ||
|
|
||
| You can use any `evaluate` metric with the `Trainer` and `Seq2SeqTrainer` as long as they are compatible with the task and predictions. |
There was a problem hiding this comment.
Maybe also worth adding an example of running evaluation in an existing model on the Hub
There was a problem hiding this comment.
I added a comment on this. Essentially replacing trainer.train to trainer.evaluate.
|
Nothing to add that hasn't been said already! |
Co-authored-by: helen <31600291+mathemakitten@users.noreply.github.com> Co-authored-by: Omar Sanseviero <osanseviero@gmail.com>
* add trainer integration docs * fix transformers links * adapt headings * add installation instructions * Apply suggestions from code review Co-authored-by: helen <31600291+mathemakitten@users.noreply.github.com> Co-authored-by: Omar Sanseviero <osanseviero@gmail.com> * integrate review feedbacks * move toc * take smaller sample * fix toctree Co-authored-by: helen <31600291+mathemakitten@users.noreply.github.com> Co-authored-by: Omar Sanseviero <osanseviero@gmail.com>
Second commit for adding issue number.
* Added Keras example for Evaluate metrics Created a markdown file to show how to use Evaluate when training and evaluate a model using Keras and Tensorflow. * Added example for using Evaluate with Keras (#325) Second commit for adding issue number. * Apply fixes from code review Fixes included some rephrasing, changing to use keras logs instead of print, and focusing on evaluating on the test set. Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com> * Added extra sentence to clarify rounding and fixed typo Typo from last commit accidently broke a code box. * Added ref in toctree and accuracy in Keras Docs Fixed incorrect ref addition to toctree, and added a comment describing the acc score of the example in the keras documentation. Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com>
This PR adds an example of how to use
evaluatewith the 🤗 TransformersTrainerclasses.cc @osanseviero