Evaluator device placement #193
Conversation
|
The documentation is not available anymore as the PR was closed or merged. |
There was a problem hiding this comment.
Thanks for adding this killer feature @lvwerra - finally evaluations can go brum brum brum 🏎️ !
I've left a few nits and a question about the device variable being set when neither torch or tf are installed (edge case). Otherwise, this LGTM!
| except ImportError: | ||
| device = -1 | ||
|
|
||
| if device == -1: |
There was a problem hiding this comment.
In the rare case that neither torch or tensorflow are installed, this will error out because device is not assigned a value. Would it make sense to have a warning in that case?
There was a problem hiding this comment.
Ah sorry I missed that - all good then!
| # neither pt or tf are available | ||
| pt_available = False | ||
| tf_available = False | ||
| self.assertEqual(Evaluator._infer_device(), -1) |
There was a problem hiding this comment.
Interesting, I wonder how this test passes since my understanding is that device is never explicitly defined when neither torch or tf are installed 🤔
There was a problem hiding this comment.
If neither tf or torch are installed then the second except sets it to -1:
evaluate/src/evaluate/evaluator/base.py
Lines 149 to 150 in b48e48c
Co-authored-by: lewtun <lewis.c.tunstall@gmail.com>
This PR ads device placement for the pipeline in the evaluator. There are two mechanisms:
Evaluator.compute.Alternatively, a user can always use a custom configuration and initialize the pipeline before passing it to the
evaluator. If the an initialized pipeline is passed the device has no effect.