-
Notifications
You must be signed in to change notification settings - Fork 39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support execution of metrics on a remote host #568
Conversation
Signed-off-by: Assaf Toledo <assaf.toledo@ibm.com>
Signed-off-by: Assaf Toledo <assaf.toledo@ibm.com>
Signed-off-by: Assaf Toledo <assaf.toledo@ibm.com>
Signed-off-by: Assaf Toledo <assaf.toledo@ibm.com>
…is first, and the reference (e.g. the question) is second
Signed-off-by: Assaf Toledo <assaf.toledo@ibm.com>
# Conflicts: # src/unitxt/test_utils/metrics.py
…/metrics.py update the metric service api
…service. only return the computed scores, avoid returning the inputs in the MetricResponse.
Discussed an alternative solution suggested by @elronbandel : |
change service data classes base class to the unitxt Dataclass
update corresponding imports, and paths in Dockerfile
fix use of relative imports
# Conflicts: # src/unitxt/eval_utils.py # src/unitxt/metric_utils.py # src/unitxt/metrics.py # src/unitxt/test_utils/metrics.py
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We have a mechanism to add and read new environment variables in 'settings_utils.py' I think it's better if you use it. It gives us more control over the settings and allow to review all the settings in one place. It also log new settings changes and have some more nice features.
…vice update main.py and Dockerfile accordingly move the service requirements to service.rqr, and use it in Dockerfile
…vice add transformers as a service requirement
# Conflicts: # requirements/base.rqr
MetricPipeline
objects at this point).New requirements: