Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support execution of metrics on a remote host #568

Merged
merged 125 commits into from
Mar 4, 2024
Merged

Support execution of metrics on a remote host #568

merged 125 commits into from
Mar 4, 2024

Conversation

matanor
Copy link
Member

@matanor matanor commented Feb 11, 2024

  • A metrics http-based service (under service/metrics), optionally supports token-based authentication.
  • A new dockerfile and relevant commands in makefile which exemplify how to containerize a unitxt metrics service.
  • The evaluate(..) tool supports usage of remote metrics (only MetricPipeline objects at this point).
  • The remote metrics client configuration is loaded from environment variables.

New requirements:

  • httpretty - for mocking http servers, in tests

assaftibm and others added 30 commits January 15, 2024 10:40
Signed-off-by: Assaf Toledo <assaf.toledo@ibm.com>
Signed-off-by: Assaf Toledo <assaf.toledo@ibm.com>
Signed-off-by: Assaf Toledo <assaf.toledo@ibm.com>
Signed-off-by: Assaf Toledo <assaf.toledo@ibm.com>
…is first, and the reference (e.g. the question) is second
Signed-off-by: Assaf Toledo <assaf.toledo@ibm.com>
Signed-off-by: Assaf Toledo <assaf.toledo@ibm.com>
# Conflicts:
#	src/unitxt/test_utils/metrics.py
…service.

only return the computed scores, avoid returning the inputs in the MetricResponse.
@matanor
Copy link
Member Author

matanor commented Feb 18, 2024

@elronbandel re moving the new pydantic dependency from base.rqr to a new service.rqr file --

The thing is that pydantic.BaseModel is used as a base class for two new classed in metric_utils (here and here). So it is imported in the module level here. If i move the dependency, then this module becomes unusable, right?

If this module was only used within a function, i would have just made a local import. Do you know if there is any similar alternative for the above case?

Discussed an alternative solution suggested by @elronbandel :
We will not use pydantic at all. The MetricRequest and MetricResponse classes will inherit from unitxt DataClass. Then will use to_dict() to convert the request into json (with json.dumps too). In the other direction, will simply construct the MetricResonse from the json response using the default DataClass constructor.

Copy link
Member

@elronbandel elronbandel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a mechanism to add and read new environment variables in 'settings_utils.py' I think it's better if you use it. It gives us more control over the settings and allow to review all the settings in one place. It also log new settings changes and have some more nice features.

…vice

update main.py and Dockerfile accordingly
move the service requirements to service.rqr, and use it in Dockerfile
…vice

add transformers as a service requirement
@matanor matanor requested a review from elronbandel March 4, 2024 07:58
@elronbandel elronbandel merged commit f3745a9 into main Mar 4, 2024
4 of 7 checks passed
@elronbandel elronbandel deleted the service branch April 8, 2024 17:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants