Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add concurency_limit parameter to WMLInferenceEngine #1231

Merged
merged 2 commits into from
Sep 24, 2024
Merged

Add concurency_limit parameter to WMLInferenceEngine #1231

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6947d00
Add concurency_limit parameter to WMLInferenceEngine
elronbandel Sep 24, 2024
943f169
update
allyssonf Sep 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion src/unitxt/inference.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -559,6 +559,7 @@ class WMLInferenceEngine(
parameters (WMLInferenceEngineParams, optional): Instance of WMLInferenceEngineParams
which defines inference parameters and their values. Deprecated attribute, please
pass respective parameters directly to the WMLInferenceEngine class instead.
concurrency_limit (int): number of requests that will be sent in parallel, max is 10.

Examples:
from .api import load_dataset
Expand Down Expand Up @@ -592,7 +593,7 @@ class WMLInferenceEngine(
}
data_classification_policy = ["public", "proprietary"]
parameters: Optional[WMLInferenceEngineParams] = None

concurrency_limit: int = 10
_client: Any = InternalField(default=None, name="WML client")

def verify(self):
Expand Down Expand Up @@ -666,6 +667,7 @@ def _infer(self, dataset):
return model.generate_text(
prompt=dataset["source"],
params=self.to_dict([WMLInferenceEngineParamsMixin], keep_empty=False),
concurrency_limit=self.concurrency_limit,
)


Expand Down
Loading