Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added subclass of OpenAI class to support custom endpoints #560

Merged

Conversation

shannonsands
Copy link
Contributor

@shannonsands shannonsands commented Jan 19, 2024

Adds a new class to support for providing a custom base_url to the OpenAI client, and a couple of small updates to remove the checks on the model via the "retrieve" api (since it's not always available), sets tiktoken to use gpt4 by default (with the option to override it), and enables passing temperature as a named parameter to the __call__method to override the config.

Usage is basically the same as the regular OpenAI model wrapper, with the addition of the base_url parameter. Eg:

hermes = CustomOpenAI(
        model_name="NousResearch/Nous-Hermes-2-Mixtral-8x7B-DPO",
        system_prompt=system_prompt,
        api_key="xxx",
        base_url='https://api.together.xyz',
 )

print(hermes(test_input))

The environment variables checked to set the API key was also updated, so as not to conflict with any OpenAI configurations in place, and sound a little more generic, along with a new env var INFERENCE_BASE_URL to act as an alternative to the constructor parameter.

@rlouf rlouf merged commit 6c6966c into outlines-dev:main Jan 20, 2024
5 checks passed
@rlouf
Copy link
Member

rlouf commented Jan 20, 2024

I think it would be nice eventually to have users pass an OpenAIClient themselves, which I think should simplify the codebase?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants