Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for using 'langchain serve' endpoints #588

Merged
merged 18 commits into from
Apr 26, 2024

Conversation

GustavFredrikson
Copy link
Contributor

@GustavFredrikson GustavFredrikson commented Apr 9, 2024

  • New LangChainServeLLMGenerator Class: Implements HTTP POST request handling to interact with LangChain Serve LLM APIs, allowing integration with Garak probes.

Copy link
Contributor

github-actions bot commented Apr 9, 2024

DCO Assistant Lite bot All contributors have signed the DCO ✍️ ✅

@GustavFredrikson
Copy link
Contributor Author

I have read the DCO Document and I hereby sign the DCO

@GustavFredrikson
Copy link
Contributor Author

recheck

github-actions bot added a commit that referenced this pull request Apr 9, 2024
@GustavFredrikson GustavFredrikson changed the title Added support for using langchain serve endpoints Support for using 'langchain serve' endpoints Apr 9, 2024
@leondz
Copy link
Owner

leondz commented Apr 9, 2024

Thanks for this! We'll get it reviewed

@leondz leondz added generators Interfaces with LLMs new plugin Describes an entirely new probe, detector, generator or harness labels Apr 9, 2024
Copy link
Owner

@leondz leondz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, looking pretty good

  • We need to fix how model_name is used
  • Config variables need to be handled cleaning
  • Exception handling can get clarified a bit - interested in your feedback re: intended behavior with this generator & when it might make sense to backoff, skip, or abort
  • What do you think we can do in regard to tests?

garak/generators/langchain_serve.py Outdated Show resolved Hide resolved
garak/generators/langchain_serve.py Outdated Show resolved Hide resolved
@GustavFredrikson
Copy link
Contributor Author

  • Implemented URI validation to ensure the API endpoint is correctly formatted before attempting to connect.
  • Replaced model_name usage with an environment variable (LANGCHAIN_SERVE_URI) for setting the API endpoint, let me know if other usage is preferred!
  • Error handling:
    • Logging for different types of HTTP errors (4xx and 5xx).
    • Changed return values from empty strings to None.
  • Removed unused config params i.e. temperature and penalties that are managed server-side.
  • Added tests covering URI validation, initialization, mocked generation, and error handling scenarios using requests_mock.

@leondz leondz self-requested a review April 18, 2024 17:20
Copy link
Collaborator

@jmartin-tech jmartin-tech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor thought and I think this will be ready to go!

garak/generators/langchain_serve.py Outdated Show resolved Hide resolved
garak/generators/langchain_serve.py Outdated Show resolved Hide resolved
garak/generators/langchain_serve.py Outdated Show resolved Hide resolved
leondz and others added 6 commits April 26, 2024 10:13
Co-authored-by: Jeffrey Martin <jmartin@Op3n4M3.dev>
Signed-off-by: Leon Derczynski <leonderczynski@gmail.com>
Co-authored-by: Jeffrey Martin <jmartin@Op3n4M3.dev>
Signed-off-by: Leon Derczynski <leonderczynski@gmail.com>
Co-authored-by: Jeffrey Martin <jmartin@Op3n4M3.dev>
Signed-off-by: Leon Derczynski <leonderczynski@gmail.com>
Signed-off-by: Leon Derczynski <leonderczynski@gmail.com>
@leondz leondz merged commit 6455c86 into leondz:main Apr 26, 2024
4 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Apr 26, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
generators Interfaces with LLMs new plugin Describes an entirely new probe, detector, generator or harness
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants