Integrate Vertex AI Models into GuardRails #402
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…text-bison models
…dling and added user guide
…VertexAI to langchain_google_vertexai.VertexAI due to deprecation
drazvan
requested changes
Mar 14, 2024
| ------ | ||
| ``` | ||
|
|
||
| Now we can test Vertex AI models with GuardRails. The `config` directory includes a variation of the ABC Bot but calling the Gemini 1.0 Pro model from Vertex AI. The main difference is in the `config.yml` file. The relevant section is |
There was a problem hiding this comment.
We should not duplicate the config in this folder. The notebook should copy the config when running and update the models part. The config folder itself should also be added to .gitignore so it doesn't get committed.
There was a problem hiding this comment.
Updated in commit 5319dbc5b89201e1e4d7046386183dbd7c3b4a12.
|
@trebedea : can you please review the changes to the evaluation part? Thanks! |
trebedea
approved these changes
Mar 15, 2024
drazvan
approved these changes
Mar 15, 2024
|
Thanks @aishwaryap ! I've made a few additional changes. Looks good 👍 ! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Update to PR #387 (which will be closed)
This PR provides support for calling Vertex AI models in GuardRails. Topical evaluation has been done with
gemini-1.0-proandtext-bisonmodels and self check evaluation for hallucination, fact checking and content moderation has been done withgemini-1.0-pro.Note that GoogleCloudPlatform/generative-ai#344 with Vertex AI models results in calls occasionally erroring out. This PR still contains some very general
try-catchblocks in hallucination and moderation evaluations to handle this. Errors are reported as a separate category in evaluation.