Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

✨ Ability to use Amazon Bedrock models such as Claude 3.5 Sonnet #289

Merged
merged 2 commits into from
Aug 10, 2024
Merged

✨ Ability to use Amazon Bedrock models such as Claude 3.5 Sonnet #289

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b446946
Ability to use Amazon Bedrock models such as Claude 3.5 Sonnet
jwmatthews Aug 9, 2024
4e17b85
Fix so ran: pip-compile --allow-unsafe
jwmatthews Aug 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions kai/config.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,16 @@ password = "dog8code"
# model = "gpt-3.5-turbo"
# ```

# **Amazon Bedrock served Anthropic Claude 3.5 Sonnet **
# ```
# [models]
# provider = "ChatBedrock"

# [models.args]
# model_id = "anthropic.claude-3-5-sonnet-20240620-v1:0"
# ```


[models]
provider = "ChatIBMGenAI"

Expand Down
12 changes: 11 additions & 1 deletion kai/service/llm_interfacing/model_provider.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
from genai import Client, Credentials
from genai.extensions.langchain.chat_llm import LangChainChatInterface
from genai.schema import DecodingMethod
from langchain_community.chat_models import ChatOllama, ChatOpenAI
from langchain_community.chat_models import BedrockChat, ChatOllama, ChatOpenAI
from langchain_community.chat_models.fake import FakeListChatModel
from langchain_core.language_models.chat_models import BaseChatModel
from pydantic.v1.utils import deep_update
Expand Down Expand Up @@ -78,6 +78,16 @@ def __init__(self, config: KaiConfigModels):
model_args = deep_update(defaults, config.args)
model_id = model_args["model_id"]

case "ChatBedrock":
model_class = BedrockChat

defaults = {
"model_id": "anthropic.claude-3-5-sonnet-20240620-v1:0",
}

model_args = deep_update(defaults, config.args)
model_id = model_args["model_id"]

case "FakeListChatModel":
model_class = FakeListChatModel

Expand Down
3 changes: 3 additions & 0 deletions requirements.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@ vcrpy @ git+https://github.com/fabianvf/vcrpy.git@httpx-async-threadpool
python-dotenv==1.0.1
pyyaml==6.0.1

# --- To allow Amazon Bedrock to work ---
boto3==1.34.157

# --- Testing dependencies ---
coverage==7.6.0

Expand Down
24 changes: 20 additions & 4 deletions requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ anyio==4.4.0
# httpx
# jupyter-server
# openai
appnope==0.1.4
# via ipykernel
argon2-cffi==23.1.0
# via jupyter-server
argon2-cffi-bindings==21.2.0
Expand Down Expand Up @@ -49,6 +51,12 @@ beautifulsoup4==4.12.3
# via nbconvert
bleach==6.1.0
# via nbconvert
boto3==1.34.157
# via -r requirements.in
botocore==1.34.157
# via
# boto3
# s3transfer
certifi==2024.7.4
# via
# httpcore
Expand All @@ -64,6 +72,8 @@ click==8.1.7
# typer
comm==0.2.2
# via ipykernel
coverage==7.6.0
# via -r requirements.in
dataclasses-json==0.6.7
# via langchain-community
debugpy==1.8.2
Expand Down Expand Up @@ -97,8 +107,6 @@ gitdb==4.0.11
# via gitpython
gitpython==3.1.43
# via -r requirements.in
greenlet==3.0.3
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wonder why this is no longer necessary

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm unsure why greenlet was removed, I don't think its from this PR.

I ran pip-compile on current main (not this PR) and I see greenlit is being removed

$ pip-compile --allow-unsafe

$ git diff
diff --git a/requirements.txt b/requirements.txt
index 1cb26bc..683b395 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -20,6 +20,8 @@ anyio==4.4.0
     #   httpx
     #   jupyter-server
     #   openai
+appnope==0.1.4
+    # via ipykernel
 argon2-cffi==23.1.0
     # via jupyter-server
 argon2-cffi-bindings==21.2.0
@@ -64,6 +66,8 @@ click==8.1.7
     #   typer
 comm==0.2.2
     # via ipykernel
+coverage==7.6.0
+    # via -r requirements.in
 dataclasses-json==0.6.7
     # via langchain-community
 debugpy==1.8.2
@@ -97,8 +101,6 @@ gitdb==4.0.11
     # via gitpython
 gitpython==3.1.43
     # via -r requirements.in
-greenlet==3.0.3
-    # via sqlalchemy
 gunicorn==22.0.0
     # via -r requirements.in
 h11==0.14.0

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not answering your question directly, but it is supposed to be an optional dependency:
sqlalchemy/sqlalchemy#10197

I can't install it without greenlet even after removing langchain and langchain-community, which appear to be the only two things requiring it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As long as everything still works I'm good with it, just pointing it out

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://docs.sqlalchemy.org/en/20/faq/installation.html

The greenlet dependency does not install by default for CPU architectures for which greenlet does not supply a [pre-built binary wheel](https://pypi.org/project/greenlet/#files). Notably, this includes Apple M1. To install including greenlet, add the asyncio [setuptools extra](https://packaging.python.org/en/latest/tutorials/installing-packages/#installing-setuptools-extras) to the pip install command:

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pretty sure the above explains it.

Copy link
Member

@jmontleon jmontleon Aug 9, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I update sqlalchemy to 2.0.32 in the requirements.in it looks like it is recognized as optional and unnecessary on Fedora rawhide / python 3.13 as well; not having any effect on Fedora 40 / 3.12

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And 2.0.32 should be fine to satisfy langchain and langchain-community:

https://github.com/langchain-ai/langchain/blob/langchain%3D%3D0.2.11/libs/langchain/pyproject.toml#L40

https://github.com/langchain-ai/langchain/blob/langchain-community%3D%3D0.2.10/libs/community/pyproject.toml#L35

# via sqlalchemy
gunicorn==22.0.0
# via -r requirements.in
h11==0.14.0
Expand Down Expand Up @@ -149,6 +157,10 @@ jinja2==3.1.4
# jupyterlab
# jupyterlab-server
# nbconvert
jmespath==1.0.1
# via
# boto3
# botocore
json5==0.9.25
# via jupyterlab-server
jsonpatch==1.33
Expand Down Expand Up @@ -352,6 +364,7 @@ python-dateutil==2.8.2
# via
# -r requirements.in
# arrow
# botocore
# jupyter-client
python-dotenv==1.0.1
# via
Expand Down Expand Up @@ -406,6 +419,8 @@ rpds-py==0.19.1
# via
# jsonschema
# referencing
s3transfer==0.10.2
# via boto3
send2trash==1.8.3
# via jupyter-server
six==1.16.0
Expand Down Expand Up @@ -493,6 +508,7 @@ uri-template==1.3.0
# via jsonschema
urllib3==2.2.2
# via
# botocore
# requests
# vcrpy
vcrpy @ git+https://github.com/fabianvf/vcrpy.git@httpx-async-threadpool
Expand Down Expand Up @@ -521,7 +537,7 @@ yarl==1.9.4
# vcrpy

# The following packages are considered to be unsafe in a requirements file:
pip==24.1.2
pip==24.2
# via ghapi
setuptools==71.1.0
setuptools==72.1.0
# via jupyterlab
Loading