genai: fix pydantic structured_output with array

[nobu007]

PR Description

Fix pydantic structured_output with array

Relevant issues

#24225
langchain-ai/langchain#24225

Type

🐛 Bug Fix

Testing(optional)

This PR can pass like here.

class TestClass(BaseModel):
    """_summary_

    Args:
        BaseModel (_type_): _description_
    """
    test_val: str = Field(default="", description="value for test(works).")
    test_val_list: List[str] = Field(
        default_factory=list,
        description="values for test(not works).",
    )

Test result

jinno@jinno-desktop:~/git/drill/gamebook/codeinterpreter_api_agent/langchain-google/libs/genai$ poetry run pytest --extended tests/integration_tests/test_chat_models.py::test_chat_vertexai_gemini_function_calling
============================================================== test session starts ===============================================================
platform linux -- Python 3.10.10, pytest-7.4.4, pluggy-1.5.0
rootdir: /home/jinno/git/drill/gamebook/codeinterpreter_api_agent/langchain-google/libs/genai
configfile: pyproject.toml
plugins: anyio-4.4.0
collected 1 item                                                                                                                                 

tests/integration_tests/test_chat_models.py .                                                                                              [100%]

================================================================ warnings summary ================================================================
../../../../../../../.cache/pypoetry/virtualenvs/langchain-google-genai-ov3b6nnP-py3.10/lib/python3.10/site-packages/_pytest/config/__init__.py:1373
  /home/jinno/.cache/pypoetry/virtualenvs/langchain-google-genai-ov3b6nnP-py3.10/lib/python3.10/site-packages/_pytest/config/__init__.py:1373: PytestConfigWarning: Unknown config option: asyncio_mode
  
    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

Note1

This code shoud be updated later.
_set_schema_items() can't create correct function-calling scema.
https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/function-calling?hl#schema

Note2

This PR may be fix #492.
integration_test filed by this issue.

[lkuligin]

could you add a unit test too, please?

[nobu007]

@lkuligin
I added condition for unittest and integtest.
Please confirm it.

poetry run pytest --extended tests/integration_tests/test_chat_models.py::test_chat_vertexai_gemini_function_calling

[lkuligin]

can we maybe just have a one-liner here instead creating a separate function?
TYPE_ENUM.get(stype, "str")

[nobu007]

@lkuligin
I fixed as one-liner.

[TannerW]

Any progress on this?

[nobu007]

@lkuligin
#506 will fix integration_test.
So I think #469 can be merged.
It looks randomly failed. I retried but failed again.
If passing integration_test is needed, please merge #506 first.

[nobu007]

@lkuligin
integration_test was caused by #469, but it is fixed now.

[tudoanh]

Encountered this issue too

[lkuligin]

nits: we probably don't need it anymore, do we?

[RafaelMCarvalho]

Thanks for the contribution.

I've noticed we still have issues when there's an array like this:

class Person(BaseModel):
    name: str = Field(..., description="The name of the person")
    height_in_meters: float = Field(
        ..., description="The height of the person expressed in meters."
    )

class People(BaseModel):
    people: List[Person]

So the following code raised some spec ignoring warning and the result leads to invalid Pydantic input:

llm = ChatGoogleGenerativeAI(model='gemini-1.5-pro', api_key=settings.google_api_key, temperature=0.2)
query = "Anna is 23 years old and she is 6 feet tall; John is 34 and about 171cm"
prompt = ChatPromptTemplate.from_messages([("system", "Answer the user query."), ("human", "{query}") ])
chain = prompt | llm.with_structured_output(People, include_raw=True)
# WARNING OUTPUT:
# Value 'Information about a person.' is not supported in schema, ignoring v=Information about a person.
# Value '['name', 'height_in_meters']' is not supported in schema, ignoring v=['name', 'height_in_meters']
# Value 'Person' is not supported in schema, ignoring v=Person
# Value 'object' is not supported in schema, ignoring v=object
# Value 'Information about a person.' is not supported in schema, ignoring v=Information about a person.
# Value '['name', 'height_in_meters']' is not supported in schema, ignoring v=['name', 'height_in_meters']
# Value 'Person' is not supported in schema, ignoring v=Person
# Value 'object' is not supported in schema, ignoring v=object

chain.invoke({"query": query})
# OUTPUT:
# {'name': 'People', 'parameters': {'type_': 6, 'properties': {'people': {'type_': 5, 'items': {'type_': 1, 'format_': '', 'description': '', 'nullable': False, 'enum': [], 'max_items': '0', 'min_items': '0', 'properties': {}, 'required': []}, 'format_': '', 'description': '', 'nullable': False, 'enum': [], 'max_items': '0', 'min_items': '0', 'properties': {}, 'required': []}}, 'required': ['people'], 'format_': '', 'description': '', 'nullable': False, 'enum': [], 'max_items': '0', 'min_items': '0'}, 'description': ''}
# {'raw': AIMessage(content='', additional_kwargs={'function_call': {'name': 'People', 'arguments': '{"people": ["Anna", "John"]}'}}, response_metadata={'prompt_feedback': {'block_reason': 0, 'safety_ratings': []}, 'finish_reason': 'STOP', 'safety_ratings': [{'category': 'HARM_CATEGORY_DANGEROUS_CONTENT', 'probability': 'NEGLIGIBLE', 'blocked': False}, {'category': 'HARM_CATEGORY_HATE_SPEECH', 'probability': 'NEGLIGIBLE', 'blocked': False}, {'category': 'HARM_CATEGORY_HARASSMENT', 'probability': 'NEGLIGIBLE', 'blocked': False}, {'category': 'HARM_CATEGORY_SEXUALLY_EXPLICIT', 'probability': 'NEGLIGIBLE', 'blocked': False}]}, id='run-d4f260cb-1ec2-49a9-befc-e99b8ba235de-0', tool_calls=[{'name': 'People', 'args': {'people': ['Anna', 'John']}, 'id': 'df592d4a-a4a7-48b3-a212-983a97e100d1', 'type': 'tool_call'}], usage_metadata={'input_tokens': 65, 'output_tokens': 16, 'total_tokens': 81}), 'parsing_error': 2 validation errors for People
# people.0
#   Input should be a valid dictionary or instance of Person [type=model_type, input_value='Anna', input_type=str]
#     For further information visit https://errors.pydantic.dev/2.8/v/model_type
# people.1
#   Input should be a valid dictionary or instance of Person [type=model_type, input_value='John', input_type=str]
#     For further information visit https://errors.pydantic.dev/2.8/v/model_type, 'parsed': None}

Is it a Gemini limitation that I'm missing?