Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FEAT Add AzureOpenAIGPTVChatTarget to Support MultiModal #160

Conversation

rdheekonda
Copy link
Contributor

@rdheekonda rdheekonda commented Apr 22, 2024

Description

  • Added a new AzureOpenAIGPTVChatTarget class supports multimodal input (text and image) and generates textual output.

  • Created a demo notebook to demonstrate how to run AzureOpenAIGPTVChatTarget.

  • Implemented functionality to retrieve headers from the environment file and could pass them to the AzureOpenAI clients.

  • Added a new class ChatMessageListContent for GPTV target and tests

  • Added an end to end multimodal demo using AzureOpenAIGPTVChatTarget and PromptSendingOrchestrator in 3_send_all_prompts notebook

  • Fixed two bugs: validate function in normalizer_request.py module and set same conversation ID across all prompt request pieces in PromptRequestResponse object

  • Moving the chat message to its own module.

  • Adjusted the validation method in PromptRequestResponse to handle chat conversation alternates, fixing existing test cases.

Tests and Documentation

  • Added unit tests and jupytext modules.

@rdheekonda rdheekonda changed the title Users/rdheekonda/add multimodal support to aoai chat target [DRAFT] FEAT Add multimodal support to AzureOpenAIChatTarget Apr 22, 2024
.env_example Outdated Show resolved Hide resolved
doc/code/targets/prompt_targets.py Outdated Show resolved Hide resolved
doc/code/targets/prompt_targets.py Outdated Show resolved Hide resolved
pyrit/memory/memory_chat_message_builder.py Outdated Show resolved Hide resolved
pyrit/memory/memory_chat_message_builder.py Outdated Show resolved Hide resolved
pyrit/models/models.py Outdated Show resolved Hide resolved
doc/code/targets/prompt_targets.py Outdated Show resolved Hide resolved
@rdheekonda rdheekonda marked this pull request as ready for review April 25, 2024 08:55
@rdheekonda rdheekonda changed the title [DRAFT] FEAT Add multimodal support to AzureOpenAIChatTarget [DRAFT] FEAT Add AzureOpenAIGPTVChatTarget to support MultiModal Apr 25, 2024
@rdheekonda rdheekonda changed the title [DRAFT] FEAT Add AzureOpenAIGPTVChatTarget to Support MultiModal FEAT Add AzureOpenAIGPTVChatTarget to Support MultiModal Apr 25, 2024
doc/code/targets/gpt_v_target.py Show resolved Hide resolved
pyrit/models/models.py Outdated Show resolved Hide resolved
Copy link
Contributor

@rlundeen2 rlundeen2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only thing in my review I feel most strongly about is the testing comments, but everything is looking great. I'd love to see most of these changes but no blockers

@rdheekonda rdheekonda merged commit e46d5df into Azure:main Apr 26, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants