[Feature Request] Add support for image data in prompt logging

[luca-medeiros]

Introduction:

With the introduction of the gpt4v API, it becomes increasingly important to expand the capabilities of the library to accommodate other modality data in prompt logging.

Rationale:

Including image data in prompts can significantly enhance the tracking ability, enabling users to evaluate text-based responses based on visual input too. At first glance, I thought about adding image_path/s3 uri to the metadata but would be awesome if we could see the image on the UI directly.

Feasibility Assessment:

Would like to request feedback from the development team regarding the feasibility of implementing this feature. Are there any technical challenges or considerations I should be aware of?

Contribution Interest:

I'm eager to contribute to this feature's development. If the team finds it feasible, could you provide pointers on where to start contributing? Any guidance or documentation would be greatly appreciated.

[jverre]

Hi Luca,
Great suggestion !

Last week we pushed a new version of the SDK that integrates with v1 of the OpenAI SDK. We should now be auto logging the link to the image automatically.

Having the image render in the UI would be a good improvement. Since this repo is just for the SDK code, it will be something we have to build internally so I've raised it with the team.

We can keep you updated on our progress here and also let you know if there are other features you could contribute to.

[luca-medeiros]

Hey @jverre would love to know when this gets implemented. (would migrate all code from smith to comet)

We should now be auto logging the link to the image automatically.

where is the image hosted?

[Danselem]

Hi @jverre,

Still looking forward to the feature.

I'm also interested in knowing if one can implement comet ml log_image function into comet llm log_prompt.

Let me know what you think.