Use a multimodal LLM to extract the likely Product name and brand as JSON

[teolemon]

Problem

• We get very little product names using the official app, because it is cumbersome to input

Proposed solution

• Use a multimodal LLM like Gemini to extract the likely Product name and brand as JSON, if and only if:
• the user requests it (with a button ?)
• no product name input has been made 2hrs later (in that case, create a detection and insight to be applied)

Additional context

• The size of the opportunity is around 100K products: https://world.openfoodfacts.org/state/product-name-to-be-completed/state/front-photo-selected
• The opportunity also lies in a better 1st time experience for users

Mockups

Can you provide the likely product name, and the brand as a JSON output ?
Yes, the product in the image is likely Pedigree Dentastix Chewy Chunks Small/Medium Chicken Flavor Dog Treats.

Here is the information in JSON format:

{
  "brand": "Pedigree",
  "product name": "Dentastix Chewy Chunks Small/Medium Chicken Flavor Dog Treats"
}

Part of

• Leverage Generative AI across Open Food Facts (tracker) #289