Supercharge the app with artificial intelligence (Tracker)

[teolemon]

What

Tasks

Beta Give feedback

1. Simplify the basic details input by using Robotoff predictions #3279
   Product addition ✏️ Editing ✨ enhancement 🎨 Mockups available 🎯 P1 🐛 bug 🤖 Robotoff +7
2. We can add Robotoff questions filtered on labels in the labels editing section #3986
   ✏️ Editing ✨ enhancement 🎯 P1 🤖 Robotoff +4
   
3. We can add Robotoff questions filtered on category in the category editing section #3985
   ✏️ Editing ✨ enhancement 🎯 P1 🤖 Robotoff +4
4. We can add Robotoff questions filtered on brands in the basic details editing section #4069
   ✏️ Editing ✏️ Editing - Basic info input ✨ enhancement 🎯 P1 +4
5. Inline Robotoff questions to increase the answer rate #3326
   ✨ enhancement 🎯 P0 🤖 Robotoff +3
   
6. Instrument Robotoff answers in Matomo #3775
   ✨ enhancement 📈 Analytics 📈 Matomo +3
   
7. Offer in-line validation of Robotoff predictions for nutrition facts #3278
   ✏️ Editing - Nutrition input ✨ enhancement 🎨 Mockups available 🎯 P1 🤖 Robotoff 🤖 blocked on Robotoff +6
8. Random Robotoff insight UX suggestions #3340
   Hunger Games ✨ enhancement 🤖 Robotoff +3
   
9. Ensure Robotoff questions works offline (and local answers are dropped in the right cases after reconnection) #2457
   Offline Offline - Contribution ✨ enhancement 🤖 Robotoff +4
10. Add optional offline ingredient extraction using MLKIT (disabled on the F-Droid build)
11. Add offline packaging extraction using MLKIT (disabled on the F-Droid build)
12. Explore the feasability of bringing OCR analysis done by Robotoff offline
13. Explore the feasability of bringing nutrition analysis done by Robotoff offline
14. Explore the feasability of bringing category prediction done by Robotoff offline
15. Explore segmentation of photos on device #5007
    On-device ML ✨ enhancement +2
16. Offer on-device translation of products using MLKit
17. Use automatic language recognition to suggest additional languages to add, or suggest correcting product language before sending #5006
    On-device ML ✨ enhancement +2
18. Bring minimum on-device moderation using MLKit #5008
    On-device ML ✨ enhancement +2
19. Add custom fields to the nutrition input form based on the category (eg for waters, chocolate, alcohol) #5401
    ✏️ Editing - Nutrition input ✨ enhancement +2
20. Attempt getting Robotoff predictions during product addition #5402
    Product addition ✨ enhancement 🤖 Robotoff +3
21. Add an optional button on the product page to jump to Hunger Games crops and logos view for the product
22. Use on-device classification to detect the type of product (Food, Cosmetic, PetFood, Products, or more granular) and show a warning
23. Automatic warning when adding an old receipt with today's date #5387
    On-device ML Prices ✨ enhancement 🤳 MLKit +4
24. Towards a try harder scan mode ? #5322
    On-device ML 🤳 MLKit +2
25. Create a custom Tensorflow model for MLKit to classify Nutrition facts, Packaging instruction, Front and Ingredients photo on device #5010
    On-device ML ✨ enhancement 🤳 MLKit +3
26. Use the DUMP API to make an offline Hunger Games ? https://openfoodfacts.github.io/robotoff/references/api/#tag/Insights/paths/~1insights~1dump/get

Part of

• 🎯 What can I work on ? #525

[teolemon]

@raphael0202 I added the idea to do on-device language recognition here

[teolemon]

What

Create a custom Tensorflow model for MLKit to classify Nutrition facts, Packaging instruction, Front and Ingredients photo on device
Create a custom Tensorflow model for MLKit to classify food, pet food, cosmetics and other products on device

How

https://developers.google.com/ml-kit/vision/image-labeling?hl=fr
https://developers.google.com/ml-kit/vision/face-detection?hl=fr

[teolemon]

Explore using the on-device LLM for various tasks

• Food intake journaling
• Ingredient analysis
• FAQ/Support (chat?)
• other use-cases TBD

How

• https://ai.google/build/project-guides/llm-on-android-with-keras-and-tensorflow-lite
• https://android-developers.googleblog.com/2023/12/leverage-generative-ai-in-your-android-apps.html
• https://developer.android.com/studio/preview/gemini-template