Support for phi3-v Vision Model

[saaraahfar]

Feature request

I encountered a KeyError while loading the phi3-v vision model into Optimum Huggingface. The error message states:

KeyError: 'phi3-v model type is not supported yet in NormalizedConfig. Only albert, bart, bert, blenderbot, blenderbot-small, bloom, falcon, camembert, codegen, cvt, deberta, deberta-v2, deit, distilbert, donut-swin, electra, encoder-decoder, gemma, gpt2, gpt-bigcode, gpt-neo, gpt-neox, gptj, imagegpt, llama, longt5, marian, markuplm, mbart, mistral, mixtral, mpnet, mpt, mt5, m2m-100, nystromformer, opt, pegasus, pix2struct, phi, phi3, phi3small, poolformer, regnet, resnet, roberta, speech-to-text, splinter, t5, trocr, vision-encoder-decoder, vit, whisper, xlm-roberta, yolos, qwen2 are supported. If you want to support phi3-v please propose a PR or open up an issue.'

Could you please add support for the phi3-v model?

Motivation

I think it would be beneficial to have a vision model included in the supported list.

Your contribution

---

[fxmarty]

Hi @saaraahfar, if using e.g. https://huggingface.co/microsoft/Phi-3-vision-128k-instruct/tree/main, the ONNX export is not supported in Optimum as it uses custom modeling code. However, we could support it similar to #1874