Support for Florence 2 model

[BrainSlugs83]

Feature request

When trying to export Florence 2, it fails with a bizarre error message that leads me to believe it's not supported.

D:\Redacted\>optimum-cli export onnx --model microsoft/Florence-2-large --trust-remote-code --framework pt flo2
D:\miniconda3\envs\onnx-export\Lib\site-packages\huggingface_hub\file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "D:\miniconda3\envs\onnx-export\Scripts\optimum-cli.exe\__main__.py", line 7, in <module>
  File "D:\miniconda3\envs\onnx-export\Lib\site-packages\optimum\commands\optimum_cli.py", line 163, in main
    service.run()
  File "D:\miniconda3\envs\onnx-export\Lib\site-packages\optimum\commands\export\onnx.py", line 265, in run
    main_export(
  File "D:\miniconda3\envs\onnx-export\Lib\site-packages\optimum\exporters\onnx\__main__.py", line 280, in main_export
    model = TasksManager.get_model_from_task(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\miniconda3\envs\onnx-export\Lib\site-packages\optimum\exporters\tasks.py", line 1950, in get_model_from_task
    model = model_class.from_pretrained(model_name_or_path, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\miniconda3\envs\onnx-export\Lib\site-packages\transformers\models\auto\auto_factory.py", line 566, in from_pretrained
    raise ValueError(
ValueError: Unrecognized configuration class <class 'transformers_modules.microsoft.Florence-2-large.ef29c9b007f906bd278c39bc12ae620398d88c88.configuration_florence2.Florence2Config'> for this kind of AutoModel: AutoModelForVision2Seq.
Model type should be one of BlipConfig, Blip2Config, GitConfig, Idefics2Config, InstructBlipConfig, Kosmos2Config, LlavaConfig, LlavaNextConfig, PaliGemmaConfig, Pix2StructConfig, VideoLlavaConfig, VipLlavaConfig, VisionEncoderDecoderConfig.

Motivation

I want to export an onnx model of Florence 2. I kind of thought that's the type of thing this tool is used for, yeah? Take a non-onnx repo from hugging face, and export an onnx model.

Your contribution

Not really. I'm willing to test if there's something y'all want me to try?

[qingfengcss]

+1

[dragen1860]

anyone give some tips on how to export?

[amaye15]

I've refactored the DaVit part of Florence to be compatible with Huggingface, if this helps

https://huggingface.co/amaye15/DaViT-Florence-2-large-ft

[dragen1860]

I've refactored the DaVit part of Florence to be compatible with Huggingface, if this helps

https://huggingface.co/amaye15/DaViT-Florence-2-large-ft

So do you succed to export onnx?

[amaye15]

Here's the code to export the vision tower (DaVit) to onnx.

Link

[dragen1860]

I've refactored the DaVit part of Florence to be compatible with Huggingface, if this helps

https://huggingface.co/amaye15/DaViT-Florence-2-large-ft

hi， thank for your reply.
Did you export the language model of florenc2？

[amaye15]

Not yet, might have a look at that this weekend.