Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ImageInferenceEngine can only consume a very specific ImageModel class #802

Open
corcra opened this issue Feb 17, 2023 · 1 comment
Open
Labels
enhancement New feature or request hi-ml-multimodal Issues related to the hi-ml-multimodal package

Comments

@corcra
Copy link
Member

corcra commented Feb 17, 2023

I want to use the ImageInferenceEngine from hi-ml-multimodal with an image model that can't cleanly be fit into the existing ImageModel class.

I'd like to propose something like:

  • We define a more abstract class for ImageModels that just requires the methods actually used by ImageInferenceEngine / ImageTextInferenceEngine (from what I can see, this is forward, get_patchwise_projected_embeddings, and the training attribute). The inference engine can then just expect something from this class as an input.
  • The existing ImageModel class can subclass this, so no expected change to current behaviour.
  • I can write a custom subclass for my special model and be able to use it in the inference engine.
@fepegar fepegar added enhancement New feature or request hi-ml-multimodal Issues related to the hi-ml-multimodal package labels Feb 17, 2023
@fepegar
Copy link
Contributor

fepegar commented Feb 17, 2023

Sounds good!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request hi-ml-multimodal Issues related to the hi-ml-multimodal package
Projects
None yet
Development

No branches or pull requests

2 participants