Get Image Embedding Vector and Store it in DATABASE

[cepa995]

Hi,

I am trying to get image embeddings from SAM and save them in a Database, but I am a bit confused with the embedding size.

For example:

I see that size is [1, 256, 64, 64]. This is because of the batch size, correct? The embedding should be (1, 256), right?

How can I safely retrieve the correct embedding vector?

[raoxinyu4977]

The input image has an original size of (1, 3, 1024, 1024). After passing through the VIT encoder (patch_size=16, out_channels=256), the output shape is (1, 256, 1024/16, 1024/16), which then reshapes to (1, 64*64, 256) during subsequent self-attention processing.

[cepa995]

Aha, I see. So it is not typical image encoding into a single vector, such as CLIP does, but a matrix instead.

Does that mean that there is no use in using these image encodings to compute similarity between two images?

[heyoeyo]

Does that mean that there is no use in using these image encodings to compute similarity between two images?

The embeddings are always going to be the same shape, so if you wanted to treat them as a single vector to make comparison easier, they could always be combined. For example just by stacking them, averaging, taking the max of each feature etc. or maybe even building a model that merges them into a single vector for a specific use case.

Alternatively, the comparison could be made in 2D, to get a measure of spatial similarity between two images.