How can I generate heatmap without category index

[curefate]

Hi, all

I want to use cam to generate heatmaps, from a model that produces a feature vector instead of a classification. Can I do that by cam?
This is a bit like what is done in https://openaccess.thecvf.com/content/CVPR2021/html/Selvaraju_CASTing_Your_Model_Learning_To_Localize_Improves_Self-Supervised_Representations_CVPR_2021_paper.html. They apply gradcam in MoCo, with key and query images. I don't have pair images so I can't just copy their method.
Sorry I didn't understand their code until now, if possible, can anyone give me any advice, thanks a lot !

[frgfm]

Hey @curefate 👋

Interesting paper, thanks for sharing! Could you be more specific about the part you're asking about in the paper?
This library is meant to accommodate as many CAM methods as possible in a flexible way. In most of the papers, the CAM is queried with a model + an input + class index. This class index can be informed by the model output (top1 class index for instance) or by other things similar to what the paper seems to be diving into.

I'd need to spend more time on the paper to fully understand it, but it looks like on Figure 3, they're using the class activation map on conv5 of the GT class. If that's the case, they're indeed passing the class index

Let me know your thoughts!

[curefate]

I am trying to use cam to help update the encoder of the network in vision-based reinforcement learning. And you are right, I misunderstood the approach in the paper, they did use class index, same as gradcam in this library. Maybe I can try to add a classification head to the encoder.

Anyway, sorry for wasting your time, I will close this disscussion. 😃

[frgfm]

No worries at all, always good to challenge the approach :)
If we can make this library useful to more people & usecases, I'm all in!