Obtaining accurate uncertainty scores at inference/production

[vahuja4]

Is it possible to use BAAL just for the purpose of obtaining accurate uncertainty scores at the time of prediction, but not use the Active Learning loop? In other words, I don't want to do the back and forth between training and labelling, but only obtain accurate uncertainty scores. Also, in terms of methods, I can use only dropconnect because resnets don't have dropouts.

[Dref360]

Hello!

Yes it is often the case that we want to do uncertainty estimation at test time.

If you patch your Resnet with MCDropConnect, the model will be stochastic and you can do multiple forward pass per input. Then you can do the following:

1. Take the mean prediction, this is Bayesian Model Averaging
2. Compute BALD on the predictions, this will give you the epistemic uncertainty ie. the model's uncertainty.

This link should help you set everything up, if something is missing, let us know and we can improve the documentation.

https://baal.readthedocs.io/en/latest/faq.html#how-to-predict-uncertainty-per-sample-in-a-dataset

[vahuja4]

Thank you @Dref360! Based on the link you have posted, I can see that DropConnect can be simply applied at inference time, and we can then obtain the uncertainty score from the multiple predictions (on a single data point) using some heuristic. But if we use DropConnect only at the inference time, does it not mean that we are using a sub-par network during inference? Would it not lead to a drop in the model accuracy?

[Dref360]

Yes if you do not use enough iterations, you might see a small dip in accuracy.

With enough iterations, you will get similar accuracy, better calibration and you would have access to the uncertainty as well which is useful.

[vahuja4]

Got it @Dref360! One more clarification needed please: So, with dropconnect, we modify the network while keeping the datapoint the same and by doing multiple passes we obtain the uncertainty measure. How about we keep the network the same, and modify the data (test time augmentation) and then use some heuristic to obtain the uncertainty measure? Are these two techniques similar? Can you please share your insight? Thank you!

[Dref360]

From what I know, you would estimate the aleatoric uncertainty instead of the epistemic uncertainty.
Depends on your use case I guess, if you are doing OoD detection, computing aleatoric uncertainty might be useful, whereas if you want to know the model's uncertainty to detect particularly hard input, you would need to do Bayesian sampling.

Hopefully that helps!

This has been tried before here:
https://openaccess.thecvf.com/content_CVPRW_2020/papers/w42/Combalia_Uncertainty_Estimation_in_Deep_Neural_Networks_for_Dermoscopic_Image_Classification_CVPRW_2020_paper.pdf
https://openreview.net/pdf?id=rJZz-knjz
https://arxiv.org/abs/1807.07356