EarlyStopper cannot deal with complex Metrics

[priba]

🚀 Feature Request

EarlyStopper and ThresholdStopper cannot deal with complex Metrics such as MAP.

See:

composer/composer/callbacks/early_stopper.py

Line 122 in 8d4dfaf

if not torch.is_tensor(metric_val):

composer/composer/callbacks/threshold_stopper.py

Line 100 in 8d4dfaf

if not torch.is_tensor(metric_val):

This line should only convert the metric to a tensor if it is a numeric value.
For complex metrics, we should either rely on self.comp_func and some distance metric or allow some way to extract the desired value in self.monitor. Otherwise it will also crash here when checking the absolute distance.

composer/composer/callbacks/early_stopper.py

Line 128 in 8d4dfaf

elif self.comp_func(metric_val, self.best) and torch.abs(metric_val - self.best) > self.min_delta:

Motivation

Metrics for object detection come as a dictionary. See MAP.

[mvpatel2000]

Apolgoies for the delayed response. We are tracking this issue, but it is unfortunately not in our immediate roadmap. We welcome community PRs on this subject :)