Pass preprocess_image function to generators in evaluate.py

[foghegehog]

I've encountered a problem similar to this issue: #647 (with custom backbone, mobilenetv3, not in the "zoo"). Here #1055 (comment) @mariaculman18 offered a solution that worked for me and seems to be reasonable. The author is not going to create a pull request, but I decided to do so, as other users with non-resnet50 backbones can be affected.

[codecov-io]

Codecov Report

Merging #1290 into master will increase coverage by <1%.
The diff coverage is 33%.

@@           Coverage Diff           @@
##           master   #1290    +/-   ##
=======================================
+ Coverage      23%     23%   +<1%     
=======================================
  Files          43      43            
  Lines        2535    2539     +4     
=======================================
+ Hits          576     577     +1     
- Misses       1959    1962     +3

[hgaiser]

This can also be passed to the constructor of generator. It does mean that the order of creating generators followed by loading of the model has to be the other way around, but I don't think this is an issue.

[hgaiser]

Thank you for the PR! I haven't gotten around to do this so it is much appreciated. Does the callback during training work properly? I could imagine it needs a similar treatment.

[foghegehog]

@hgaiser

Thank you for the PR! I haven't gotten around to do this so it is much appreciated. Does the callback during training work properly? I could imagine it needs a similar treatment.

Do you mean the make_shapes_callback and that it should be passed directly to generators' constructors for all models except resnet50, instead of this:

keras-retinanet/keras_retinanet/bin/train.py

Lines 494 to 498 in e27df4c

	# this lets the generator compute backbone layer shapes using the actual backbone model
	if 'vgg' in args.backbone or 'densenet' in args.backbone:
	train_generator.compute_shapes = make_shapes_callback(model)
	if validation_generator:
	validation_generator.compute_shapes = train_generator.compute_shapes

?

I'm not getting high mAP with my backbone during training, so it can indeed be an issue. But not for sure, as I'm just started and there can be other problems.

[hgaiser]

Ah it looks like preprocess_image is passed correctly. Are you getting different results with the evaluation callback versus running evaluate.py?

[foghegehog]

Ah it looks like preprocess_image is passed correctly. Are you getting different results with the evaluation callback versus running evaluate.py?

Yes I do, when running without the fix. I'm getting 20-30-40% mAP in the evaluation of the first epochs during training, but in the evaluate.py restored snapshots always give 0.

It looks reasonable to apply the same image preprocessing during evaluation as in the training process, however I can't get why resnet50 works without it. May be the 'tf' preprocessing mode for mobilenet backbone instead of 'caffe' for resnet causes that?

[hgaiser]

It looks reasonable to apply the same image preprocessing during evaluation as in the training process

Oh yeah definitely, was just an oversight that this wasn't handled properly.

I can't get why resnet50 works without it.

It's because the default value follows resnet50 style preprocessing.

I was asking because I thought you meant that, even with this fix, you get different results whether you evaluate during training or evaluate after training. I get now that this isn't the case for you. Then everything looks fine to me, thanks again for the PR.

[foghegehog]

@hgaiser
Could you please also explain about make_shapes_callback and code snippet from train.py I've mentioned above?
if 'vgg' in args.backbone or 'densenet' in args.backbone:

Should there be vgg and densenet only or other backbones as well, except the resnet50?

[hgaiser]

The shapes of the feature pyramid for resnet50 and probably some others are well defined, ie. you can predict how they will end up. For others, like vgg and densenet they were more difficult to predict so the decision was made to write a function that, given a model, extracts the sizes of the feature pyramid. I recall this from memory, so I could be wrong on some details.

[foghegehog]

The shapes of the feature pyramid for resnet50 and probably some others are well defined, ie. you can predict how they will end up. For others, like vgg and densenet they were more difficult to predict so the decision was made to write a function that, given a model, extracts the sizes of the feature pyramid. I recall this from memory, so I could be wrong on some details.

Wouldn't it be better to change the line:
if 'vgg' in args.backbone or 'densenet' in args.backbone:
with
if not args.backbone.startswith('resnet50):
?

As I see, there is only one guess_shape function. Is it designed specially for resnet50?

keras-retinanet/keras_retinanet/utils/anchors.py

Lines 186 to 198 in a81b313

	def guess_shapes(image_shape, pyramid_levels):
	"""Guess shapes based on pyramid levels.
	Args
	image_shape: The shape of the image.
	pyramid_levels: A list of what pyramid levels are used.
	Returns
	A list of image shapes at each pyramid level.
	"""
	image_shape = np.array(image_shape[:2])
	image_shapes = [(image_shape + 2 ** x - 1) // (2 ** x) for x in pyramid_levels]
	return image_shapes

[hgaiser]

I'm pretty sure other backbones (like mobilenet and efficientnet) work similarly as resnet50, so you would get some list of backbones there either way.

[foghegehog]

I'm pretty sure other backbones (like mobilenet and efficientnet) work similarly as resnet50, so you would get some list of backbones there either way.

Ok, thank you!