TensorFlow SavedModel: export failure: Message third_party.py.keras.protobuf.SavedMetadata exceeds maximum protobuf size of 2GB: 2283672738

[glenn-jocher]

Search before asking

• I have searched the YOLOv5 issues and found no similar bug report.

YOLOv5 Component

Export

Bug

YOLOv5x SavedModel format is over a 2GB TF limit:

TensorFlow SavedModel: export failure: Message third_party.py.keras.protobuf.SavedMetadata exceeds maximum protobuf size of 2GB: 2283672738

First noted in #251 (comment)

@zldrobit could you help on this?

Environment

Colab Notebook
https://colab.research.google.com/github/ultralytics/yolov5/blob/master/tutorial.ipynb

Minimal Reproducible Example

In Colab notebook after running Setup cell:
https://colab.research.google.com/github/ultralytics/yolov5/blob/master/tutorial.ipynb

!python export.py --weights yolov5x.pt --img 640 --include saved_model

Additional

See tensorflow/tensorflow#45041 for the official TF issue

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[zldrobit]

@glenn-jocher I can confirm the problem with your provided notebook. I also found some links about the 2GB size limit: https://stackoverflow.com/a/42906762/3036450, tensorflow/tensorflow#4291 and https://stackoverflow.com/questions/59558170/tensorflow-graph-bigger-than-2gb. According to those links, we should not break the 2GB limit of protobuf. One way to get a saved model smaller than 2GB is wrap the yolov5x pb model (~350MB) in a saved model format.

EDIT:
The exported pb model is 90% smaller than the saved_model.

[zldrobit]

@glenn-jocher I sent a PR to address this issue, but it breaks backward compatibility for saved_model export. Here's a notebook for validation.