Allow saving weights of a very deep model into a HDF5 file.

[tunystom]

I encountered a problem while saving weights of very deep Keras models into HDF5 file. The culprit is the HDF5 file format's inability to save object headers larger than 64 kB (according to https://support.hdfgroup.org/HDF5/faq/limits.html).

This pull request should solve the following issues (without the need to implement any custom saving/loading methods as proposed by some comments)

• #6766
• #5253

I also answered to a question to a related issue on StackOverflow:

• Callbackfunction modelcheckpoint causes error in keras

The fix is really simple. If the data array (layer_names and weight_names specifically) being saved to HDF5 group is too large, it is simply chunked into several pieces (until they fit the memory limit) and saved individually under the original attribute name with an associated chunk number appended to it. The loading from a HDF5 file is implemented correspondingly.

[fchollet]

Please add a unit test that covers the case where header chunking occurs.

What's the number of layers past which you need this?

[tunystom]

@fchollet FYI, I added the unit tests you asked for. I am not sure whether it is required to check the internals of the created HDF5 file or not, but the current tests do so.

The number of layers of the model I could not save was over 1K. But the issue was not with the layer_names attribute inside HDF5 file (which also suffers from the same memory limit constraint and is fixed similarly), but with a different attribute array - weights_names. This happened as a result of the way I designed the whole model. I have actually 2 models: one is used as a nested model (feature extractor), this is the huge one, and the other servers as a backend built on top of it. If you take a look at the test method test_saving_model_with_long_weights_names you will see what I mean.

So, when you try to save a model containing a nested model, what happens is that the names of weights of the nested model are all saved within a single weights_names attribute of the HDF5 file. Since the list of names is converted to numpy array inside h5py, which effectively means every string inside the array will occupy the same amount as the longest string, it requires a lot of memory even for not very deep models after all.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.

[fchollet]

Sorry for not reviewing this earlier. It got forgotten in the PR queue. But it seems like it would still be a useful fix.

This PR has merge conflicts and will need a review (at a glance there will be at least small style issues to fix). Are you still interested in working on it?

[qieaaa]

Thanks for @tunystom !
I met this h5py error when I run Nasnet large.
I tried to replace your topology.py script in keras folder and there is still an error about training.py under engine folder.Which is
AttributeError: 'Model' object has no attribute '_internal_output_shapes'
I add one line in line 608 in training.py
self._internal_output_shapes= self.internal_output_shapes
and it works well.

[qieaaa]

I tried this and it works well 👍

[floydium]

I was able to rebase this PR against the current master. This required a minor conflict resolution in a unit test file within the test folder (which frankly I resolved quite randomly - only a few lines of code). Then I copied the newly rebased topology.py file into my keras library folder, and it did work without any issues. I was able to save NASNetMobile weights.

[ahundt]

merged and created an updated PR in #9398

[ahundt]

@floydium @qieaaa could you review the new PR #9398? Is an additional change needed for self._internal_output_shapes= self.internal_output_shapes?

[qieaaa]

@ahundt Thank you so much ! I just tried it, it's cool
I replace your new topology.py and delete my additional self._internal_output_shapes= self.internal_output_shapes. I run it and all work well，thanks! I can review the new PR

[fchollet]

Merged updated PR.

[Anurag27031994]

#Assuming that this is your model architecture. However, you may use whatever architecture, you want to (big or small; any).
def mymodel():
inputShape= (28, 28, 3);
model= Sequential()
model.add(Conv2D(20, 5, padding="same", input_shape=inputShape))
model.add(Activation('relu'))
model.add(Flatten())
model.add(Dense(500))
model.add(Activation('relu'))
model.add(Dense(2, activation= "softmax"))
return model
model.fit(....) #paramaters to start training your model

################################################################################
################################################################################
#once your model has been trained, you want to save your model in your PC
#use get.weights() command to get your model weights
weigh= model.get_weights()

#now, use pickle to save your model weights, instead of .h5
#for heavy model architectures, .h5 file is unsupported.
pklfile= "D:/modelweights.pkl"
try:
fpkl= open(pklfile, 'wb') #Python 3
pickle.dump(weigh, fpkl, protocol= pickle.HIGHEST_PROTOCOL)
fpkl.close()
except:
fpkl= open(pklfile, 'w') #Python 2
pickle.dump(weigh, fpkl, protocol= pickle.HIGHEST_PROTOCOL)
fpkl.close()

################################################################################
################################################################################
#in future, you mave want to load your model back
#use pickle to load model weights

pklfile= "D:/modelweights.pkl"
try:
f= open(pklfile) #Python 2

weigh= pickle.load(f);                
f.close();

except:

f= open(pklfile, 'rb')     #Python 3                 
weigh= pickle.load(f);                
f.close();

restoredmodel= mymodel()
#use set_weights to load the modelweights into the model architecture
restoredmodel.set_weights(weigh)

################################################################################
################################################################################
#now, you can do your testing and evaluation- predictions
y_pred= restoredmodel.predict(X)