Automatic checkpointing

[Zahlii]

Hi,

Would it be possible to automatically save the "best" model run in terms of global loss?

Maybe in a way similar to how https://keras.io/callbacks/#modelcheckpoint works?
I had different runs with about 2.000 epochs where the loss would go down to about 5.5 after 1.200 runs just to skyrocket and settle at around 1E7 in the end, with no way to access the intermediate results.

Best regards

[geffy]

Hi, yes, it's possible, will add it to my todos.
Possible workaround for now: in loop fit model for several iterations, check loss and save if needed.