Add a callback to write huggingface checkpoints during the training run

[dakinggg]

This adds a callback that does the huggingface checkpoint conversion during the training job to avoid further difficulty after the job is complete, and take advantage of backgrounded uploads while training is happening.

• add precision
• manual mpt test (7b-mpt-hf-ckpt-4-jc3uSG)
• manual llama2 test (l7b-hf-ckpt-4-HpZUAg)

[mvpatel2000]

Should this be in Composer?

[mvpatel2000]

Will defer to Evan on this one

[eracah]

Nice looks pretty good. Def doable to make a SaveForInferenceCallback I think in the future. Also some good nuggets here for refactoring CheckpointSaver

[eracah]

Should this be in Composer?

At some point yes

[mvpatel2000]

Can we add a flag to only save at end? I imagine the most common use case is I just want to convert to HF once im done training

[dakinggg]

@mvpatel2000 can't you just specify 1dur as the save interval?

[mvpatel2000]

@mvpatel2000 can't you just specify 1dur as the save interval?

🤯 ur so big brained

[eracah]

Aight LGTM

[germanjke]

@eracah @dakinggg hi guys! This thing supports no-MPT models, like LLaMA?

[dakinggg]

yes