log details to metadata for run analytics

[angel-ruiz7]

This will log information via the MosaicMLLogger to place the following keys in a run's metadata for analytics purposes. The data to log includes

• model_name: string
• script: 'Training', 'Eval'
• train_task_type: PRETRAIN, CONTINUED_PRETRAIN, INSTRUCTION_FINETUNE
• train_loader_name: string
• train_dataset_hf_name: string
• eval_loader_name: string
• eval_dataset_hf_name: string
• tokenizer_name: string
• n_heads: number
• d_model: int
• callbacks: string[]
• train_loader_workers: int
• eval_loader_workers: int
• gauntlet_configured: boolean
• icl_configured: boolean

Screenshots

Using the Quickstart example

Using the gpt2-small example

[angel-ruiz7]

still need to add the subtype. what's the best approach for this? currently, we classify the run by looking through the command to find the name of specific yaml or python files. what would be the best approach to do this inside of train.py?

cc @aspfohl @irenedea

[aspfohl]

More of an ask for foundry team: Is there anything else that would be useful for analytics?

Should we make it configurable to turn this on or off? Or is presence of MosaicMLLogger enough (users could always turn it off via MOSAICML_PLATFORM env var)

[irenedea]

Can you include evidence of this working in the PR description? Some manual tests and screenshots would be good.

[irenedea]

🚀 LGTM! Just one super super tiny formatting comment :) Thanks Angel! Will be great to have more logging and data 😄