Enable gauntlet training

[bmosaicml]

This PR enables us to run the gauntlet during training. It also enables you to set a subset of batches to run eval over in order to speed up ICL during training.

It also changes eval to no longer use the InMemoryLogger and instead pull the metrics from the State.

Eval test run: all-eval-F2oe1D
Training test run: test-1b-C5QG5k test-1b-c7sqCh

https://wandb.ai/mosaic-ml/gauntlet/runs/rrbn0qbb?workspace=user-jemdohmann

Eval results still good:

| model_name               |   average |   world_knowledge |   commonsense_reasoning |   language_understanding |   symbolic_problem_solving |   reading_comprehension |
|:-------------------------|----------:|------------------:|------------------------:|-------------------------:|---------------------------:|------------------------:|
| mosaicml/mpt-7b-instruct |  0.354255 |          0.398764 |                0.415097 |                 0.371509 |                   0.171216 |                0.414691 |

[hanlint]

Thanks for putting this PR together, left a few comments:

• the new yaml files have unexplained changes (batch size, fsdp config), and also have several hardcoded internal OCI bucket names and cluster names. Please remove the hardcoded values prior to pushing PRs.
• can we avoid silently adding in an InMemoryLogger, purely for the purposes of passing around data? Seems like there's a deeper design issue if we have to resort to this approach.

[eracah]

Might be worth using state to get the metrics instead of inmemorylogger

[bmosaicml]

The tests that you have added are really slow, nearly doubling the time it takes CI to run

196.68s call     tests/test_training.py::test_train_gauntlet
161.70s call     tests/test_training.py::test_train[cpu]
120.18s call     tests/test_model_gauntlet.py::test_gauntlet_callback[True-False]
120.00s call     tests/test_model_gauntlet.py::test_gauntlet_callback[False-True]
119.20s call     tests/test_model_gauntlet.py::test_gauntlet_callback[True-True]
118.90s call     tests/test_model_gauntlet.py::test_gauntlet_callback[False-False]

Can you please reduce the intensity of the tests? Probably run less eval tasks or subset batches or something?

I don't think test_train_gauntlet can be sped up anymore as it's now about the same speed as test_train. Should I just cut it? Or make it GPU only? Alternatively we could cut test_train since it is merely a subset of the functionality of test_train_gauntlet

[dakinggg]

see remaining comments around backwards compatibility and a couple code changes that I think were a bad merge.