Add registry for ICL datasets

[sanjari-orb]

Purpose of PR: Create a registry for ICL eval dataset types. This will allow users to create custom in-context learning datasets and add them to the registry to run custom ICL evaluations during training.

[dakinggg]

Hi @sanjari-orb could you please add a PR description describing the change? Thank you!

[sanjari-orb]

Hi @dakinggg Sorry this was still a draft because I was still trying to get it to work. But thanks for the comments, I'll take them into account and update the PR soon.

[dakinggg]

No worries, thanks for the contribution!

[sanjari-orb]

Hi @dakinggg could you point me to the steps to run the unit tests locally?

[dakinggg]

Please see the makefile here (https://github.com/mosaicml/llm-foundry/blob/main/Makefile). Sorry there aren't better instructions!

CPU tests

make test

Multi CPU tests

make test-dist

Single GPU tests

make test-gpu

Multi GPU tests

make test-dist-gpu

[dakinggg]

Thanks! Had a couple comments. Also, once the tests are passing, I'll run this PR through our regression tests to check and make sure we aren't accidentally changing any evals.

[sanjari-orb]

@dakinggg Is there a linter I can use to fix the code quality checks?

[dakinggg]

@sanjari-orb yeah, running pre-commit should do it

[dakinggg]

Successful regression test run name: llm-foundry-regression-tests-runner-ActFp3

LGTM, thank you for the contribution!

[sanjari-orb]

@dakinggg I'm not sure why the PR GPU tests failed: https://github.com/mosaicml/llm-foundry/actions/runs/9514022193/job/26225300100?pr=1252. Could you take a look?