Update gauntlet v0.2 to reflect results of calibration

[bmosaicml]

In order to filter out noisy tasks and set appropriate number of few shot per task, we used the following methodology:

1. Evaluate each benchmark on several 3B models trained for [20x, 50x, 100x, 250x, 500x] chinchilla token duration for num_fewshot=[0, 1, 3, 5, 10]
2. Determine which num_fewshot settings the models have monotonically increasing performance. If multiple num_fewshot have monotonically increasing performance we broke ties by choosing the industry standard if there is one, or 3.
3. For benchmarks which were scoring 0 (and therefore not monotonic), we excluded them unless they are known to show good scaling at larger model scales. These tasks will average in as 0 in expectation, but they will introduce slight variance.

Later versions of this experiment will estimate the exact confidence intervals of the gauntlet sub-scores + overall average.

Google sheet

[abhi-mosaic]

small comment, but otherwise looks good to me! I spot checked the most important ICLs and the k-shots look like what was discussed.

[dakinggg]

LGTM from my perspective. Please wait for whoever you want to take a look from research.

[tbarton16]

LGTM

[sashaDoubov]

LGTM 🚀