support for explicit test_dataset definition for evals

[winglian]

first pass for supporting different datasets for evals rather than splitting the test dataset.

[NanoCode012]

Hmm, our data pipeline is getting a bit complex. I had a bit of hard time following the flow

[NanoCode012]

Adding this config to Readme would be helpful

[winglian]

This will address #875

[NanoCode012]

I think will need to add to doc about this and also, whether it would be appropriate to hardcode the train/test split or allow more open control?

[winglian]

I think will need to add to doc about this and also, whether it would be appropriate to hardcode the train/test split or allow more open control?

@NanoCode012 What do you mean about hardcoding?

[DreamGenX]

Do you have an example or documentation of how this can be used?

[DreamGenX]

I was trying to reserve engineer how it's supposed to work, and maybe there's a bug here:

https://github.com/OpenAccess-AI-Collective/axolotl/blob/18f811978c01d567c2294140f53abcf8c086e337/src/axolotl/utils/data.py#L443

    dataset, prompters = load_tokenized_prepared_datasets(
        tokenizer, cfg, default_dataset_prepared_path
    )

Shouldn't you pass split=split to load_tokenized_prepared_datasets? Otherwise it will never load from cfg.test_datasets.