Support datasets from huggingface

[SingL3]

🚀 Feature Request

Now this package can load data from local path / http / s3, is there a plan to support huggingface datasets?

Motivation

Some datasets supply non-jsonl datasets, like parquet.

[Optional] Implementation

Additional context

[dakinggg]

The composer Trainer accepts an arbitrary train_dataloader, so I'm not sure what you mean here. Could you please clarify?

[SingL3]

As you can see here:

composer/composer/datasets/in_context_learning_evaluation.py

Line 145 in f2a2dc8

dataset = load_dataset('json', data_files=destination_path, split='train', streaming=False)

This mean users should be local and it does not support other format of data like parquet.
The benefit of datasets may be it can download automatically if there is no local file.

[dakinggg]

Ah, got it! The code actually does automatically download from object store (

composer/composer/datasets/in_context_learning_evaluation.py

Line 333 in ff59e86

get_file(dataset_uri, destination_path, overwrite=True)

), and the ICL classes expect the data to be in a particular format that probably isn't super common on the HF hub, but we can look more into supporting that!