Feat: Add dataset loading from S3, GCS

[NanoCode012]

Closes #750

This allows loading from S3/GCS.

Usage: path: s3://path/to/data.jsonl or path: s3://path/to/parquet/dir/. Use same convention as local dataset, passing ds_type where appropriate.

Test:

• S3
• GCS

Note: Azure is commented out till we figure out how to pass creds securely

[winglian]

I don't believe this is equivalent, since the previous load_dataset call also has a data_files component to only load specific files from the path.

[gordicaleksa]

This indeed broke my workflow: I have a directory with txt files inside, the load_dataset knew how to handle that, load_from_disk raises an exception

[NanoCode012]

@gordicaleksa , could you please open an Issue along with a sample config? It'll make it easier to remember this.

[jphme]

Very cool feature! Would it in theory be possible to extend the same code/approach also for model loading and checkpoint saving?

[NanoCode012]

@jphme , thanks! That's my next plan. However, this approach relies partially on dataset implementation for cloud loading. We would need a separate (maybe similar) pipeline for model saving.

[casper-hansen]

How will we differentiate between downloading vs streaming from your cloud storage in the future when streaming is added? Could we just use the same path but differentiate between the two somehow?

[NanoCode012]

How will we differentiate between downloading vs streaming from your cloud storage in the future when streaming is added? Could we just use the same path but differentiate between the two somehow?

Perhaps, we might have a stream: true parameter? Also, if I recall correctly, streaming would be with the Streaming library, right?