Try using polars for fasting data loading or even use Hugging Face Datasets

[mrdbourke]

Right now the data_loader.py script works with a pandas DataFrame.

This is fine with ~25,000 images but perhaps not ideal for larger datasets.

Alternatives are:

• polars, a DataFrame-like library built in Rust (very fast) - https://pola-rs.github.io/polars/py-polars/html/index.html
• Hugging Face Datasets, also very fast apparently, see the docs - https://huggingface.co/docs/datasets/index

Can also test the speed of data loading using:

• PyTorch profiler - https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html
• PyTorch trace analysis - https://pytorch.org/blog/trace-analysis-for-masses/