Commits on Sep 28, 2023

1. Make dataset_processes configurable …

   I'm using the Axolotl script to train models on https://modal.com serverless GPUs. Unfortunately, their environment seems to have some kind of bug where if I try to run `datasets.filter` with too high a `num_proc`, it throws an error and dies.
   
   This PR adds a new configuration option `dataset_processes`, which lets you explicitly set the number of processes used to map/filter the dataset. If not included, this defaults to the current behavior of setting that to `os.cpu_count()`.

   

   corbt committed Sep 28, 2023
   Configuration menu

   • View commit details
   • Copy full SHA for 66b3d5f
   • Browse repository at this point

   Copy the full SHA

   66b3d5f View commit details

   Browse the repository at this point in the history