add support for s3 checkpoint

[eliebak]

Goal

Allow the upload checkpoint to s3 during training + resuming from ckpt

How

Add S3UploadArgs class, define like this in the config

s3_upload:
  remove_after_upload: true
  s5cmd_concurrency: 5
  s5cmd_numworkers: 16
  s5cmd_path: path_to_s5cmd 
  upload_s3_path: path_to_s3

Add support for s3 path in resume_checkpoint_path and it will copy (with s5cmd) it toocheckpoints_path. This is done in the pre_init phase with the parse_ckpt_path(config=self.config, parallel_context=self.parallel_context) function.

[3outeille]

cleanup comment

[3outeille]

remove as it seems to be unused

[3outeille]

Please run the pre-commit so that it fix automatically everything related to coding styles

pip install pre-commit
pre-commit install

In any case, good job !