Add axolotl training example

[casper-hansen]

Axolotl is one of the most popular open-source pertaining and fine-tuning packages that make use of the Huggingface ecosystem.

Speed ups achieved in axolotl:

• Implements sample packing (can be up to 10x faster)
• Implements flash attention
• Implements fused RMSNorm that speeds training 15-20
• Memory efficiency with bitsandbytes 8-bit optimizer

Ease of use:

• Specify a config, start training
• Implements LoRA and QLoRA
• Implements DeepSpeed and FSDP
• Easy monitoring with wandb
• Built-in prompt templates
• Pulls and pushes datasets and models from and to Huggingface

https://github.com/OpenAccess-AI-Collective/axolotl

[concretevitamin]

This would be great to have. Some users are already using axolotl:

#2671 (comment)

#2689 (comment)

[manishiitg]

# jupyter.yaml

name: axolotl

resources:
  accelerators: A100:1
  cloud: gcp
  use_spot: True

file_mounts:
  /sky-notebook:
    source: gs://sky-notebook
    mode: MOUNT

workdir: mistral

setup: |
  docker pull winglian/axolotl:main-py3.10-cu118-2.0.1
  pip install autoawq peft

  

run: |
  docker run --gpus all \
    -v /home/gcpuser/sky_workdir:/sky_workdir \
    -v /root/.cache:/root/.cache \
    winglian/axolotl:main-latest \
    huggingface-cli login --token XXX

  docker run --gpus all \
    -v /home/gcpuser/sky_workdir:/sky_workdir \
    -v /root/.cache:/root/.cache \
    winglian/axolotl:main-latest \
                accelerate launch -m axolotl.cli.train /sky_workdir/hi-qlora.yaml

  
  
  
  


  

this would work

[concretevitamin]

Awesome @manishiitg !! Would you like to open a PR and push it to llm/axolotl?

[manishiitg]

@concretevitamin #2784

check this out

[concretevitamin]

@casper-hansen This has been added by @manishiitg. Closing this.