Update README with some explanations

[seungduk-yanolja]

Description

Added some explanation and examples to the YAML config documentation to help future users.

Motivation and Context

Sharing lesson-learned

How has this been tested?

MD file viewer

Screenshots (if appropriate)

N/A

Types of changes

Comments

[winglian]

micro batch size is the per gpu number of samples to accumulate in each forward pass.
micro batch size * gradient accumulations steps * # of gpus = total batch size

[seungduk-yanolja]

added more explanation below

[flexchar]

I absolutely appreciate the work on this! If I may, I'd really like to ask for a couple short examples (a sentence or two) on each option for dummies - technical people who are not from Machine Learning/AI background.

For example:
lora_r - specifies how many layers should be trained. The more layers the longer the training will take but it can yield better results with bigger dataset. It's recommended to use 32 or 16.
num_epochs - how many times should the whole training be repeated. It's also known as training steps. The more epochs the better can model learn the data. Recommended starting point is 10.
micro_batch_size - how many trainers run at the same time. Running more can speed up the progress but can also cause OOM... best to leave at 2.
lr_scheduler - this means abc and xyz. Recommended to use cosine unless you know what you are doing. And so on so for.

NOTE I have no idea if this is correct, I have technical non ML background and my goal is to learn the practical aspect of training with some navigation in theory too. :)

[seungduk-yanolja]

I absolutely appreciate the work on this! If I may, I'd really like to ask for a couple short examples (a sentence or two) on each option for dummies - technical people who are not from Machine Learning/AI background.

For example: lora_r - specifies how many layers should be trained. The more layers the longer the training will take but it can yield better results with bigger dataset. It's recommended to use 32 or 16. num_epochs - how many times should the whole training be repeated. It's also known as training steps. The more epochs the better can model learn the data. Recommended starting point is 10. micro_batch_size - how many trainers run at the same time. Running more can speed up the progress but can also cause OOM... best to leave at 2. lr_scheduler - this means abc and xyz. Recommended to use cosine unless you know what you are doing. And so on so for.

NOTE I have no idea if this is correct, I have technical non ML background and my goal is to learn the practical aspect of training with some navigation in theory too. :)

I added a link in the doc for more details about the LoRA hyperparameters.
https://www.anyscale.com/blog/fine-tuning-llms-lora-or-full-parameter-an-in-depth-analysis-with-llama-2

PTAL

[winglian]

@seungduk-yanolja thanks for doing this! much needed. are you happy with the state of this? Should I go ahead and merge?

[seungduk-yanolja]

@seungduk-yanolja thanks for doing this! much needed. are you happy with the state of this? Should I go ahead and merge?

yes, please! thanks

[seungduk-yanolja]

I added one more explanation about lora_modules_to_save as follows.

# If you added new tokens to the tokenizer, you may need to save some LoRA modules because they need to know the new tokens.
# For LLaMA and Mistral, you need to save `embed_tokens` and `lm_head`. It may vary for other models.
# `embed_tokens` converts tokens to embeddings, and `lm_head` converts embeddings to token probabilities.
# https://github.com/huggingface/peft/issues/334#issuecomment-1561727994
lora_modules_to_save:
#  - embed_tokens
#  - lm_head

[flexchar]

Many thanks for this! It means the world to me and to many others. I also found this resource written up for StableDiffusion LoRAs. It seems to be quite relevant https://github.com/bmaltais/kohya_ss/wiki/LoRA-training-parameters.