Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support device_map=sequential & max_memory config parameters #903

Merged
merged 3 commits into from
Dec 4, 2023

Conversation

brthor
Copy link
Contributor

@brthor brthor commented Nov 28, 2023

In order to train a large model, I recently had to use the combination of device_map=sequential and max_memory in order to balance the memory usage per gpu (in model parallel) manually.

  • device_map=auto should not be forced when a value has been specified in the config
  • max_memory allows for setting memory limits per gpu and is most useful with device_map=sequential

This was the simplest way I found to implement supporting these two options.

cc @winglian

Copy link
Collaborator

@winglian winglian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

README.md Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
@winglian winglian merged commit 992e742 into axolotl-ai-cloud:main Dec 4, 2023
4 checks passed
mkeoliya pushed a commit to mkeoliya/axolotl that referenced this pull request Dec 15, 2023
…-ai-cloud#903)

* Support device_map sequential (and others). Support max_memory in cfg.

* Update documentation in README accordingly.

* Update README.md

---------

Co-authored-by: Wing Lian <wing.lian@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants