-
-
Notifications
You must be signed in to change notification settings - Fork 780
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feature: better device mapping for large models #918
feature: better device mapping for large models #918
Conversation
Apart from the one minor change, this looks good. I haven't had a chance to test this yet though. |
Thanks. I could have sworn I did that and it failed when I wrote the initial code, but it works now. |
examples/llama-2/qlora.yml
Outdated
gpu_memory_limit: 20GiB | ||
lora_on_cpu: true |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think adding these for default yaml would be good as a user might not need this and build off this yaml unknowingly.
gpu_memory_limit: 20GiB | |
lora_on_cpu: true |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed.
2679cbd
to
68982f8
Compare
No idea why the test is failing. Feedback would be appreciated. |
Should check if exist before delete? |
Thanks. This one no longer exists as it's been mapped into a proper device map with this pull request. Removing the |
dbbce93
to
cd34680
Compare
I was banging my head for hours about this issue, then i stumble upon this PR, i try it, and it works flawlessly! Was trying to merge a 13b base model with fine tuned qlora adapters on 24GB VRAM GPU (and 32GB system RAM) EC2 instance. so basically this PR allows the merging process to use as much GPU as possible and allows spilling to RAM if needed! very convenient when merging large models without enough VRAM resources!! Thank you very much @kallewoof ! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey, I apologize for the late review. I think this is very nice as seen in other's posts.
Can I ask for one last thing, which is to add a validation check in def validate_config(cfg):
within utils/config.py
to make sure that if both max_memory and gpu_memory_limit is passed, to raise a ValueError and warn user to use one of them only?
Afterwards, I think this is good to merge.
@NanoCode012 Thanks for the review. Added to validation checks. |
For big models where the models are taking up the entire GPU VRAM, the LoRA part will fail unless it is loaded on CPU only.
Co-authored-by: Wing Lian <wing.lian@gmail.com>
4425a6f
to
05ed3d1
Compare
Thank you for the PR! |
When a model does not fit completely into the GPU (at 16-bits, if merging with a LoRA), a crash occurs, indicating we need an offload dir. If we hide the GPUs and do it purely in CPU, it works, but now we are not using the GPUs at all.
If we try to do offloading while using GPUs, accelerate ends up trying to offload types that are only supported on the GPUs, which results in the possibly infamous
NotImplementedError: Cannot copy out of meta tensor; no data!
error.This pull requests adds two new configuration options. One,
gpu_memory_limit
is a convenience option that can be used instead of manually setting themax_memory
config. (For per-GPU maxes, you need to set it manually though.) It defaults to GBs if an integer, and is assumed to be a proper memory string otherwise (e.g. "123MiB
").The other,
lora_on_cpu
is a flag which if set will force the PeftModel loading part to be done purely on CPU end. This slows things down, but if the model is taking up too much of the GPU VRAM, the only alternative is to crash and/or buy more GPUs.Main, attempting to merge a 34b codellama model with a lora, on a 24 GB A10G with 30GB RAM a,d 64GB swap:
This pull request, with
(V)RAM stats (from a previous run, so log times will differ):
Peak (V)RAM usage:
Post-run (V)RAM usage: