feature: better device mapping for large models #918
Conversation
|
Apart from the one minor change, this looks good. I haven't had a chance to test this yet though. |
|
Thanks. I could have sworn I did that and it failed when I wrote the initial code, but it works now. |
examples/llama-2/qlora.yml
Outdated
| gpu_memory_limit: 20GiB | ||
| lora_on_cpu: true |
There was a problem hiding this comment.
I don't think adding these for default yaml would be good as a user might not need this and build off this yaml unknowingly.
| gpu_memory_limit: 20GiB | |
| lora_on_cpu: true |
kallewoof
force-pushed
the
202312-better-devicemapping
branch
4 times, most recently
from
2679cbd
to
68982f8
Compare
|
No idea why the test is failing. Feedback would be appreciated. |
Should check if exist before delete? |
Thanks. This one no longer exists as it's been mapped into a proper device map with this pull request. Removing the |
kallewoof
force-pushed
the
202312-better-devicemapping
branch
2 times, most recently
from
dbbce93
to
cd34680
Compare
|
I was banging my head for hours about this issue, then i stumble upon this PR, i try it, and it works flawlessly! Was trying to merge a 13b base model with fine tuned qlora adapters on 24GB VRAM GPU (and 32GB system RAM) EC2 instance. so basically this PR allows the merging process to use as much GPU as possible and allows spilling to RAM if needed! very convenient when merging large models without enough VRAM resources!! Thank you very much @kallewoof ! |
There was a problem hiding this comment.
Hey, I apologize for the late review. I think this is very nice as seen in other's posts.
Can I ask for one last thing, which is to add a validation check in def validate_config(cfg): within utils/config.py to make sure that if both max_memory and gpu_memory_limit is passed, to raise a ValueError and warn user to use one of them only?
Afterwards, I think this is good to merge.
|
@NanoCode012 Thanks for the review. Added to validation checks. |
For big models where the models are taking up the entire GPU VRAM, the LoRA part will fail unless it is loaded on CPU only.
Co-authored-by: Wing Lian <wing.lian@gmail.com>
kallewoof
force-pushed
the
202312-better-devicemapping
branch
from
4425a6f
to
05ed3d1
Compare
|
Thank you for the PR! |
When a model does not fit completely into the GPU (at 16-bits, if merging with a LoRA), a crash occurs, indicating we need an offload dir. If we hide the GPUs and do it purely in CPU, it works, but now we are not using the GPUs at all.
If we try to do offloading while using GPUs, accelerate ends up trying to offload types that are only supported on the GPUs, which results in the possibly infamous
NotImplementedError: Cannot copy out of meta tensor; no data!error.This pull requests adds two new configuration options. One,
gpu_memory_limitis a convenience option that can be used instead of manually setting themax_memoryconfig. (For per-GPU maxes, you need to set it manually though.) It defaults to GBs if an integer, and is assumed to be a proper memory string otherwise (e.g. "123MiB").The other,
lora_on_cpuis a flag which if set will force the PeftModel loading part to be done purely on CPU end. This slows things down, but if the model is taking up too much of the GPU VRAM, the only alternative is to crash and/or buy more GPUs.Main, attempting to merge a 34b codellama model with a lora, on a 24 GB A10G with 30GB RAM a,d 64GB swap:
This pull request, with
(V)RAM stats (from a previous run, so log times will differ):
Peak (V)RAM usage:
Post-run (V)RAM usage: