minor fixes 20231211 #943
minor fixes 20231211 #943
Conversation
| @@ -191,6 +191,7 @@ def load_model( | |||
|
|
|||
| # TODO refactor as a kwarg | |||
There was a problem hiding this comment.
there's this todo here.
There was a problem hiding this comment.
I'm not even sure what this TODO is for atm
There was a problem hiding this comment.
maybe it's for the .from_pretrained call where these are kwargs for that? should we add this to model_kwargs and remove those from all the from_pretrained calls?
There was a problem hiding this comment.
This is just above
load_in_8bit = cfg.load_in_8bit
load_in_4bit = cfg.load_in_4bit
| @@ -535,7 +536,7 @@ def load_model( | |||
|
|
|||
| model, lora_config = load_adapter(model, cfg, cfg.adapter) | |||
|
|
|||
| if cfg.ddp and not load_in_8bit: | |||
| if cfg.ddp and not load_in_8bit and not load_in_4bit: | |||
| model.to(f"cuda:{cfg.local_rank}") | |||
There was a problem hiding this comment.
what's the side effect of this change? Does it mean, the models will all live on gpu0 for 4 bit now?
There was a problem hiding this comment.
there's an edge case that I had to have this set when doing DPO qlora for Mixstral. maybe it was a deepspeed thing and I should change this?
|
I found that you separated your apt installs into 2 separate docker file. Maybe, these should also be moved to base |
No description provided.