Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

minor fixes 20231211 #943

Closed
wants to merge 3 commits into from
Closed

minor fixes 20231211 #943

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2de1bfa
misc fixes
winglian Dec 12, 2023
2f600e2
nano editor in base docker
winglian Dec 12, 2023
8c78197
move nano install to axolotl image
winglian Dec 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docker/Dockerfile-runpod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ ENV HF_DATASETS_CACHE="/workspace/data/huggingface-cache/datasets"
ENV HUGGINGFACE_HUB_CACHE="/workspace/data/huggingface-cache/hub"
ENV TRANSFORMERS_CACHE="/workspace/data/huggingface-cache/hub"
ENV HF_HOME="/workspace/data/huggingface-cache/hub"
ENV HF_HUB_ENABLE_HF_TRANSFER=1

COPY scripts/runpod-entrypoint.sh /root/runpod-entrypoint.sh

Expand Down
4 changes: 4 additions & 0 deletions src/axolotl/cli/merge_lora.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,10 @@ def do_cli(config: Path = Path("examples/"), **kwargs):
parsed_cli_args.merge_lora = True
parsed_cfg = load_cfg(config, merge_lora=True, **kwargs)

parsed_cfg.load_in_4bit = False
parsed_cfg.load_in_8bit = False
parsed_cfg.flash_attention = False

do_merge_lora(cfg=parsed_cfg, cli_args=parsed_cli_args)


Expand Down
3 changes: 2 additions & 1 deletion src/axolotl/utils/models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,7 @@ def load_model(

# TODO refactor as a kwarg
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there's this todo here.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not even sure what this TODO is for atm

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe it's for the .from_pretrained call where these are kwargs for that? should we add this to model_kwargs and remove those from all the from_pretrained calls?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just above

   load_in_8bit = cfg.load_in_8bit
     load_in_4bit = cfg.load_in_4bit

load_in_8bit = cfg.load_in_8bit
load_in_4bit = cfg.load_in_4bit

if hasattr(model_config, "model_type") and model_config.model_type == "btlm":
if cfg.flash_attention:
Expand Down Expand Up @@ -535,7 +536,7 @@ def load_model(

model, lora_config = load_adapter(model, cfg, cfg.adapter)

if cfg.ddp and not load_in_8bit:
if cfg.ddp and not load_in_8bit and not load_in_4bit:
model.to(f"cuda:{cfg.local_rank}")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's the side effect of this change? Does it mean, the models will all live on gpu0 for 4 bit now?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there's an edge case that I had to have this set when doing DPO qlora for Mixstral. maybe it was a deepspeed thing and I should change this?


if torch.cuda.device_count() > 1 and int(os.getenv("WORLD_SIZE", "1")) == 1:
Expand Down