Fix bug in dataset loading #284
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This fixes a bug when loading datasets. `d.data_files` is a list, so it cannot be directly passed to `hf_hub_download`
|
Hey, in my case, I use data_files as a string. If you would like to add this, could you consider checking the type of the variable (whether str or list) and deal appropriately? |
|
@ethanhs there is a syntax error in this changeset. Can you make sure the pre-commit hooks get run when committing please? |
|
Fixed/ran pre-commit, and checked the type of |
|
Hey, could you please clarify on the original bug? Could you provide some examples and error? accelerate launch scripts/finetune.py examples/openllama-3b/lora.yml --prepare_ds_only --debugdatasets:
- path: teknium/GPT4-LLM-Cleaned
type: alpaca
data_files:
- alpaca_gpt4_data_unfiltered.json
- unnatural_instructions_unfiltered.jsonI can run this fine on the current master. |
|
@NanoCode012 I think the code he is modifying is fallback code. Typically dataset loading from hf hub happens in the conditional block before this one. I honestly don't know when this else clause might get called, but it's probably worth having for that edge case. |
Sounds good! |
mkeoliya
pushed a commit
to mkeoliya/axolotl
that referenced
this pull request
Dec 15, 2023
* Fix bug in dataset loading This fixes a bug when loading datasets. `d.data_files` is a list, so it cannot be directly passed to `hf_hub_download` * Check type of data_files, and load accordingly
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This fixes a bug when loading datasets.
d.data_filesis a list, so it cannot be directly passed tohf_hub_download, which only takes a single filename.