Fix bug when using pretokenized datasets (axolotl-ai-cloud#652)

* fix pretokenized datasets readme * check if dataset type is not set to handle pretokenized datasets
mkeoliya · Sep 29, 2023 · 16c136a · 16c136a
1 parent 22a3fe9
commit 16c136a
Show file tree

Hide file tree

Showing 2 changed files with 3 additions and 1 deletion.
diff --git a/README.md b/README.md
@@ -317,7 +317,7 @@ Using file:
 #### How to use your custom pretokenized dataset
 
 - Do not pass a `type:`
-- Dataset must contain `input_ids`, `attention_mask`, `labels` in columns
+- Columns in Dataset must be exactly `input_ids`, `attention_mask`, `labels`
 
 
 ### Config

diff --git a/src/axolotl/utils/config.py b/src/axolotl/utils/config.py
@@ -293,6 +293,8 @@ def validate_config(cfg):
 
     if cfg.datasets:
         for idx, ds_cfg in enumerate(cfg.datasets):
+            if not ds_cfg.type:
+                continue
             if ds_cfg.type == "sharegpt:chat":
                 LOG.warning(
                     PendingDeprecationWarning(