Skip to content

Commit

Permalink
just cast straight to string for ds hash
Browse files Browse the repository at this point in the history
  • Loading branch information
winglian committed Jan 22, 2024
1 parent 1c1a732 commit 584074d
Showing 1 changed file with 3 additions and 9 deletions.
12 changes: 3 additions & 9 deletions src/axolotl/utils/data.py
Original file line number Diff line number Diff line change
Expand Up @@ -116,17 +116,11 @@ def load_tokenized_prepared_datasets(
(
str(cfg.sequence_len)
+ "@"
+ str(int(cfg.sample_packing))
+ str(cfg.sample_packing)
+ "@"
+ str(
int(
-1
if (cfg.eval_sample_packing is None)
else cfg.eval_sample_packing
)
)
+ str(cfg.eval_sample_packing)
+ "@"
+ str(int(cfg.group_by_length))
+ str(cfg.group_by_length)
+ "@"
+ "|".join(
sorted(
Expand Down

0 comments on commit 584074d

Please sign in to comment.