Circular import error when using data/packing.py

[Paladiamors]

Hello,

Getting the following circular import error when trying to run the data/packing.py script

using the command below:

python llmfoundry/data/packing.py --yaml-path /home/justin/code/ai/llm-foundry/scripts/train/yamls/finetune/mpt-7b_dolly_sft.yaml

probably worth looking at llm-foundry/llmfoundry/data/__init__.py as this might be the cause of the error

Error:

Traceback (most recent call last):
  File "/home/justin/home/ai/llm-foundry/llmfoundry/data/packing.py", line 264, in <module>
    from llmfoundry import (build_finetuning_dataloader,
  File "/home/justin/home/ai/llm-foundry/llmfoundry/__init__.py", line 8, in <module>
    from llmfoundry.data import (ConcatTokensDataset,
  File "/home/justin/home/ai/llm-foundry/llmfoundry/data/__init__.py", line 7, in <module>
    from llmfoundry.data.finetuning import (Seq2SeqFinetuningCollator,
  File "/home/justin/home/ai/llm-foundry/llmfoundry/data/finetuning/__init__.py", line 5, in <module>
    from llmfoundry.data.finetuning.dataloader import build_finetuning_dataloader
  File "/home/justin/home/ai/llm-foundry/llmfoundry/data/finetuning/dataloader.py", line 14, in <module>
    from llmfoundry.data.finetuning.tasks import dataset_constructor
  File "/home/justin/home/ai/llm-foundry/llmfoundry/data/finetuning/tasks.py", line 38, in <module>
    import datasets
  File "/home/justin/home/ai/llm-foundry/llmfoundry/data/datasets.py", line 15, in <module>
    class NoConcatDataset(IterableDataset):
  File "/home/justin/home/ai/llm-foundry/llmfoundry/data/datasets.py", line 22, in NoConcatDataset
    hf_datasets.Dataset]):
AttributeError: partially initialized module 'datasets' has no attribute 'Dataset' (most likely due to a circular import)

Probably similar error to the following issue: #127

[hanlint]

Thanks @Paladiamors , I am able to reproduce this issue on our side. We will push out a fix.

[hanlint]

Hello, we pushed a fix in #175 . Closing for now, please re-open if you run into the same issue again!