-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
URL passed to fs.listdir instead of a path #15379
Comments
changing to this fixes it:
I think using |
@leoleoasd Thanks for reporting this. |
Yes, |
I think, to follow the examples, we should update all usages of |
Bug description
https://github.com/Lightning-AI/lightning/blob/master/src/pytorch_lightning/trainer/connectors/checkpoint_connector.py#L582
This line is called
fs.listdir(dir_path)
wheredir_path
is an URL instead of a path.pyarrow will complain about it:
How to reproduce the bug
use a HDFS path in trainer's default_root_dir
Error messages and logs
Environment
fsspec: 2022.10.0
pyarrow: 8.0.0
pytorch_lightning: 1.7.7
More info
No response
cc @awaelchli @ananthsub @ninginthecloud @rohitgr7 @otaj
The text was updated successfully, but these errors were encountered: