find alternative to gfile

[Borda]

🚀 Feature

at this moment we are using GFile for saving checkpoints which are imported from tensorboard which underhood uses TF core...

Motivation

remove the hard dependency on tensorboard as it seems tidy linked to TensorFlow and for visualization user can install it independently

Pitch

Avoid tensorboard limitations

[awaelchli]

It is used for saving to remote locations.
pinging @f4hy who made the original PR, maybe has an idea or comments

[f4hy]

I agree it would be nice to remove a dependency on tensor-board but I felt it was ok because it already was a dependency of the project.

I want to note that tensorboard can be installed without tensorflow and the gfile operations work in 'compatible' mode where only local and s3 is supported. This does not require full tensorflow to be installed, just the standalone tensorboard package.

The problem is there are not really any good alternatives. Projects like smart_open can support writing to remote file storage but don't support things removing/moving files or checking for existence which lightning does. I think the community could really use a good remote_filesystem package which can support the various cloud compute platforms gcs/s3/hdfs the way gfile does but standalone and minimal dependencies. I am currently unaware of a good standalone solution.

If there is a package I don't know about I would be happy to use it. The other option if we want to remove tensorboard support is refactor all the IO further to check if a user does have tensorboard/tensorflow installed, and if so support remote directories, and otherwise warn that only local file operations are supported.

[Borda]

I agree it would be nice to remove a dependency on tensor-board but I felt it was ok because it already was a dependency of the project.

sure, no one is blaming you, at the time it was a great solution, just later we found this limitation... :]

The problem is there are not really any good alternatives.

Well I also doe not say explicitly remove tensorboard, just have look if we can without losing the actual functionality

If there is a package I don't know about I would be happy to use it.

that is pretty much what this issue is, just to have a look around and eventually circle back that there is nothing better then what we have 🐰

[f4hy]

I'm very happy to help in this as well to make sure any new solution works for our use case. At the moment my org needs hdfs and s3 support.

I think the main issue I see with the current implementation is nothing preventing someone from adding IO operations to lightning which would only work locally. In that case all tests would still pass but would then crash when trying to use with a remote dir.

I like that currently things should all still work if all paths are local and a user doesn't have tensorflow installed. But a better way to notify and fail with a proper message if a remote directory is passed in and the user doesn't have tf. It currently does print some messages but not sure all possible cases are caught with a good message.

[EspenHa]

Suggestion: fsspec

[f4hy]

Suggestion: fsspec

Oh! Looks promising :) thanks for the suggestion let me give that a try.

[f4hy]

I played with fsspec a bit. I think this is the right solution. :) Very cool. I will start working on implementing this.

[awaelchli]

Now that we don't depend on gfile anymore, can we remove tensorboard from dependencies and just make it optional?

[Borda]

yes, it is in #3132

[awaelchli]

Nice, that looks good! But maybe rename the PR title :)

[igorgad]

Hi, I am receiving ValueError: No recognized filesystem for prefix gs but it works if I install tensorflow. Is there a way to log directly to gcs bucket without installing tensorflow? I'm running from a docker container with lightning 1.1.2.
Thanks