[help] File sync timed out (file does not exist or has incorrect hash)

[gadenbuie]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I'm working on a targets pipeline with a dynamic branching target that creates .parquet files. As a dynamic branching target, the target maps over a vector of just under 5k inputs. Eventually, every run of the branch will fail with the following message

Error:
! Error running targets::tar_make()
  Error messages: targets::tar_meta(fields = error, complete_only = TRUE)
  Debugging guide: https://books.ropensci.org/targets/debugging.html
  How to ask for help: https://books.ropensci.org/targets/help.html
  Last error: timed out after retrying for 60 seconds. Path /Users/garrick/..[redacted]../part-0.parquet does not exist or has incorrect hash. File sync timed out.

The file does exist at this path, so it must be an issue with the hash (at least according to the error message). Unfortunately, this error doesn't follow the error option; even though I've set that to "null", this error brings down the entire tar_make() pipeline.

Does anyone have any experience with this issue or know how to work around it?

[wlandau]

The error is part of a safeguard for network file systems and distributed computing. If you run a pipeline on a traditional cluster and a parallel worker writes a file, that file might not fully update for another minute or so. That's why targets waits for the local hash computed by the main process to agree with the hash computed by the worker. If you are running a pipeline on a slow network drive or NFS/EFS mount, you might want to increase the timeout using e.g. tar_option_set(resources = tar_resources(network = tar_resources_network(seconds_timeout = 360))).

[wlandau]

It could be that normally read-only system calls are somehow modifying the file's hash. This is a strange and new problem, so it's hard to speculate without a minimal reprex.

[gadenbuie]

Two follow up questions:

1. Could switching to format = "file_fast" be a quick way to work around this?
2. I may have written some of these parquet files during interactive testing (or certainly during previous runs). Could those files that were created outside of tar_make() be problematic somehow?

I'll see if I can come up with a reprex.

[wlandau]

I don't think file_fast will help, but (2) might be the cause. While the pipeline is running, targets assumes nothing else is modifying the input or output files. If you wrote one of the parquet files while a tar_make() call was running, that could have changed the expected hash for a target that was trying to wrap up its computation.

[gadenbuie]

Okay thanks but I'm not modifying the files during the run, just walking away from my computer and coming back later to find the pipeline failed. That will help me try to recreate a reprex, though.

Is there an error mode that will continue the pipeline through these errors?

[wlandau]

They all apply now: 96d6d85