Promtail: stale NFS file handle

[mxsiegle]

Hey,

I'm trying to log files that are located on a mounted NFS volume and noticed that Promtail v2.2.0 was excessively spamming the following log messages over and over:

Apr 22 11:23:07 <hostname> promtail[5430]: level=info ts=2021-04-22T11:23:07.477188057Z caller=tailer.go:173 component=tailer msg="skipping update of position for a file which does not currently exist"

I found PR #3457 and upgraded to Promtail v2.2.1 which indeed gave me more information on the issue:

Apr 22 15:35:58 <hostname> promtail[19673]: level=info ts=2021-04-22T15:35:58.423440228Z caller=tailer.go:141 component=tailer msg="tail routine: tail channel closed, stopping tailer" path=/path/to/file/file.log reason="Error reading /path/to/file/file.log: read /path/to/file/file.log: stale NFS file handle"
Apr 22 15:35:58 <hostname> promtail[19673]: level=info ts=2021-04-22T15:35:58.424899792Z caller=tailer.go:134 component=tailer msg="tail routine: exited" path=/path/to/file/file.log

I understand that this is somewhat not Promtail's fault and more like an OS and NFS 'issue', but maybe there is something that can be done about it regardless, e.g. by closing and re-opening the file when this issue occurs.
Interestingly, collecting logs was still mostly working although the behavior was a bit flaky at times.

Our servers are all running SLES15 and I could observe that behavior randomly on multiple hosts with different NFS servers mounted on each of them, so it doesn't look like a temporary issue or with a specific NFS server.

Any ideas?

At the very least I think the log level should be changed if the issue is repeating. I had 50 files open and it would print 50 messages every 10s (default file refresh) or 150 with Promtail v2.2.1.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.