[Bug]: Data Loss during External Storage Disconnect

[swartzlib7]

⚠️ Before submitting, please verify the following: ⚠️

• This is a bug, not a question or a configuration issue.
• This issue is not already reported on Github (I've searched it).
• Nextcloud Server and Desktop Client are up to date. See Server Maintenance and Release Schedule and Desktop Releases for supported versions.
• I agree to follow Nextcloud's Code of Conduct

Bug description

Data Loss during External Storage Disconnect

Firstly, thank you for the awesome Desktop Sync client and Nextcloud in general. I love it. We do however have a small problem to be addressed rather rapidly.

This issue is fairly serious and hits at confidence in sync client behavior which will cause loss of data locally due to a common technical issue of having your external storage device fail or be turned off without turning while still running your Nextcloud instance.

When the Nextcloud external storage device becomes unavailable, the mount points configured on Nextcloud are unavailable in the back-end with a visible error, but the sync client remains operational - data deletion of all data from the configured local storage will result!

Hardware and Configuration

• NextcloudPi on a Raspberry Pi 4.
• 2TB raid via USB 3 for external storage.
• Nextcloud v 27.0.1.
• Linux sync client v3.10.0 (KDE).

Situation

• My raid storage device didn't start due to a suspected power loss during the last 10 days.
• I was running Nextcloud v26.x.x (latest) at this time and the instance stopped functioning completely after power was restored.
• Usually, I manually run the desktop sync client as needed - I didn't use the sync client until I got the instance back online.
• I upgraded to Nextcloud v27.0.1 which brought my instance back online.
• I was working locally on my synced folder for about a week with the sync client inactive.

Result

• I opened the sync client after Nextcloud v27.0.1 was back online, but the external storage was still offline.
• Data loss resulted when the sync client started. All files and folders from the local storage location was removed.
• Inspection of the Nextcloud External Storage configuration showed warnings present about mounting failures of the external storage device, but these warnings were not taken into account by the sync client.
• After mounting the storage device the sync client is now again syncing data locally.
• Any new files created during the offline period and present in the configured sync folder for external storage has been deleted by the sync client and lost.

Suggestion

• Inspect external storage mounting status before deleting files.
• When deleting files send them to the recycle bin or something similar without simply unlinking files from disc.
• Add a data threshold for deletions e.g. anything over 100 MB of data will require confirmation.

Steps to reproduce

1. Run Nextcloud v27.0.1.
2. Have external storage connected.
3. Run Sync client with external storage folder to local machine configured and let it sync.
4. Stop/Pause sync client.
5. Create new files in local external storage folder (not on storage device).
6. Disconnect external storage from Nextcloud device (unmount and turn it off while leaving Nextcloud server running).
7. Resume sync client and watch all existing and new data disappear without warning.

Expected behavior

• Sync client to prompt for large delete operation.
• Sync client to check external storage mounting status before deleting.
• Move files to a "deleted" folder or something such as recycle bin before just removing them.

Which files are affected by this bug

no_idea.log

Operating system

Linux

Which version of the operating system you are running.

Pop!_OS 22.04 LTS

Package

Appimage

Nextcloud Server version

27.0.1

Nextcloud Desktop Client version

v3.10.0 (KDE)

Is this bug present after an update or on a fresh install?

Fresh desktop client install

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

Are you using an external user-backend?

• Default internal user-backend
• LDAP/ Active Directory
• SSO - SAML
• Other

Nextcloud Server logs

No response

Additional info

No response

[mgallien]

@swartzlib7
I cannot reproduce this with a setup using NFS external storage and versions of server and desktop files client matching your versions.
I will need a debug archive from your client to debug further

[swartzlib7]

Hi Matthieu,

thank you for getting back to my issue here.

I updated my client AppImage since the last issue, but the logs might still be available if this is stored in my home directory.

Can you tell me where I could find it? I'm running linux.

[mgallien]

@swartzlib7 the logs are by default (unless you changed the settings) kept for 24 hours
I would assume you no longer have them.
what kind of external storage are you using ?

[mgallien]

so the logs are by default located in
~/.config/nextcloud/logs

[swartzlib7]

Yeah those logs are long gone. Thank you for helping me with this!
So the drive I use is connected via USB3 and configured as an external storage device.
The external device is a Yottamaster 2 Bay RAID configured for Raid 1.

I suspect the storage device didn't start or come back online after a power failure, I'm not certain which.
What I do know is that after the data loss on my local machine, I proceeded to check the admin backend configuration in Nextcloud. There I noticed that each of my configured External storage path entries had warnings indicating that the path was not mounted / unavailable. I don't have the exact wording, but that was the main point.

[mgallien]

@swartzlib7 I am investigating
the nominal behavior is to skip folders that are from external storage but unmounted at time of sync
I tested with NFS external storage without having been able to reproduce the issue
I was running the same versions you were running
I will try again with other types of external storage just in case

[l0drex]

If I understood correctly, I just ran into the same issue. The desktop client deleted a bunch of files with the message File was deleted (File with name //Festplatte could not be located). //Festplatte would be the mount point of my external storage in Nextcloud.

Here is the debug archive exported from the desktop client (was too big for GitHub). Nextcloud desktop is installed through pacman.

My system information

Operating System: Arch Linux
KDE Plasma Version: 5.27.9
KDE Frameworks Version: 5.111.0
Qt Version: 5.15.11
Kernel Version: 6.6.1-zen1-1-zen (64-bit)
Graphics Platform: X11
Processors: 12 × Intel® Core™ i5-10400F CPU @ 2.90GHz
Memory: 15.5 GiB of RAM
Graphics Processor: NVIDIA GeForce RTX 2080/PCIe/SSE2
Product Name: H570 Phantom Gaming 4

[l0drex]

This is a rather urgent issue, and I would love to see this addressed quickly. If you need anything else, please let me know.

[Byter3]

Bump

Had the same issue twice already.

[swartzlib7]

@swartzlib7 I am investigating the nominal behavior is to skip folders that are from external storage but unmounted at time of sync I tested with NFS external storage without having been able to reproduce the issue I was running the same versions you were running I will try again with other types of external storage just in case

Thank you for this and to everyone else who also confirmed the same thing happening on their clients.

I have since moved away from the NextCloud Pi solution and the Pi4 I used. I am now on different hardward and the NextCloud AIO solution. So far this is far better and also a bit faster due to the new hardware.

That said, I still have this in the back of my mind and would love to see a fix with clear UI / log indication of an event where an external mount goes missing and then to have its files being "orphaned" or left alone in-place rather than kicked out into a "black hole".

What I have done to prevent this is the following workaround:

• Install a fresh Linux (I used Ubuntu latest stable).
• Mount my external drive on this new OS.
• Install a fresh NextCloud AIO (should work on any solution such as NextCloud PI).
• During the installation of NextCloud AIO I installed the master container onto the mounted external drive as allowed by the installation tool.
• I then completed the install and configured all my external drive entries as Local types and their respective paths.

Now with this I know: when my external drive fails or does not mount for any reason, my entire Nextcloud server will be taken down with it and become unavailable to the sync client which will prevent it from going psychotic on me by deleting all my external storage files.

Sincerely,
Stephen

[theroch]

My nextcloud desktop client (3.12.0 windows) also deleted all my local files after the SMB share becomes unavailable for nextcloud (27.1.7) because of a software package update for samba was installed.

I think it is almost not a good behavior of a file client to delete GBs of data, without asking the user.
Please implement a dialog to ask the user for such situations.

[bigretromike]

My nextcloud desktop client (3.12.0 windows) also deleted all my local files after the SMB share becomes unavailable for nextcloud (27.1.7) because of a software package update for samba was installed.

I think it is almost not a good behavior of a file client to delete GBs of data, without asking the user. Please implement a dialog to ask the user for such situations.

was your SMB mount point added as external storage ?
Because there was a fix for something like this some time ago: nextcloud/server#39707 maybe its recursing issue ?

[joshtrichards]

Related: #1984