skip_dir deletes remote data

[dalemartin]

The docs:

Description: This configuration option controls whether the application skips certain directories from being synced. ...

I take that to mean the client will do nothing with any directory matching a skip_dir entry. Instead what I found is that if the directory does not exist locally, it deletes a matching remote directory.

Maybe the intent of the option needs minor clarification in the docs. My intent was to skip a remote directory. If it is meant to skip only local directories from syncing to the remote OneDrive/SharePoint, that is not stated clearly in the docs. I do see a note that it is a 'Client Side Filtering Rule', but I did not understand that to mean one-directional behaviour.

Furthermore, even if the user's intent was to skip syncing a local directory, if there is a typo in the path and it happens to match a remote directory, the remote directory will be deleted. This is, in my humble opinion, a bug, because it is unexpected behaviour for an option named "skip", which implies no action.

I was able to achieve the desired result of skipping a remote directory with an exclude entry in the sync_list. It seems this is the intended avenue.

[abraunegg]

The application option operates as intended - it skips the directories and does not delete them.

What you are articulating if true would be an application bug, that, after ~350 different use cases which includes this option - and 9+ months of testing before v2.5.0 was released - is either a regression or a serious bug that has slipped through the cracks somehow.

Please raise a bug, with all correct evidence for your use of 'skip_dir' so this can be evaluated and verified.

[dalemartin]

I have raised a bug here: #2910

[abraunegg]

@dalemartin
I have specifically just tested this:

skip_dir = "/asdf"

And the directory is correctly excluded and not deleted online.

Please provide way more details and/or raise a bug with all details + create a verbose debug log as per: https://github.com/abraunegg/onedrive/wiki/Generate-debug-log-for-support

[dalemartin]

Thanks @abraunegg, I will do this when I have time.

One nuance I think is that the skip_dir path has to be contained in the directory tree of a path named in sync_list. E.g. in sync_list you have "/asdf", and you specify skip_dir = "/asdf/foo*". Locally it does not match anything, but remote "/asdf/foobar" matches and is deleted. This is just from memory, I'll update you once I have more details.

[dalemartin]

Starting with a fresh setup, I was only able to reproduce it by first syncing, then adding one of the synced directories to skip_dir, then deleting it locally. On the next sync, it deletes the remote instead of skipping the deletion. More details in the bug ticket #2910. Thanks.

[dalemartin]

As a general side note, I noticed something else while trying to reproduce the deletion behaviour: sync_list takes priority over skip_dir if there is a collision. Maybe this is intended.

For example, adding a file to the directory /foo/barbar uploads the file if sync_list has /foo in it, even with skip_dir = "/foo/bar*" in the config file. Maybe that is intended. And it is easily worked around using !/foo/barbar in sync_list instead of a skip_dir entry.

$ echo "fuzz" > barbar/fuzz.txt
$ onedrive -s -v --confdir="~/.config/onedrive/accounts/SharePoint_SDI_Documents"
Reading configuration file: /home/dmartin/.config/onedrive/accounts/SharePoint_SDI_Documents/config
Configuration file successfully loaded
Using 'user' configuration path for application config and state data: /home/dmartin/.config/onedrive/accounts/SharePoint_SDI_Documents
WARNING: D-Bus message bus daemon is not available; GUI notifications are disabled
Using the following path to store the runtime application log: /home/dmartin/.local/sharepoint/sdi
Using IPv4 and IPv6 (if configured) for all network operations
Attempting to contact Microsoft OneDrive Login Service
Successfully reached Microsoft OneDrive Login Service
Checking Application Version ...
Attempting to initialise the OneDrive API ...
Configuring Global Azure AD Endpoints
The OneDrive API was initialised successfully
Opening the item database ...
WARNING: OneDrive quota information is being restricted. Please fix by speaking to your OneDrive / Office 365 Administrator.
Application Version:  onedrive v2.5.2-1+np1+1.1
Account Type:         documentLibrary
Default Drive ID:     *********
Default Root ID:      *********
Remaining Free Space: Not Available
Sync Engine Initialised with new Onedrive API instance
All application operations will be performed in the configured local 'sync_dir' directory: /home/dmartin/SharePoint/SDI_Documents
Fetching /delta response from the OneDrive API for Drive ID: *********
Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 0
Finished processing /delta JSON response from the OneDrive API
No changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive
Performing a database consistency and integrity check on locally stored data
Processing DB entries for this Drive ID: *********
WARNING: OneDrive quota information is being restricted. Please fix by speaking to your OneDrive / Office 365 Administrator.
Processing: /home/dmartin/SharePoint/SDI_Documents
The directory has not changed
Processing: foo
The directory has not changed
Processing: foo/barbar
The directory has not changed
Processing: foo/barbar/bar.txt
The file has not changed
Processing: foo/baz
The directory has not changed
Processing: foo/baz/baz.txt
The file has not changed
Scanning the local file system '/home/dmartin/SharePoint/SDI_Documents' for new data to upload
New items to upload to Microsoft OneDrive: 1
Total New Data to Upload:        5 Bytes
WARNING: OneDrive quota information is being restricted or providing a zero value. Please fix by speaking to your OneDrive / Office 365 Administrator.
Uploading new file: ./foo/barbar/fuzz.txt ... done
WARNING: OneDrive quota information is being restricted or providing a zero value. Please fix by speaking to your OneDrive / Office 365 Administrator.
Performing a last examination of the most recent online data within Microsoft OneDrive to complete the reconciliation process
Fetching /delta response from the OneDrive API for Drive ID: *********
Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 4
Including path - included by sync_list config: ./foo
Including path - included by sync_list config: foo/barbar
Including file - included by sync_list config: foo/barbar/fuzz.txt
Finished processing /delta JSON response from the OneDrive API
Processing 3 applicable changes and items received from Microsoft OneDrive
Processing OneDrive JSON item batch [1/1] to ensure consistent local state
Skipping file - file path is excluded by skip_dir config: foo/barbar/fuzz.txt

Sync with Microsoft OneDrive is complete
Attempting to perform a database vacuum to optimise database
Database vacuum is complete

There are these competing log messages

Including file - included by sync_list config: foo/barbar/fuzz.txt
...
Skipping file - file path is excluded by skip_dir config: foo/barbar/fuzz.txt

...and the behaviour is that the file is actually uploaded.

[abraunegg]

if there is a collision. Maybe this is intended.

All Client Side Filtering rules follow this process:

https://github.com/abraunegg/onedrive/blob/master/docs/usage.md#using-client-side-filtering-rules-to-determine-what-should-be-synced-with-microsoft-onedrive

Checking 'sync_list' comes after all others - so either the path is not being correctly calculated for 'skip_dir' exclusion, or there is no match.