Unexpected behavior of --cleanup-local-files #2499

Grimmwicht · 2023-09-25T10:36:50Z

Grimmwicht
Sep 25, 2023

Hey folks,

I struggle to understand the behavior of this client.

First of all: I want to achieve a pure mirror from online state to my local drive. Changes happen online by many people, and I want this mirrored locally. Today, I created some new files and dropped them online via web interface, then moved some old files into an Archiv folder.

After that, I executed locally this command:
onedrive --synchronize --download-only --cleanup-local-files

Now this is what happens:

Configuration file successfully loaded
Configuring Global Azure AD Endpoints
User Configured Rate Limit: 125000000
Initializing the Synchronization Engine ...
WARNING: Application has been configured to cleanup local files that are not present online.
WARNING: Local data loss MAY occur in this scenario if you are expecting data to remain archived locally.
Syncing changes and items from OneDrive ...
Syncing this OneDrive Business Shared Folder: Kinderladen Schildkröte
Downloading file Kinderladen Schildkröte/00_Leitung Kinderladen/Dienstpläne/2023/DP 2023.xlsx ... done.
Downloading file Kinderladen Schildkröte/15_Haus und Hof/Schlüsselausgabe/2023-09-25 Schlüsselausgabeblatt.odt ... done.
Downloading file Kinderladen Schildkröte/15_Haus und Hof/Schlüsselausgabe/2023-09-25 Schlüsselausgabeblatt.pdf ... done.
Moving Kinderladen Schildkröte/15_Haus und Hof/Schlüsselausgabe/2013-03-27 Schlüsselordnung.doc to Kinderladen Schildkröte/15_Haus und Hof/Schlüsselausgabe/Archiv/2013-03-27 Schlüsselordnung.doc
Moving Kinderladen Schildkröte/15_Haus und Hof/Schlüsselausgabe/2020-11-20 Schlüsselausgabeblatt.odt to Kinderladen Schildkröte/15_Haus und Hof/Schlüsselausgabe/Archiv/2020-11-20 Schlüsselausgabeblatt.odt
Moving Kinderladen Schildkröte/15_Haus und Hof/Schlüsselausgabe/2020-11-20 Schlüsselausgabeblatt.pdf to Kinderladen Schildkröte/15_Haus und Hof/Schlüsselausgabe/Archiv/2020-11-20 Schlüsselausgabeblatt.pdf
Downloading file Kinderladen Schildkröte/51_AG Orga/Projekte/Dokumente/Dokumente Begruessungsordner/Unterlagen Teil 1 einfach/Willkommensbrief.odt ... done.
Searching local filesystem for extra files and folders which need to be removed
Removing local file as --download-only & --cleanup-local-files configured
Removing local file: ./Kinderladen Schildkröte/15_Haus und Hof/Schlüsselausgabe/Archiv/2013-03-27 Schlüsselordnung.doc
Removing local file as --download-only & --cleanup-local-files configured
Removing local file: ./Kinderladen Schildkröte/15_Haus und Hof/Schlüsselausgabe/Archiv/2020-11-20 Schlüsselausgabeblatt.pdf
Removing local file as --download-only & --cleanup-local-files configured
Removing local file: ./Kinderladen Schildkröte/15_Haus und Hof/Schlüsselausgabe/Archiv/2020-11-20 Schlüsselausgabeblatt.odt
Sync with OneDrive is complete

It correctly sees the changes, downloads the new files and moves the files that I moved online. But then it deletes the three files, it moved just a moment before. Do I miss something fundamental here? I'm using onedrive v2.4.25 on Manjaro on an SSD with ext4 filesystem.

Thanks in advance for any enlightenment.

Answered by abraunegg

Sep 25, 2023

@Grimmwicht
Unsure why this occurring here, but, that is potentially a bug in v2.4.25 code ... this being said, this issue does not exist in v2.5.x, and similar logic is used, however v2.5.x is a complete re-write:

Before online move:

./onedrive -s -v --download-only
Reading configuration file: /home/alex/.config/onedrive/config
Configuration file successfully loaded
Using 'user' Config Dir: /home/alex/.config/onedrive
Checking Application Version ...
Attempting to initialise the OneDrive API ...
Configuring Global Azure AD Endpoints
The OneDrive API was initialised successfully
Opening the item database ...
Sync Engine Initialised with new Onedrive API instance
Application version:  v2.5…

View full answer

abraunegg · 2023-09-25T19:30:04Z

abraunegg
Sep 25, 2023
Maintainer

@Grimmwicht
Unsure why this occurring here, but, that is potentially a bug in v2.4.25 code ... this being said, this issue does not exist in v2.5.x, and similar logic is used, however v2.5.x is a complete re-write:

Before online move:

./onedrive -s -v --download-only
Reading configuration file: /home/alex/.config/onedrive/config
Configuration file successfully loaded
Using 'user' Config Dir: /home/alex/.config/onedrive
Checking Application Version ...
Attempting to initialise the OneDrive API ...
Configuring Global Azure AD Endpoints
The OneDrive API was initialised successfully
Opening the item database ...
Sync Engine Initialised with new Onedrive API instance
Application version:  v2.5.0-alpha-1 GitHub version: v2.5.0-alpha-1
Account Type:         business
Default Drive ID:     b!bO8V7s9SSk6r7mWHpIjURotN33W1W2tEv3OXV_oFIdQimEdOHR-1So7CqeT1MfHA
Default Root ID:      01WIXGO5V6Y2GOVW7725BZO354PWSELRRZ
Remaining Free Space: 1058488127 KB
All application operations will be performed in: /home/alex/OneDrive
Fetching items from the OneDrive API for Drive ID: b!bO8V7s9SSk6r7mWHpIjURotN33W1W2tEv3OXV_oFIdQimEdOHR-1So7CqeT1MfHA ..
Performing a database consistency and integrity check on locally stored data ... 
Processing DB entries for this Drive ID: b!bO8V7s9SSk6r7mWHpIjURotN33W1W2tEv3OXV_oFIdQimEdOHR-1So7CqeT1MfHA
Processing ~/OneDrive
The directory has not changed
Processing Attachments
The directory has not changed
Processing Documents
The directory has not changed
Processing Documents/important.docx
The file has not changed
Processing Documents/debug_output.log
The file has not changed

Sync with Microsoft OneDrive is complete

Move file from 'Documents' to 'Attachments' online, then sync again:

./onedrive -s -v --download-only --cleanup-local-files
Reading configuration file: /home/alex/.config/onedrive/config
Configuration file successfully loaded
Using 'user' Config Dir: /home/alex/.config/onedrive
Checking Application Version ...
Attempting to initialise the OneDrive API ...
Configuring Global Azure AD Endpoints
The OneDrive API was initialised successfully
Opening the item database ...
WARNING: Application has been configured to cleanup local files that are not present online.
WARNING: Local data loss MAY occur in this scenario if you are expecting data to remain archived locally.
Sync Engine Initialised with new Onedrive API instance
Application version:  v2.5.0-alpha-1 GitHub version: v2.5.0-alpha-1
Account Type:         business
Default Drive ID:     b!bO8V7s9SSk6r7mWHpIjURotN33W1W2tEv3OXV_oFIdQimEdOHR-1So7CqeT1MfHA
Default Root ID:      01WIXGO5V6Y2GOVW7725BZO354PWSELRRZ
Remaining Free Space: 1058488127 KB
All application operations will be performed in: /home/alex/OneDrive
Generating a /delta compatible JSON response from the OneDrive API ...
Adding OneDrive folder details for processing
Adding 2 OneDrive items for processing from the OneDrive 'root' folder
Adding 1 OneDrive items for processing from Attachments
Adding 1 OneDrive items for processing from Documents
Finished processing self generated /delta JSON response from the OneDrive API
Processing OneDrive JSON item batch [1/1] to ensure consistent local state
Moving Documents/important.docx to Attachments/important.docx
Performing a database consistency and integrity check on locally stored data ... 
Processing DB entries for this Drive ID: b!bO8V7s9SSk6r7mWHpIjURotN33W1W2tEv3OXV_oFIdQimEdOHR-1So7CqeT1MfHA
Processing ~/OneDrive
The directory has not changed
Processing Attachments
The directory has not changed
Processing Attachments/important.docx
The file has not changed
Processing Documents
The directory has not changed
Processing Documents/debug_output.log
The file has not changed
Scanning local filesystem '~/OneDrive' for data to cleanup ...
Skipping item - excluded by sync_list config: ./random_25k_files

Sync with Microsoft OneDrive is complete

0 replies

tdcockers · 2023-10-05T01:43:28Z

tdcockers
Oct 5, 2023

Flagging another data point - I'm having the same issue with Sharepoint folder sync on 2.4.25. I switched to using
onedrive --monitor --download-only --cleanup-local-files
thinking it would resolve an issue I was having where moving/renaming things on Sharepoint resulted in both the original and the duplicate being kept on the client, then being pushed back to sharepoint so there was then 2 copies of the file on sharepoint.

I think this behaviour was probably introduced recently, I updated the client a couple of weeks ago but have only become aware of the issue recently... I think I was on 2.4.23 previously but not sure.

1 reply

abraunegg Oct 5, 2023
Maintainer

@tdcockers

I think this behaviour was probably introduced recently, I updated the client a couple of weeks ago but have only become aware of the issue recently... I think I was on 2.4.23 previously but not sure.

Can advise here that this is 100% not the case. Going back many releases there are no direct bug fixes or changes that impact or influence --cleanup-local-files - so this is either a new bug or a previously unsurfaced issue.

Given that this has also been 100% regression tested against v2.5.0 code (as per above) - would you be willing to review that version?

I am on the cusp of pushing v2.5.0-alpha-2 that:

Is 96%+ feature parity with v2.4.25
- Lacking webhooks
- Lacking any logging fixes
Complete application test against Personal, Business and SharePoint 'drive_id' use (I have been literally for the last week running through all the validation testing for SharePoint)

tdcockers · 2023-10-05T04:23:00Z

tdcockers
Oct 5, 2023

Given that this has also been 100% regression tested against v2.5.0 code (as per above) - would you be willing to review that version?

I've only got live data to work with, but it's backed up nightly so should be OK to test on one of the low impact libraries

6 replies

tdcockers Oct 5, 2023

Which pr number should I be pulling to switch to the code you're using?

abraunegg Oct 5, 2023
Maintainer

I will be posting a new PR specific for v2.5.0-alpha-2 in the next few days. I want to finish updating some of the documentation first.

I could load the PR, but the doco wont be complete for all the updated functions | how to use.

tdcockers Oct 5, 2023

OK thanks. The onedrive client is just taking offline backups of files... I'll just have to remember that there are some things not being backed up offline for the moment.

abraunegg Oct 5, 2023
Maintainer

@tdcockers
I have posted a PR for v2.5.0-alpha-2

Please refer to #2506 for details.

Please be mindful of the documentation / usage differences.

tdcockers Oct 5, 2023

Thanks, testing now, see other thread. Getting errors about inotify watches which in my limited understanding of them might explain some of the issues?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unexpected behavior of --cleanup-local-files #2499

{{title}}

Replies: 3 comments 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Unexpected behavior of --cleanup-local-files #2499

Grimmwicht Sep 25, 2023

Replies: 3 comments · 7 replies

abraunegg Sep 25, 2023 Maintainer

tdcockers Oct 5, 2023

abraunegg Oct 5, 2023 Maintainer

tdcockers Oct 5, 2023

tdcockers Oct 5, 2023

abraunegg Oct 5, 2023 Maintainer

tdcockers Oct 5, 2023

abraunegg Oct 5, 2023 Maintainer

tdcockers Oct 5, 2023

Grimmwicht
Sep 25, 2023

Replies: 3 comments 7 replies

abraunegg
Sep 25, 2023
Maintainer

tdcockers
Oct 5, 2023

abraunegg Oct 5, 2023
Maintainer

tdcockers
Oct 5, 2023

abraunegg Oct 5, 2023
Maintainer

abraunegg Oct 5, 2023
Maintainer