OneDrive Client Version v2.5.0 - Development Update

[abraunegg]

To those who are wondering how the development of v2.5.0 is tracking, this discussion item will be used to track where things are at.

The main focus of v2.5.0 is to implement #232 - that is - uploading | downloading files in parallel and undertake an entire re-write of how the application operates for overall performance improvement.

What is complete:

• Download items from OneDrive in parallel
• Perform integrity scan of data stored in local cache database & detect changed items to upload
• Perform local filesystem walk to determine new files to upload
• Upload new local items to OneDrive in parallel
• Client side filtering (sync_list, skip_file, skip_dir) is currently working
• Support OneDrive Personal Shared Folders is currently working
• The use of --dry-run to assist in simulating what would happen without it actually happening
• The use of --monitor

Refer to below post that details things that have been tested.

What is missing at this stage:

1. Overall feature parity with v2.4.25 | master code base
2. Support OneDrive Business Accounts
3. Support OneDrive SharePoint
4. Support OneDrive Business Shared Items (Folders and Files)

In terms of local testing, I am testing with a 4.7GB file dataset comprising of ~25K files and folders. I am excluding some files and folders via sync_list, skip_file and skip_dir. My testing / use-case validation is not going to cover everything how the client is being used today.

What is known to be not functioning right at this stage:

• Application logging output when downloading & uploading files is jumbled. The reason for this is all the parallel threads writing to terminal at the same time. This is even more evident when downloading or uploading files >4MB. This will not be fixed in alpha-0 but will need to be focused upon for the alpha-1 release.

Application Performance (without any files being uploaded | downloaded):

Current v2.5.0 code (alpha-0):

~1min 43 seconds for a --resync operation

Reading configuration file: /home/alex/.config/onedrive/config
Configuration file successfully loaded
Deleting the saved application sync status ...
Configuring Global Azure AD Endpoints
Sync Engine Initialised with new Onedrive API instance
Getting changes and JSON items from the OneDrive API for Drive ID: 66d53be8a5056eca ................................................................................................................................
Processing changes and items received from OneDrive ...
Syncing this OneDrive Personal Shared Folder: 3A
Getting changes and JSON items from the OneDrive API for Drive ID: bc7d88ec1f539dcf ..
Processing changes and items received from OneDrive ...
Performing a database consistency and integrity check on locally stored data ... 
Scanning local filesystem '~/OneDrive' for new data to upload ...
Sync with Microsoft OneDrive is complete.

real    1m43.751s
user    0m4.450s
sys     0m0.330s

Current v2.4.25 code compare:

~4min 59 seconds for a --resync operation

Configuration file successfully loaded
Deleting the saved application sync status ...
Configuring Global Azure AD Endpoints
Initializing the Synchronization Engine ...
Syncing changes and items from OneDrive ...
WARNING: OneDrive API inconsistency - this file does not have any hash: OneNote_Items/my notes
WARNING: OneDrive API inconsistency - this file does not have any hash: OneNote_Items/asdfasdfasdf
WARNING: OneDrive API inconsistency - this file does not have any hash: OneNote_Items/srqwerasdfasdf
WARNING: OneDrive API inconsistency - this file does not have any hash: OneNote_Items/ewqrazsdvfaxcvx
WARNING: OneDrive API inconsistency - this file does not have any hash: 66D53BE8A5056ECA!283323
WARNING: OneDrive API inconsistency - this file does not have any hash: 66D53BE8A5056ECA!283325
WARNING: OneDrive API inconsistency - this file does not have any hash: 66D53BE8A5056ECA!283326
WARNING: OneDrive API inconsistency - this file does not have any hash: 66D53BE8A5056ECA!283328
WARNING: OneDrive API inconsistency - this file does not have any hash: 66D53BE8A5056ECA!283330
WARNING: OneDrive API inconsistency - this file does not have any hash: 66D53BE8A5056ECA!283331
WARNING: OneDrive API inconsistency - this file does not have any hash: 66D53BE8A5056ECA!283333
WARNING: OneDrive API inconsistency - this file does not have any hash: 66D53BE8A5056ECA!283335
WARNING: OneDrive API inconsistency - this file does not have any hash: 66D53BE8A5056ECA!283339
WARNING: OneDrive API inconsistency - this file does not have any hash: 66D53BE8A5056ECA!283337
Syncing this OneDrive Personal Shared Folder: 3A
Performing a database consistency and integrity check on locally stored data ... 
Uploading differences of ~/OneDrive
Uploading new items of ~/OneDrive
Syncing this OneDrive Personal Shared Folder: 3A
Sync with OneDrive is complete

real    4m59.041s
user    0m12.762s
sys     0m0.614s

I may publish a preview | alpha of this code to allow folk who have a Test Personal Account to provide some early feedback on performance if there is enough interest to do so.

There will be bugs, there will be usage differences, there will be things not working.

[abraunegg]

Update 1/8/2023:
To those who are watching this thread, I am probably ~4 weeks away from having a solid 'pre-alpha' PR ready for folk to start testing.

Your requirements will be, should you wish to help test:

• You must be using a TEST OneDrive Personal Account - create a new account here: https://www.microsoft.com/en-au/microsoft-365/onedrive/online-cloud-storage
  • Do not use your normal OneDrive Account
  • Do not use your normal OneDrive configuration directory | or your normal local data
• You MUST be using test data. Please do not use any alpha-application-code with your actual data or data you care about.
• Business & SharePoint accounts are hard coded at the moment to be not supported.
  • This is due to the extra development required for Business Shared Folders to change how this was originally implemented to support FR-1251 , FR-1300 , BUG-2379 and BUG-2434

My ask is that you create a test report - what works, what does not work, what could be potentially changed | improved - basically I am soliciting some feedback on where this total re-write is at before I go too deep.

There will be functionality differences. There will be functionality that will be initially missing in this pre-alpha version - missing functionality will be added back in as the code progresses from an alpha-0 state to an alpha-XX state (such support for Business accounts and SharePoint, Business Shared Items ), and well before any beta-1 or beta-XX or Release Candidate is published.

[abraunegg]

Current v2.5.0 Use Case Testing Results - OneDrive Personal Account + OneDrive Personal Shared Folders

The table below will be updated as testing progresses.

If there are any obvious items missing below that you would like to see / ensure that this is tested early on, please comment below.

Case #	Use Case	Result
1	Use of --display-config displays the application configuration and exits	OK
2	Reads 'config' file from default user .config folder	OK
3	Reads 'config' from user entry when using --confdir	OK
4	Validates entries in 'config' file to flush out invalid configuration entries such as invalid 'skip_file' entries	OK
5	Modify the 'config' file changing the 'sync_dir' option - this should trigger a --resync	OK
6	Modify the 'config' file changing the 'skip_file' option - this should trigger a --resync	OK
7	Modify the 'config' file changing the 'skip_dir' option - this should trigger a --resync	OK
8	Modify the 'config' file changing the 'drive_id' option - this should trigger a --resync	OK
9	Modify the 'config' file changing the 'skip_dotfiles' option - this should trigger a --resync	OK
10	Modify the 'config' file changing the 'skip_symlinks' option - this should trigger a --resync	OK
11	Add a  'sync_list' file for the first time - this should trigger a --resync	OK
12	Modify the 'sync_list' file - this should trigger a --resync	OK
13	Files that match 'skip_file' are skipped	OK
14	Directories that match 'skip_dir' are skipped	OK
15	Inclusions and Exclusions as per 'sync_list' are adhered to	OK
16	Check for operational conflicts: Valid permissions are 000 -> 777 - anything else is invalid	OK
17	Check for operational conflicts: --upload-only and --download-only cannot be used together	OK
18	Check for operational conflicts: --sync and --monitor cannot be used together	OK
19	Check for operational conflicts: --no-remote-delete can ONLY be enabled when --upload-only is used	OK
20	Check for operational conflicts: --cleanup-local-files can ONLY be enabled when --download-only is used	OK
21	Check for operational conflicts: --list-shared-folders cannot be used with --resync and/or --resync-auth	OK
22	Check for operational conflicts: --display-sync-status cannot be used with --resync and/or --resync-auth	OK
23	Check for operational conflicts: --modified-by cannot be used with --resync and/or --resync-auth	OK
24	Check for operational conflicts: --get-file-link cannot be used with --resync and/or --resync-auth	OK
25	Check for operational conflicts: --create-share-link cannot be used with --resync and/or --resync-auth	OK
26	Check for operational conflicts: --get-O365-drive-id cannot be used with --resync and/or --resync-auth	OK
27	Check for operational conflicts: --monitor and --download-only cannot be used together	OK
28	Create file online, application downloads file locally	OK
29	Renamed file online, application renames file locally	OK
30	Create directory online, application creates directory locally	OK
31	Move directory online to another directory, on sync local data is moved to correct location	OK
32	Move file to another online directory, on sync local data is moved to correct location	OK
33	Move tree of folders & files online, on sync local data is moved to correct location	OK
34	Move tree of folders & files online including new files & folders that have never been synced before	OK
35	Delete file online, file is deleted locally	OK
36	Delete directory online, directory is deleted locally	OK
37	Directory and all sub directories are deleted online, directory tree locally is deleted	OK
38	Download many files at the same time (max 16 concurrent threads)	OK
39	Upload to OneDrive via web interface many files & folders and download at the same time (max 16 concurrent threads)	OK
40	Create directories locally and upload to OneDrive	OK
41	Create many small files (1 byte) and upload at the same time to OneDrive (max 16 concurrent threads)	OK
42	Create many random sized files and upload at the same time to OneDrive (max 16 concurrent threads)	OK
43	Use 'simple upload' to upload files < 4MB	OK
44	Use 'session upload' to upload files > 4MB	OK
45	Modify a local file < 4MB and upload to OneDrive, replacing online file, retaining online versions	OK
46	Modify a local file > 4 MB and upload to OneDrive, replacing online file, retaining online versions	OK
47	Rename a local file and have the file renamed online (file will be deleted, and uploaded as 'new')	OK
48	Rename a local directory and have the directory renamed online (folder will be deleted and uploaded as 'new')	OK
49	Delete a local file and the file will be deleted online	OK
50	Delete a local directory and entire tree will be deleted online	OK
51	When the upload of a file fails, details of uploaded file failures articulated to user before application exits	OK
52	When the download of a file fails, details of attempted download file failures articulated to user before application exits	OK
53	Attempt to delete a large number of files (>1000 default) application should halt - classify_as_big_delete	OK
54	Attempt to delete a large number of files (>1000 default) application should halt - classify_as_big_delete - use --force to proceed	OK
55	Validate that --upload-only operates as expected, only uploading changes as required to OneDrive including data deletes	OK
56	Validate that --upload-only --no-remote-delete operates as expected, uploading changes, but keeps data online that has been deleted locally	OK
57	Validate that --download-only operates as expected, only downloading data from OneDrive to Local	OK
58	Validate that --download-only --cleanup-local-files operates as expected, downloading data from OneDrive and cleaning up (deleting) local data that no longer exists on OneDrive (delete file online)	OK
59	Validate that --download-only --cleanup-local-files operates as expected, downloading data from OneDrive and cleaning up (deleting) local data that no longer exists on OneDrive (extra file locally)	OK
60	Validate that, if the 'sync_dir' is a mounted item, use 'check_nomount' to check for the presence of a '.nosync' file and if found, abort initial application startup to protect data	OK
61	Validate that --single-directory works as intended	OK
62	Validate that --single-directory works as intended with --upload-only	OK
63	Validate that --single-directory works as intended with --download-only	OK
64	Validate that --single-directory works as intended with --resync	OK
65	Validate that --single-directory performs the correct scan of data online, pre-excluding any 'sync_list' or 'skip_dir' items, quickening sync process dramatically in this scenario	OK
66	Validate that --single-directory works as intended with --monitor	OK
67	Validate that OneDrive Personal Shared Folders work as intended for standard --sync operations	OK
68	Validate that OneDrive Personal Shared Folders work as intended for standard --sync operations when using 'force_children_scan'	OK
69	Validate that OneDrive Personal Shared Folders work: Upload new data in local folder and the new data is uploaded to the Remote Shared Folder Location	OK
70	Validate that OneDrive Personal Shared Folders work: Delete data in local folder and the new data is deleted from the Remote Shared Folder Location	OK
71	Validate that OneDrive Personal Shared Folders work as intended for standard --sync --upload-only operations	OK
72	Validate that OneDrive Personal Shared Folders work as intended for standard --sync --download-only operations	OK
73	Validate that OneDrive Personal Shared Folders work as intended for standard --single-directory operations	OK
74	Validate that OneDrive Personal Shared Folders work as intended for standard --single-directory --upload-only operations	OK
75	Validate that OneDrive Personal Shared Folders work as intended for standard --single-directory --download-only operations	OK
76	Validate that on a --resync any locally modified file, that is newer, is renamed locally, then uploaded as a new file to preserve the local file and to prevent local data loss	OK
77	Validate that POSIX case sensitivity is checked where possible, especially with --single-directory use cases as OneDrive considers the names 'OSCAR', 'Oscar', and 'oscar' to be the same, but on Linux filesystems they are different	OK
78	Validate that, if a remote delete is performed, and that the delete is in a local directory that is excluded by 'sync_list' or 'skip_dir' or 'skip_file' client side filtering rules, the delete operation is not performed	OK
79	Validate that the cleanup of the local directory if --download-only & --cleanup-local-files is configured	OK
80	Validate that --logout works as intended	OK
81	Validate that --reauth works as intended	OK
82	Validate that --auth-files works as intended	OK
83	Validate that --dry-run works as intended, for all operations identified above	OK
84	Validate that --monitor works as intended - add new local folder, folder is created on OneDrive	OK
85	Validate that --monitor works as intended - add new local file, file is uploaded to OneDrive	OK
86	Validate that --monitor works as intended - delete local folder, folder (and any children) are deleted online	OK
87	Validate that --monitor works as intended - move a local file|folder to a new name locally, within the 'account default drive id' and the change is replicated online	OK
88	Validate that --monitor works as intended - move a local file|folder to a new name locally, to a folder locally that is actually a 'shared' folder online, and the change is replicated online	OK
89	Validate that, throughout all operations CTRL-C can be used to stop the application, and, appropriate cleanup & database entries are correctly committed	OK
90	Validate that exit scopes work as intended	OK

[abraunegg]

For those following this thread:

• --dry-run testing against all scenario's is almost complete
• Yet to test --monitor

As such, I am probably at least 1-2 weeks 24 hours away from making v2.5.0-alpha-0 publicly available.