OneDrive Client Version v2.5.0-alpha-0 is available for testing

[abraunegg]

After a lengthy period of development, OneDrive Client Version v2.5.0-alpha-0 is available for testing.

Your requirements will be, should you wish to help test:

• You MUST use a OneDrive TEST Account
  • OneDrive Personal Accounts are supported - create a new account here: https://www.microsoft.com/en-au/microsoft-365/onedrive/online-cloud-storage
  • OneDrive Business Accounts are supported
• Do not use your normal OneDrive configuration directory (your normal 'sync_dir') - set a 100% different 'sync_dir'
• You MUST be using test data. Please do not use any alpha-application-code with your actual data or data you care about.
• Business Shared Items are not supported at the moment.
  • This is due to the extra development required for Business Shared Folders to change how this was originally implemented to support FR-1251 , FR-1300 , BUG-2379 and BUG-2434
• SharePoint Libraries are hard coded at the moment to be not supported.

You can use your normal account if you wish (this is your choice), however this is 100% alpha quality software at the moment. If you experience data loss or something else with your real data - whilst not good, please take every effort to not use this code with your real data.

My ask is that you create a test report - what works, what does not work, what could be potentially changed | improved - basically I am soliciting some feedback on where this total re-write is at before I go too deep.

There will be functionality differences. There will be functionality that will be initially missing in this alpha-0 version - missing functionality will be added back in as the code progresses from an alpha-0 state to an alpha-XX state (such support for Business accounts and SharePoint, Business Shared Items ), and well before any beta-1 or beta-XX or Release Candidate is published.

I have gone through 90 test cases with this alpha version before pushing this to GitHub - so hopefully there are no major showstoppers - but you never know. Refer to #2415 (comment) as to what I have tested.

How can you get v2.5.0-alpha-0 ?

You need to build this client from source. Follow the high level directions here to get the development requirements installed first: https://github.com/abraunegg/onedrive/blob/master/docs/INSTALL.md#building-from-source---high-level-requirements

Next clone and checkout the PR:

git clone https://github.com/abraunegg/onedrive.git
cd onedrive
git fetch origin pull/2477/head:pr2477
git checkout pr2477
./configure; make clean; make;

To run the PR, you need to run the client from the PR build directory:

./onedrive <any options needed>

The version, when running the PR will be set at:

./onedrive --version
v2.5.0-alpha-0 GitHub version: v2.4.25-XX-abcdefgh

The current application version is:

v2.5.0-alpha-0 GitHub version: v2.4.25-30-g8ac38a2

The XX-abcdefg will change as new commits are put into the PR branch, whilst things are added/fixed/updated

What is known to be not functioning right at this stage:

• Application logging output when downloading & uploading files is jumbled. The reason for this is all the parallel threads writing to terminal at the same time. This is even more evident when downloading or uploading files >4MB. This will not be fixed in alpha-0 but will need to be focused upon for the alpha-1 release. Do not raise an issue ticket because of this.
• Existing documentation does not match what v2.5.0 does or supports at present

I have no test files or test data - what can I use?

In developing this application I use a script as per below to generate random data:

#!/bin/bash
FILECOUNT=0
DIRCOUNT=0
DESTDIR="/my/path/to/my/sync_dir/folder/OneDrive/random_100_files"
FILES=10
FOLDERS=10
BLOCKSIZE=4

mkdir -p $DESTDIR

while [ $DIRCOUNT -lt $FOLDERS ] ; do
        DIRNAME=`cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1`
        DIRPATH=$DESTDIR/$DIRNAME
        mkdir -p $DIRPATH
        while [ $FILECOUNT -lt $FILES ] ; do
                FILESIZE=$RANDOM
                FILESIZE=$(($FILESIZE+1024))
                FILENAME=$DIRPATH/file${FILECOUNT}.data
                echo "Creating $FILENAME"
                dd if=/dev/urandom of=${FILENAME} bs=$BLOCKSIZE count=$FILESIZE > /dev/null 2>&1
                ((FILECOUNT++))
        done
        FILECOUNT=0
        ((DIRCOUNT++))
done

Update DESTDIR with the right directory and adjust FILES and FOLDERS as desired. The above settings will create 10 folders, with each folder having 10 files. with each file size between 6KB and 125KB.

To increase (or decrease) the random file size, change BLOCKSIZE, generally use ^2 values: 1,2,4,8,16,32,64,128,256 and so on.

At 128 you will get files from 100KB -> ~3.8MB, at 256 most files will be between 500KB --> 7.5MB

[bpozdena]

I've tested the monitor mode with my typical daily tasks and I did not encounter any major issues. Bellow are some of my initial observations:

1. The above instructions are missing command ./configure; make clean; make;. It might not be obvious to newbies like me.
2. When config file contains empty application ID application_id="", the login URL for application authorization does not contains a newly generated ID. Older versions generated new app id in this case.

Authorise this application by visiting:
https://login.microsoftonline.com/common/oauth2/v2.0/authorize?client_id=&scope=Files.ReadWrite%20Files.ReadWrite.All%20Sites.ReadWrite.All%20offline_access&response_type=code&prompt=login&redirect_uri=https://login.microsoftonline.com/common/oauth2/nativeclient

3. OneDrive crashes when sync_dir configured in the config file does not exist.

ERROR: Unable to perform a database vacuum: out of memory

std.file.FileException@/usr/include/dlang/dmd/std/file.d(2896): /home/bob/OneDrive_boris@personal.commmm: No such file or directory
----------------
/usr/include/dlang/dmd/std/file.d:283 @trusted bool std.file.cenforce!(bool).cenforce(bool, scope const(char)[], scope const(char)*, immutable(char)[], ulong) [0x55a1c3798120]
/usr/include/dlang/dmd/std/file.d:272 @safe void std.file.chdir!(immutable(char)[]).chdir(immutable(char)[]) [0x55a1c3798013]
src/main.d:328 _Dmain [0x55a1c3763cd4]

4. When rate limit is configured, the console is being flooded with the bellow messages during sync operation.

User Configured Rate Limit: 125000000
User Configured Rate Limit: 125000000
User Configured Rate Limit: 125000000
User Configured Rate Limit: 125000000
[...]

5. It might still be expected, but multi-threading only seems to work for upload operation? I did not notice any performance boost during download and delete operations.

Tested with onedrive source code directory:
**v2.4**
upload - 60sec
download - 38sec
delete - 22sec

**v2.5.0-alfa0**
upload - 30sec
download - 35sec
delete - 22sec

6. REQUEST: Even if you can't un-jumble the output from different threads in the next alpha, it would be great if each stdout message with the progress bar also contained the file name. This would allow me to parse it by the OneDriveGUI.

EXAMPLE:
Downloading  10% |oooo                 |   ETA   00:03:09  | test/ubuntu-budgie-22.04.1-desktop-amd64.iso ... 

[abraunegg]

@bpozdena
Thanks for testing - greatly appreciated

• Point 2,3 is a bug - will fix
• Point 4 is a logging output issue - will fix
• Point 5 - Multi-threading is for uploading and downloading ... potentially the directory is too small to determine any major speed differences, but there are 16 threads spun up to download 16 items at a time - so there should be a massive difference. Delete is an odd one as the online delete just deletes the root object (unless using business accounts with a certain flag enabled where a full online tree walk is needed)

I will look at points 2,3,4 next - I have found some quirk with running in extended monitor mode & sitting idle that does not work right that I am trying to chase down at present.

[abraunegg]

@bpozdena
FYI - the updated PR has item 2 & 4 fixed, others pending

[undefiened]

Thank you very much for the update!
I will be posting bugs (if there is more than one) as separate comments so that it is easier to follow them.

--dry-run parameter doesn't seem to work for uploading files

Older version (onedrive v2.4.25-1+np1):

onedrive --synchronize --upload-only --dry-run

DRY-RUN Configured. Output below shows what 'would' have occurred.
Configuring Global Azure AD Endpoints
Initializing the Synchronization Engine ...
Syncing changes from local path only - NOT syncing data changes from OneDrive ...
Uploading differences of ~/OneDrive
Uploading new items of ~/OneDrive
Uploading new file ./upload/filename1.zip ... done.
Uploading new file ./upload/filename2.zip ... done.
Sync with OneDrive is complete

This alpha:

./onedrive --sync --upload-only --dry-run


DRY-RUN Configured. Output below shows what 'would' have occurred.
DRY-RUN: Copying items.sqlite3 to items-dryrun.sqlite3 to use for dry run operations
DRY RUN: Not creating backup config file as --dry-run has been used
DRY RUN: Not updating hash files as --dry-run has been used
Configuring Global Azure AD Endpoints
The item database is incompatible, re-creating database table structures
Sync Engine Initialised with new Onedrive API instance
Performing a database consistency and integrity check on locally stored data ... 
Scanning local filesystem '~/OneDrive' for new data to upload ...

Sync with Microsoft OneDrive is complete
DRY-RUN: Removing items-dryrun.sqlite3 as it still exists for some reason
DRY-RUN: Removing items-dryrun.sqlite3-shm as it still exists for some reason
DRY-RUN: Removing items-dryrun.sqlite3-wal as it still exists for some reason

[undefiened]

For me ./onedrive --version prints

onedrive v2.5.0-alpha-0

I am currently on commit 760b3c8

The --dry-run still seems to not work for some reason.

New version

./onedrive --sync --upload-only --dry-run 

DRY-RUN Configured. Output below shows what 'would' have occurred.
DRY-RUN: Copying items.sqlite3 to items-dryrun.sqlite3 to use for dry run operations
DRY RUN: Not creating backup config file as --dry-run has been used
DRY RUN: Not updating hash files as --dry-run has been used
Configuring Global Azure AD Endpoints
Sync Engine Initialised with new Onedrive API instance
Performing a database consistency and integrity check on locally stored data ... 
Deleting item from OneDrive: upload
Scanning local filesystem '~/OneDrive' for new data to upload ...

Sync with Microsoft OneDrive is complete
DRY-RUN: Removing items-dryrun.sqlite3 as it still exists for some reason
DRY-RUN: Removing items-dryrun.sqlite3-shm as it still exists for some reason
DRY-RUN: Removing items-dryrun.sqlite3-wal as it still exists for some reason

Old version:

onedrive --synchronize --upload-only --dry-run

DRY-RUN Configured. Output below shows what 'would' have occurred.
Configuring Global Azure AD Endpoints
The item database is incompatible, re-creating database table structures
Initializing the Synchronization Engine ...
Syncing changes from local path only - NOT syncing data changes from OneDrive ...
Uploading differences of ~/OneDrive
Uploading new items of ~/OneDrive
Uploading new file ./backup/some_filename.zip ... done.
Sync with OneDrive is complete

[abraunegg]

Please 'update' your v2.5.0 version ...

[undefiened]

Sorry, I am a bit confused, I did git pull and rebuilt the app. Should I use some other branch?

[abraunegg]

You may need to delete your local pull directory and start fresh as you are using an old commit

[undefiened]

OK, got it, thanks. I thought I could just git pull.

Now --dry-run looks to be working fine with uploading.

[undefiened]

I cannot get the errors you mentioned in #2457, not sure if it is an output issue or a bug. From #2457 it looks like the following code should output some error text in the end. I imitated network issues by reconnecting to my VPN in the middle of uploading, so the network was off for a second.

./onedrive --sync --upload-only


Configuring Global Azure AD Endpoints
Sync Engine Initialised with new Onedrive API instance
Performing a database consistency and integrity check on locally stored data ... 
Deleting item from OneDrive: Attachments
Deleting item from OneDrive: Documents
Scanning local filesystem '~/OneDrive' for new data to upload ...
New items to upload to OneDrive: 6
Uploading  16% |ooooooo                                 |   ETA   00:00:00 Uploading new file ./upload/filename6.zip ... done.                                                                            
Uploading new file ./upload/filename5.zip ... done.
ERROR: OneDrive returned an error with the following message:
  Error Message: Failure when receiving data from the peer on handle 55E2A2475F70
  Calling Function: performHTTPOperation()

Uploading new file ./upload/filename4.zip ... done.
ERROR: OneDrive returned an error with the following message:
  Error Message: Failure when receiving data from the peer on handle 7F81341C5070
  Calling Function: performHTTPOperation()

Uploading new file ./upload/filename3.zip ... done.
ERROR: OneDrive returned an error with the following message:
  Error Message: Failure when receiving data from the peer on handle 7F81401C5050
  Calling Function: performHTTPOperation()

Uploading new file ./upload/filename2.zip ... done.
ERROR: OneDrive returned an error with the following message:
  Error Message: Failure when receiving data from the peer on handle 7F81441BFFE0
  Calling Function: performHTTPOperation()

Uploading new file ./upload/filename.zip ... done.

Sync with Microsoft OneDrive is complete

Otherwise, I think it looks like it works fine (aside from output issues). Thank you again for your hard work!

[undefiened]

OK, I will do that. Sorry, I didn't know about --verbose, thanks for the tip.

[abraunegg]

You can also add --verbose --verbose to get debug logging .. even more detail as to what is going on

[abraunegg]

FYI, with the current PR, - I get this sort of output in --monitor mode:

Successfully created the remote directory ./random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB on OneDrive
New items to upload to OneDrive: 10
Uploading new file ./random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB/file7.data ... done.
Uploading new file ./random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB/file5.data ... done.
Uploading new file ./random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB/file6.data ... done.
Uploading new file ./random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB/file8.data ... done.
Uploading new file ./random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB/file9.data ... done.
Uploading new file ./random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB/file2.data ... done.
Uploading new file ./random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB/file3.data ... done.
Uploading new file ./random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB/file1.data ... done.
Uploading new file ./random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB/file4.data ... done.
Uploading new file ./random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB/file0.data ... done.
Failed to upload: random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB/file3.data
Failed to upload: random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB/file0.data
Failed to upload: random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB/file7.data
Failed to upload: random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB/file1.data
Failed to upload: random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB/file6.data
Failed to upload: random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB/file4.data
Failed to upload: random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB/file9.data
Failed to upload: random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB/file2.data
Failed to upload: random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB/file8.data
Failed to upload: random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB/file5.data

Sync with Microsoft OneDrive has completed, however there are items that failed to sync.
To fix any upload failures you may need run the application again to ensure this system is correctly synced with your Microsoft OneDrive Account
Total Run-Time Loop Number:     16
Full Scan Freqency Loop Number: 4
Application run-time thus far: 2 hours, 29 secs, 200 ms, 171 μs, and 2 hnsecs
Starting a sync with Microsoft OneDrive
Syncing changes from OneDrive ...
Failed to upload: random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB/file3.data
Failed to upload: random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB/file0.data
Failed to upload: random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB/file7.data
Failed to upload: random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB/file1.data
Failed to upload: random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB/file6.data
Failed to upload: random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB/file4.data
Failed to upload: random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB/file9.data
Failed to upload: random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB/file2.data
Failed to upload: random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB/file8.data
Failed to upload: random_100_files/GbWT9PaG0XH9GCUlpnnGGe1Nx6U6qKMB/file5.data

Sync with Microsoft OneDrive has completed, however there are items that failed to sync.
To fix any upload failures you may need run the application again to ensure this system is correctly synced with your Microsoft OneDrive Account
Total Run-Time Loop Number:     17
Full Scan Freqency Loop Number: 5
Application run-time thus far: 2 hours, 9 minutes, 15 secs, 738 ms, 310 μs, and 6 hnsecs
Starting a sync with Microsoft OneDrive
Syncing changes from OneDrive ...

The creation of the directory failed, thus, correct error .. but it should have 'fixed' itself next sync ... which it did not.

[undefiened]

After testing it on the latest alpha, it fails with an error if I reconnect to VPN during uploading. I think the result isn't supposed to be like that, but I also realize that what I do to test isn't the very best idea. Either way, it generates some scary error text, which is good.

./onedrive --sync --upload-only

Configuring Global Azure AD Endpoints
Sync Engine Initialised with new Onedrive API instance
Performing a database consistency and integrity check on locally stored data ... 
Deleting item from OneDrive: upload
Scanning local filesystem '~/OneDrive' for new data to upload ...
New items to upload to OneDrive: 42
Uploading   7% |oooo                                    |   ETA   00:00:00                                                                                                                                          
ERROR: There was a timeout in accessing the Microsoft OneDrive service - Internet connectivity issue?
  - Check DNS resolution or Firewall Rules
ERROR: OneDrive returned an error with the following message:
  Error Message: Failure when receiving data from the peer on handle 55F083FAACA0
  Calling Function: performHTTPOperation()

Uploading new file ./folder/filename.zip ... done.
Internet connectivity to Microsoft OneDrive service has been restored
Invalid response from the OneDrive API. Unable to initialise OneDrive API instance.
fish: Job 1, './onedrive --sync --upload-only' terminated by signal SIGSEGV (Address boundary error)

I also have a log from a --verbose --verbose run, where I edited out what I perceived can be personal info. I also omitted the whole beginning part because there are plenty of codes and so on which will be a pain to edit out. Please let me know if it is needed.

New items to upload to OneDrive: 43
Total New Data to Upload:        17825 MB
[DEBUG] Current Available Space Online (Account Drive ID): 5120 MB
[DEBUG] Upload Thread 0 Starting: 2023-Sep-03 17:16:42.8570956
[DEBUG] fileToUpload: ./folder/filename_10.zip
[DEBUG] read token from appConfig
[DEBUG] Authorised State: true
[DEBUG] Updating appConfig.refreshToken with updated refreshToken
[DEBUG] Updating refreshToken on disk
[DEBUG] Setting file permissions for: /home/undefiened/.config/onedrive/refresh_token
[DEBUG] Request URL = https://graph.microsoft.com/v1.0/drives/some_code/?select=quota
[DEBUG] Current Available Space Online (Upload Target Drive ID): 5120 MB
[DEBUG] read token from appConfig
[DEBUG] Authorised State: true
[DEBUG] Updating appConfig.refreshToken with updated refreshToken
[DEBUG] Updating refreshToken on disk
[DEBUG] Setting file permissions for: /home/undefiened/.config/onedrive/refresh_token
[DEBUG] Request URL = https://graph.microsoft.com/v1.0/drives/some_code/root:/.%2Fbfolder%2Ffilename_10.zip
Uploading  50% |oooooooooooooooooooo                    |   ETA   00:00:00 [DEBUG] fileDetailsFromOneDrive = checkFileOneDriveApiInstance.getPathDetailsByDriveId(parentItem.driveId, fileToUpload); generated a 404 - file does not exist online - must upload it
[DEBUG] read token from appConfig
[DEBUG] Authorised State: true
Uploading   7% |oooo                                    |   ETA   00:00:00                                                                                                                                          
ERROR: There was a timeout in accessing the Microsoft OneDrive service - Internet connectivity issue?
  - Check DNS resolution or Firewall Rules
[DEBUG] onedrive.performHTTPOperation() Generated a OneDrive CurlException
[DEBUG] Unable to resolve server - DNS access blocked?
[DEBUG] Configuring libcurl to use a fresh connection for re-try

ERROR: There was a timeout in accessing the Microsoft OneDrive service - Internet connectivity issue?
  - Check DNS resolution or Firewall Rules
  Retry attempt:          1
  This attempt timestamp: 2023-Sep-03 17:16:53
  Next retry in approx:   1 seconds
  Next retry approx:      2023-Sep-03 17:16:54
ERROR: OneDrive returned an error with the following message:
  Error Message: Failure when receiving data from the peer on handle 7FECFC1BF3D0
  Calling Function: performHTTPOperation()

Uploading new file ./folder/filename_5.tar.gz ... done.
ERROR: OneDrive returned an error with the following message:
  Error Message: Failure when receiving data from the peer on handle 7FECF41BFAC0
  Calling Function: performHTTPOperation()

Uploading new file ./folder/filename_4.zip ... done.
ERROR: OneDrive returned an error with the following message:
  Error Message: Failure when receiving data from the peer on handle 7FED081C6530
  Calling Function: performHTTPOperation()

Uploading new file ./folder/filename_3.tar.gz ... done.
ERROR: OneDrive returned an error with the following message:
  Error Message: Failure when receiving data from the peer on handle 7FED1C1C0B80
  Calling Function: performHTTPOperation()

Uploading new file ./folder/filename_2.zip ... done.
ERROR: OneDrive returned an error with the following message:
  Error Message: SSL connect error on handle 7FED0C1C01A0
  Calling Function: performHTTPOperation()

Uploading new file ./folder/filename_1.zip ... done.
[DEBUG] Configuring libcurl to use a fresh connection for re-try
Uploading  33% |oooooooooooooo                          |   ETA   00:00:00 Internet connectivity to Microsoft OneDrive service has been restored                                                                    
[DEBUG] Unsetting libcurl to use a fresh connection as this causes a performance impact if left enabled
Invalid response from the OneDrive API. Unable to initialise OneDrive API instance.
fish: Job 1, './onedrive --sync --upload-only…' terminated by signal SIGSEGV (Address boundary error)

OneDrive version is v2.5.0-alpha-0 GitHub version: v2.4.25-20-g54b3d52

[undefiened]

I also can repeat it on the newer build v2.5.0-alpha-0 GitHub version: v2.4.25-21-gc659cb2 although it is more difficult, it now takes reconnecting VPN at the right time (before the first file starts uploading) or a couple of attempts. Otherwise, it just continues uploading.

The file has not changed
Scanning local filesystem '~/OneDrive' for new data to upload ...
New items to upload to OneDrive: 40
Total New Data to Upload:        17779 MB
Uploading 100% |oooooooooooooooooooooooooooooooooooooooo| DONE IN 00:00:01                                                                                                                                          
Uploading new file ./folder/filename_10.tar.gz ... done.
Uploading   3% |oo                                      |   ETA   00:04:36 Internet connectivity to Microsoft OneDrive service has been restored                                                                    
Uploading new file ./folder/filename_something.zip ... failed.
Internet connectivity to Microsoft OneDrive service has been restored
Invalid response from the OneDrive API. Unable to initialise OneDrive API instance.
fish: Job 1, './onedrive --sync --upload-only…' terminated by signal SIGSEGV (Address boundary error)

[abraunegg]

For anyone testing the PR, please can you validate that you are testing the following version:

v2.5.0-alpha-0 GitHub version: v2.4.25-21-gc659cb2

If you are - could you please help me test the following 'config' option:

force_children_scan = "true"

This option is used (by default) in the following circumstances in v2.5.x:

• When using --single-directory (new use to avoid horrid extensive processing that v2.4.x and prior performed)
• When using a National Cloud Deployment

The change to use this method (which is selected by default if using those 2 circumstances) is a massive change for --single-directory use - and helps quicken up the process to scan OneDrive for any objects that have changed.

Whilst this was initially validated when going through 90 different test cases, there has been some significant updates to how error handling is being done, thus it would be great to get some assistance in flushing out problems. I think I have it sorted - but it would be great to get some external validation here as well.

[abraunegg]

@bpozdena

Issue 2: When config file contains empty application ID, auth URL is incorrect
Issue 3: OneDrive crashes when sync_dir configured in the config file does not exist
Issue 4: When rate limit is configured, the console is being flooded with User Configured Rate Limit: messages

FYI these identified issues should now be resolved with v2.5.0-alpha-0 GitHub version: v2.4.25-24-g4756680 or above

If you could re-pull / retest and/or continue testing that would be greatly appreciated.

[abraunegg]

To all following this thread / progress of v2.5.0-alpha-0:

With the latest version of 'alpha-0' it has been updated to support 'personal' and 'business' accounts, however Business Shared Folders and SharePoint is still a work in progress.

If you are interested in testing this against your 'business' account, please create a 'test' business account to proceed.

The version where 'business' account support has been enabled is: v2.5.0-alpha-0 GitHub version: v2.4.25-30-g8ac38a2 as per the following:

Reading configuration file: /home/alex/.config/onedrive/config
Invalid value for key in config file - using default value: application_id
Configuration file successfully loaded
Using 'user' Config Dir: /home/alex/.config/onedrive
Checking Application Version ...
Attempting to initialise the OneDrive API ...
Configuring Global Azure AD Endpoints
The OneDrive API was initialised successfully
Opening the item database ...
Sync Engine Initialised with new Onedrive API instance
Application version:  v2.5.0-alpha-0 GitHub version: v2.4.25-30-g8ac38a2
Account Type:         business
Default Drive ID:     b!bO8V7s9SSk6r7mWHpIjURotN33W1W2tEv3OXV_oFIdQimEdOHR-1So7CqeT1MfHA
Default Root ID:      01WIXGO5V6Y2GOVW7725BZO354PWSELRRZ
Remaining Free Space: 1073566084 KB
All application operations will be performed in: /home/alex/OneDrive
OneDrive syncronisation interval (seconds): 300
Maximum allowed open files:                 805297
Maximum allowed inotify watches:            60100
Initialising filesystem inotify monitoring ...
Monitoring directory: .
Monitoring directory: ./random_files
Monitoring directory: ./random_files/e0ALAQfUQzSzivoCIMcnRown4hAQWDQ7
Monitoring directory: ./random_files/9ekEVWhsLxDqFdUt5OuAKc3y1bpC1lVZ
Performing initial syncronisation to ensure consistent local state ...
Total Run-Time Loop Number:     1
Full Scan Freqency Loop Number: 1
Application run-time thus far: 1 sec, 12 ms, 937 μs, and 4 hnsecs
Starting a sync with Microsoft OneDrive
Fetching items from the OneDrive API for Drive ID: b!bO8V7s9SSk6r7mWHpIjURotN33W1W2tEv3OXV_oFIdQimEdOHR-1So7CqeT1MfHA ..
Performing a database consistency and integrity check on locally stored data ... 
Processing DB entries for this Drive ID: b!bO8V7s9SSk6r7mWHpIjURotN33W1W2tEv3OXV_oFIdQimEdOHR-1So7CqeT1MfHA
Processing ~/OneDrive
The directory has not changed
Processing random_files
The directory has not changed
Processing random_files/e0ALAQfUQzSzivoCIMcnRown4hAQWDQ7
The directory has not changed
Processing random_files/e0ALAQfUQzSzivoCIMcnRown4hAQWDQ7/file0.data
The last modified timestamp has changed however the file content has not changed
The local item has the same hash value as the item online - correcting timestamp online
Processing random_files/e0ALAQfUQzSzivoCIMcnRown4hAQWDQ7/file1.data
The last modified timestamp has changed however the file content has not changed
The local item has the same hash value as the item online - correcting timestamp online
Processing random_files/9ekEVWhsLxDqFdUt5OuAKc3y1bpC1lVZ
The directory has not changed
Processing random_files/9ekEVWhsLxDqFdUt5OuAKc3y1bpC1lVZ/file1.data
The last modified timestamp has changed however the file content has not changed
The local item has the same hash value as the item online - correcting timestamp online
Processing random_files/9ekEVWhsLxDqFdUt5OuAKc3y1bpC1lVZ/file0.data
The last modified timestamp has changed however the file content has not changed
The local item has the same hash value as the item online - correcting timestamp online
Scanning local filesystem '~/OneDrive' for new data to upload ...
Skipping item - excluded by sync_list config: ./random_25k_files
Perfoming final true up scan of online data from OneDrive
Fetching items from the OneDrive API for Drive ID: b!bO8V7s9SSk6r7mWHpIjURotN33W1W2tEv3OXV_oFIdQimEdOHR-1So7CqeT1MfHA ..
Processing OneDrive JSON item batch [1/1] to ensure consistent local state
Sync with Microsoft OneDrive is complete

[abraunegg]

Performance Update with OneDrive Business Account

Started testing uploading 10K randomly generated files small, 20 folders each with 500 files.

The upload started approx. 2pm and completed at 5.05pm local - so approx. 2hrs and 5 mins to perform this type of activity. This is a major performance shift as compared to v2.4.25 where this sort of operation would last >24hrs.

[abraunegg]

OneDrive Client Version v2.5.0-alpha-1 is now available for testing - refer to: #2495