[Bug]: v3.8.0 basically KILLS E2EE ("Error with the metadata. Getting unexpected metadata format.")

[bcutter]

⚠️ Before submitting, please verify the following: ⚠️

• This is a bug, not a question or a configuration issue.
• This issue is not already reported on Github (I've searched it).
• Nextcloud Server and Desktop Client are up to date. See Server Maintenance and Release Schedule and Desktop Releases for supported versions.
• I agree to follow Nextcloud's Code of Conduct

Bug description

After updating to desktop client v3.8.0 I started to see strange sync issues. Encrypted changed and created files could not be synced with the server anymore.

• I thought "well, just that time of the year E2EE had its hiccups, just start over" and spent almost 4 hours in resetting E2EE.
• In the end it did fix nothing.

With v3.8.0 there are metadata issues. Clients give Error with the metadata. Getting unexpected metadata format..

Steps to reproduce

1. Start fresh with E2EE for a user by resetting E2EE (meta-data, private key, public key, potential file lock entries, corresponding filecache entries, delete all previously existing E2EE folders, run occ files:scan-app-data etc.)
2. Desktop client with 3.8.0: enable E2EE by setting a passphrase
3. Another client (I used the iOS app): clear E2EE from settings, supply same (new) passphrase as set in step 2
4. Desktop client: create a new empty folder and encrypt it, put one file in it
5. iOS client: try to access the encrypted folder

Note: you can also create the folder on the iOS client and try to access it from the desktop client, same result.

Expected behavior

Access is possible as usual.

Which files are affected by this bug

All E2EEncrypted!

Operating system

Windows

Which version of the operating system you are running.

Windows 10

Package

Appimage

Nextcloud Server version

25.0.5.1

Nextcloud Desktop Client version

3.8.0

Is this bug present after an update or on a fresh install?

Updated to a major version (ex. 3.3.6 to 3.4.0)

Are you using the Nextcloud Server Encryption module?

Encryption is Enabled

Are you using an external user-backend?

• Default internal user-backend
• LDAP/ Active Directory
• SSO - SAML
• Other

Nextcloud Server logs

This seems to be the initial issue (not fully sure):

{"reqId":"zCH6yCfRIeEVQuTq6QP7","level":3,"time":"2023-04-03T20:34:34+02:00","remoteAddr":"xxx.xxx.xxx.xxx","user":"Username","app":"no app in context","method":"DELETE","url":"/ocs/v2.php/apps/end_to_end_encryption/api/v1/lock/871525","message":"Intermediate meta-data file missing","userAgent":"Mozilla/5.0 (Windows) mirall/3.8.0stable-Win64 (build 20230331) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)","version":"25.0.5.1","exception":{"Exception":"OCA\\EndToEndEncryption\\Exceptions\\MissingMetaDataException","Message":"Intermediate meta-data file missing","Code":0,"Trace":[{"file":"/var/www/nextcloud/apps/end_to_end_encryption/lib/Controller/LockingController.php","line":133,"function":"saveIntermediateFile","class":"OCA\\EndToEndEncryption\\MetaDataStorage","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":225,"function":"unlockFolder","class":"OCA\\EndToEndEncryption\\Controller\\LockingController","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":133,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/App.php","line":172,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/Route/Router.php","line":298,"function":"main","class":"OC\\AppFramework\\App","type":"::"},{"file":"/var/www/nextcloud/ocs/v1.php","line":63,"function":"match","class":"OC\\Route\\Router","type":"->"},{"file":"/var/www/nextcloud/ocs/v2.php","line":23,"args":["/var/www/nextcloud/ocs/v1.php"],"function":"require_once"}],"File":"/var/www/nextcloud/apps/end_to_end_encryption/lib/MetaDataStorage.php","Line":174,"CustomMessage":"--"}}

Nothing in logs for the client messages "Error with the metadata. Getting unexpected metadata format." !!!

Additional info

You should maybe have a look at what you did in e. g. (quick screening for E2E relevant changes)

• skip e2e encrypted files with empty filename in metadata #5448
• E2EE cut extra zeroes from derypted byte array. #5534
• Feature/e2ee fixes #5560

I downgraded to 3.7.4 and everything is working as expected/normal! No more sync issues or error messages. Just a bit of data loss and loss of 4 hours... thanks.

Currently in rage mode and not willing to upgrade to 3.8.0 again to provide desktop client logs. Sorry.

[mgallien]

@bcutter sorry to hear that
can you share a debug archive ?
I can provide a secure share link for you to share it privately with us (and yes we have to keep this secret via a signed paper)

[VanHell1638]

Hi,
@mgallien i can provide the needed debug archive. Please send me the secure share link.

[mgallien]

@VanHell1638 I will delay this as I have found an issue in the end-to-end encryption implementation of the desktop client that is not conform to the specification
I will work on a fix and will see if I can find a workaround to allow recovery from the bug
very sorry for that

[mgallien]

just as a note, I did fix an issue with a missing part in the e2ee implementation of desktop client in 3.8.0
#5570
release is a work in progress #5569

[mgallien]

not sure how you can recover
at least if the files are still correct and accessible on your client, doing any modification may be enough
I will look into this after the release is done

[bcutter]

TBH after almost 5 lost hours yesterday I'm still not (yet) in the needed mood so providing debugs... better refer to @VanHell1638.

Just as note: I fixed this FOR ME by doing a full E2EE reset, downgrading desktop client (and blocking auto update on all other machines to stay at 3.7.4 for now) and re-syncing all E2EE content. Probably just downgrading would have been enough and the 3 to 4 hours of E2EE reset party was oversized.

I will test any new desktop client version claiming to have this fixed of course.

[mgallien]

#5572 should enable automatic recovery from broken checksums

[mgallien]

both fix will be in 3.8.1 release

[bcutter]

I will test 3.8.1 once it is released.

[bcutter]

Just a small note (not sure if it's worth posting @ https://github.com/nextcloud/end_to_end_encryption/issues) since updating the E2EE app from 1.11.1 to 1.11.3 yesterday:

1. iPhone updated encrypted folder content (created new files)
2. Desktop Client with v3.7.4 is in endless sync queue ("waiting", but never finishes)
3. Disabling E2EE on the 3.7.4 desktop client made it sync other queued changes (outside of encrypted folders)
4. Reenabling E2EE on the 3.7.4 desktop client starts syncing the encrypted folder content, but stucks and never finishes.
   

Seems like 3.7.4 desktop client and 1.11.3 server app version don't work very well. Hopefully 1.11.3 will with 3.8.1... 🙄

Trouble continues. Had to disable E2EE on that client (and probably all other clients running the desktop client) to make the non-encrypted rest sync. Argh...

When will 3.8.1 gonna be released? Otherwise I would need to look into how to downgrade a server app (e2ee 1.11.3 to 1.11.1)...

[bcutter]

How to downgrade end_to_end_encryption server app? I'm going crazy here with stuck sync of really really REALLY important files ⚠

[bcutter]

Downgrading the server app back to 1.11.1 (theory: desktop clients back to 3.7.4, app back to 1.11.1 - combination where everything was simply WORKING) changed nothing. Still stuck sync:

initial sync after E2EE enabled on desktop client:

restarted sync later:

Had to upgrade server app back to 1.11.3 as iOS client refused changes (e. g. deleting E2EE files) with error message "Error - The server version of the end to end encryption is not compatible with this client". I guess some meta data generated with a newer server app version might cause this.

It's a complete mess, nothing is working anymore. I'm so upset. Especially cause it's like a déjà vu to what we saw back a few years ago when E2EE was originally marketed as production ready even it was (and is) technically still in beta according to the seriousness of still existing issues:

This is the marketing https://nextcloud.com/blog/desktop-3-8-end-to-end-encryption-levels-up-with-sharing-and-file-drop/, this #5564 is what's really going on. Just inacceptable, I should refuse and delay NC client updates for a few weeks in future, like proven for the server part. My mistake doing the update.

Need to cool down and hope some day someone will help on the 3.8.1 question (nightly, beta, hotfix, stable release) so I can stop praying I can finally sync again.

[bcutter]

After spending another 2 hours I am giving up now. Meanwhile desktop client only syncs encrypted files and complains about missing encryption information.

Congrats, you guys just really screwed up things. Out of abilites from a user perspective.

Wasted enough of my time. You'll fix it for sure. Just ping me once you wake up and have some news on this.

[ckaspro]

Awesome, just ran into the same trap. Even worse - a few days ago my laptop/ssd with the encrypted source files died - no access anymore.

Honestly: why is the 3.8.0 release still online? Can you imagine how many people are loosing (or not having access to - what is it?!) their files right now?

Can you please tell us:

• which files are safe, which have to be seen as lost
• when will the 3.8.1 release be available, and is there anything to consider while upgrading

[thoniGH]

I ran into the same issue a 6 days ago (Apr-08). My observations might help:

• After sync failure white spaces in secret are caught
• Android client still has access to E2E encrypted files, but filenames turned into hexcodes names
• (Quite sure) just before the issue raised there was a log "renamed 'SecureFolder' to 'SecureFolder'" (same name)

[Jardo-51]

Also ran into this issue when upgrading desktop client from 3.7.4 to 3.8.0. Downgrading back to 3.7.4 didn't help.

I know this is free software and I appreciate it. But I also find it hard to comprehend why version 3.8.0 is still online with such a serious bug and why haven't you released the hotfix version when you had the problem fixed for 2 weeks already.

[ghost]

Same from my end. The error has been known for 2 weeks and according to the information here on Github and has already been solved. Where is the hot fix? As an alternative to Microsoft products, an urgent bug fix should not wait 2 weeks and a faulty version should still not still available for download. The quality assurance should be checked again at this point.

[mgallien]

please test again with the latest release
it should fix the bugs in the current release and recover from broken metadata automatically
https://github.com/nextcloud/desktop/releases/tag/v3.8.1

[bcutter]

@bcutter my understanding is that this is now resolved in the 3.9.0 latest release so far I would like to close it to not clutter the open issues

@mgallien it looked like it was fixed, correct. Unfortunately, after a longer time I experienced a follow-up issue probably of the fixes: folders deleted on the endpoint with NC desktop client are not deleted on the server with E2EE enabled. Please urgently have a look at #5918. Thank you.

[bcutter]

This bug report did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!

@mgallien see last post, thx.

[bcutter]

Well since this issue, #5918 exists which has been closed based on false assumptions by @mgallien .

Would love to see every mess being cleaned up instead of being quickly closed for no reason.

Reopen and READ #5918 ! Thanks.

[rklasen]

I was affected by this too. I enabled E2EE for two folders only (the most important ones naturally), and ever since I accessed them with the mobile app sometime around April or May I can't access them anymore.

When they get downloaded to the PC, the folder names are some hex values instead of the real file names.

Seeing how this is supposedly fixed now, is there any way I can recover my lost folders? I have the 12 word mnemonic of course.

EDIT:

This is what the client looks like for me. The mnemonic is set up, but all filesnames are messed up, most of the folders are empty and files are missing from the folders that aren't empty.

[grosshau]

I am affected by this too. Any updates planed on this in the near future?

[Phrow]

After spending another 2 hours I am giving up now. Meanwhile desktop client only syncs encrypted files and complains about missing encryption information.

I have the very same issue with the desktop client on windows 10

I'm using desktop client version 3.13.0 and the server is a managed nextcloud offer (v28.0.3, end_to_end_encryption: 1.14.3), so I don't really have raw file access to try some offline decryption.
Interestingly I'm also using the iOS app version 5.3.0.21, which is showing me error messages like "Unable to decode the metadata file", but not in the same folders in which I have sync issues on my desktop. So, on my desktop I can still see some files, which are inaccessible on the phone and vice versa - but unfortunately I never get the whole set of all my data together.
Any ideas how this can be fixed? I tried a reinstall of the desktop app as well as disabling and re-enabling e2e using the mnemonic, but nothing changed. I'm still seeing the same errors in the very same folders with the very same files.

[gravelfreeman]

I'm having this issue as well. I'm running a 3 weeks brand new Nextcloud install. The Nextcloud desktop client version is 3.13.0 which I believe is the latest version.

Step that lead to this error;

1. Installed E2EE application in webui
2. Restarted Windows client so that I can see the encryption option
3. Setup encryption in Windows client
4. Created a folder, right click and selected encryption
5. I copied 5 test video files and they all encrypted successfully
6. Then I added all my video files (roughly 55 GB)

This is where the client started throwing the errors.

I can't really share my logs because there is so much personal information in them which would take a lot of time to anonymize.

If anyone got a fix to get the sync to complete. Right now, no matter what I'm trying the sync fails and Nextcloud is useless.

[MichaelL93]

Same for me now. Added a new file to the E2E folder, restarted my mac -> missing metadata error. Re-sync, remove the folder & re-add it, nothing currently resolves it. nextcloud server version 28.0.6. Client 3.13.0. e2e version 1.14.5

[mainboarder]

Now I run into that issue.
Server 29.0.4
Client 3.13
E2EE 1.15.2

Am I questioning everything, because of this highly active, but still unresolved thread? - Yes.

Beside this error, the sync seems to work. A file changed on client1, generating this error, is in the expected state on client2.

[mainboarder]

I was able to solve the conflict.

(Enter docker container)
Scan for files in the path with problem
Restart/Refresh Clients

root@server:/PATCH/TO/NC/data/USER/files/dir1/dir2# docker exec -u 33 -ti a2c3049d9167 /bin/bash
www-data@22ab8669e9db:~/html$ ./occ files:scan       
Please specify the user id to scan, --all to scan for all users or --path=...
www-data@22ab8669e9db:~/html$ ./occ files:scan --path USER/files/dir1/dir2
Starting scan for user 1 out of 1 (USER)
+---------+-------+-----+---------+---------+--------+--------------+
| Folders | Files | New | Updated | Removed | Errors | Elapsed time |
+---------+-------+-----+---------+---------+--------+--------------+
| 6       | 35    | 0   | 1       | 1       | 0      | 00:00:00     |
+---------+-------+-----+---------+---------+--------+--------------+

[executed]

This error affects me too.

occ files:scan didn't help - why would it? :)

Server: 29.0.5.
Windows client: 3.13.3

UPD: I think I was able to fix this... We will see how it goes.
Ping me to post the solution if you have problems with the PC client and not the app client + you have an unencrypted backup of problematic data + Docker installation (could be a showstopper if not Docker).