Weird problem with most recent CalyxOS OTA (March) - claims attempt to downgrade security patch and abort #51
Comments
|
Okay I just re-ran it against OTA that I have previously successfully installed link to ota https://release.calyxinstitute.org/redfin-ota_update-24503000.zip and it says Verifying OTA signature... No indication of a "2024-01-05" patchlevel anyplace anywhere (the patchlevel the Custota now, for some terrifying reason, expects) Custota keeps saying Bad Format Exception - downgrading to older security patch is not allowed and refusing these updates I really don't know how I ... screwed up this bad. Fatal breakage? Do I have any chance of fixing this ? |
|
I have managed to extract two old logs (check and install log) from back when it successfully loaded the OTA, as well as current, terrifying failure check log where it complains about (seemingly totally non-existent?!) 2024-01-05 patch level Attaching them |
|
Can you post the output of I took a look at the OTA you linked and I'm seeing (I'm guessing the old date is because Google no longer supports the device and CalyxOS doesn't increment the security patch date when only the OS is updated, but the drivers aren't.) |
|
Oh dammit, I see the problem. Thanks for posting a link to the OTA you're currently running. CalyxOS is lying about the security patch level. Custota queries for |
|
ro.build.version.security_patch I don't have anything that should be faking properties (there's LSposed and Sui, but Lsposed loads one little tweak that has nothing to do with props and was last updated in 2022 and sui is there mostly for appops stuff, so nothing should be "imagining" a patch level. |
Thank you very much. |
|
Oh, so THAT is why my update process for my Calyx rooted phones is broken |
|
Yep, they're indeed ignoring Please give #52 a try. Test build: Custota-4.0.r7.gc43e3a6-release.zip |
|
Thanks! Will try to run it as soon as I get back to my build box. BTW, I reckon I should manually disable/uninstall "normal" custota before trying this test build? Also, any risk of explosions (how worried should I be about first siphoning off 20 GB+ data off my phone ;-) ? |
|
Nope, you should be able to just directly flash it. It will replace the release build you have installed. My test builds are signed with the same keys as the release builds. There shouldn't be any risk of explosions. If there is, it'd be In all seriousness, it's really hard for any Custota to cause any damage. Even if Custota incorrectly allows an invalid OTA to be flashed, |
|
Hi @chenxiaolong I have some good news and some bad news. Now for the bad news: something about the update has initially sort of soft-locked the phone (permanent "phone is starting" screen) which I resolved by booting into safe mode (which disables most things but preserves root and ADB debug and I was so wise and clever as to set adb debug on before update yay me) and wipe all magisk modules. That allowed to boot into normal mode, complete the update configuration (it still initially jumped to "phone starting" screen but this time it had a working progress bar and went away in like two seconds) and proceed with fiddling What works - call recorder's magisk module and the Sui Zygisk module for the fat app-ops manager I'm using What died
It installs, but the app can't be opened (immediate crash, "keeps stopping" message) Re-installing (remove - reboot - install - reboot) does not help. Using older versions doesn't seem to help but I didn't try them all. I caught a logcat tho, hope it contains some useful information . Logcat starts before first attempt to open custota app and continues through several attempts to open it (multiple crashes). Could you please take a look at it ? |
|
Glad to hear the update worked! I'm sad, but not surprised, that LSPosed no longer works correctly. With how Google changed its development model recently, the quarterly releases are closer to full Android releases. Android 14 QPR2 (March update) pulled in many changes from the upcoming Android 15. For Custota, it looks like the root cause is this: Android is blocking Custota from calling a method that allows it to connect to Can you run this in a root shell and check that the permissions on Custota's file is the same as all the other files? ls -lZ /system/etc/sysconfig/Also, can you upload EDIT: Also, if you happen to use Magisk's deny list (or other root hiding things), which apps is it enabled for? |
|
Okay here it goes: redfin:/ # ls -lZ /system/etc/sysconfig/ redfin:/ # cat /data/local/tmp/custota_selinux.log Denylist or other hiding measures are currently disabled and were not enabled during (or after) the update |
|
Hmm, I'm baffled then. That all looks good. Can you check one last thing? Whether the config file is visible to the cat /proc/$(pidof system_server)/root/system/etc/sysconfig/config-com.chiller3.custota.xmlIf that prints out the file contents, then I have no idea why the config file isn't being respected. Enabling access to hidden APIs globally might work as a workaround: https://developer.android.com/guide/app-compatibility/restrictions-non-sdk-interfaces#how_can_i_enable_access_to_non-sdk_interfaces |
Some devices seem to have trouble respecting the `hidden-api-whitelist` option in the sysconfig file. When this happens, show a useful notification instead of crashing hard. Issue: #51 Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
|
Okay I have same behavior and here's a fun one - I uninstalled Custota fully via magisk menu and tried to install the 3.1 version. Mysterious |
Hey, I just tried this and it only works if I do it as root. If non-root then no cigar (permission denied when run in adb without first calling su) OH one more thing - the Custota apk is not in Magisk root list and does not at any point try to ask for magisk root permission. Does that help? |
|
Here are some wild photos (taken by another phone) First, installing custota 3.1 after full uninstall
Then, after reboot, check the version according-to-magisk
then, after that, check out what the system's app info tool says about the app version
Somehow, the 4.0-rc version persists ! Does magisk have some cursed cache squirreled away somewhere ? |
|
I suspect I have a bearing on what's going on. I have a multi-user config AND calyxos, so I have user0 with work profile and I have user 10 with work profile @oddgirl you same ? |
|
I think @nad0vs might be on to something. I wonder if CalyxOS is (re)installing Custota's system app as a user app. After uninstalling the module, run this to list all users (which includes both multi-user and work profiles): husky:/ $ pm list users
Users:
UserInfo{0:Owner:4c13} running
# ^ the user IDFor each user, you can run: pm path --user <user ID> com.chiller3.custotato get the path of the APK if it's installed. If it is, it can be uninstalled for that user by running: pm uninstall --user <user ID> com.chiller3.custotaAfter it's gone from every user, Android should delete the APK.
This is expected. Custota use a script that runs during boot to configure SELinux, but that's the thing that ever uses root. The app only requires system app permissions, which is accomplished by the Magisk module putting files into the right places (eg. |
|
Okay, I've tried the above and also ran pm uninstall --user [number] com.chiller3.custota Needless to say when I re-install (and I tried reinstalling version 2.5 for crying out loud) nothing good happens and the system weirdly still reports the RC version It's like it's cached - somewhere. |
|
Hmm, wtf. I've never seen Android have an app show up, but not be recognized by
Sure. Here's a build from the latest commit in the |
|
One more observation (I'll download and install the new build right after) - every time I re-install the module (no matter the version) it always gets the same base UID Packages: Is that normal for app to always get same UID upon re-installs ? I was under impression that it is assigned randomly no ? |
|
Ooookay this is interesting, I flashed the above release candidate and it
and
I am about 70% positive it is cached somewhere, somehow. Would it be safe to run pm uninstall --user x when the module is still installed ? |
|
Done the pm unistall --user [number] thing while the module was installed, then re-installed the entire thing and no good news exact same behaviors and exact same weird report in system app picker ("stuck" on r7 version) |
|
A trial balloon - would it be very hard to make a custom "custota-test" module with a "custota-test" name and new identifiers and signing keys and everything ? I realize you probably would not rename the entire project to just help out one weird stuck edge case (which can get re-stuck later) but making an app and module with different "fingerprints" may help diagnose what ghostly hell is going on here, I suspect |
If nothing else was uninstalled around that time, then this is pretty normal. If I remember correctly, Android does not use random IDs. It starts sequentially from 10000 and picks the first available unused number.
Darn, yeah, it's definitely not picking up the new apk for some reason then.
Yes, that's always safe to run. Worst case is you lose Custota's settings. Out of curiosity, does the second line of |
|
I get 1970s too despite the system time settings being correct. |
|
OK, give me a few minutes to port the workaround from my other project to Custota. There's a chance it might fix the issue. |
Some devices don't set the system time properly during boot, causing Android's package manager to fail to invalidate cached package info. Work around this by forcibly deleting the relevant cache files. Issue: #51 See: chenxiaolong/BCR#323 Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
|
Please give this a try and upload If it did anything, I'd expect to see something like this in the log: |
|
Haven't run the app yet but redfin:/ # cat /data/local/tmp/custota_selinux.log Timestamp is still 1970s tho. WTF. Anyway will try launching the app now |
|
oooookay the app launched and didn't show any warnings. Any way to positively established we defeated the weird evil problem ? |
|
Nice! Yep, try the "check for updates" option. If that doesn't fail, then we should be all good. |
|
I have no idea what's causing the incorrect timestamps, but I'm guessing it's some bug introduced in the latest CalyxOS update. |
|
"OS Already Up to Date" Should be allrightey-o ? |
|
Yep! Should be completely fixed. I'll merge #54 and release version 4.1. |
|
a couple more questions if you don't mind :-)
I am most comfortable with it staying in user 0 and just there
I think it is worth reporting to the Calyx folks. While support of magisk modules is obviously outside of scope for them, the anomalous timestamp behavior might be involved in more than that and eventually come back to bite in some other, more "mainstream" parts of the system on unmodified installs (my gut feel) |
Yep, perfectly safe. It shouldn't really be installed in other profiles anyway, but I don't know how to prevent Android from doing that.
Custota's boot script grabs the timestamp by running the I think it may be most useful to them if you're able to grab a logcat as soon as possible after booting. Maybe the log file will catch the point during the boot process where the system time gets "fixed". If you're able to adb before unlocking the device, that'd probably work best. Otherwise, the logs from the earlier parts of the boot process might already be gone. |
|
Version 4.1 has been released with all the fixes! (Since the root cause appears to be fixed/worked around, I'm closing this issue. Feel free to keep discussing stuff--I get notifications for everything.) |
|
Hi! Thank you very much, great to see it resolved. Or maybe even for all the apps on the system (frankly don't quite grok what benefit package manager cache's existence brings)? I suspect my inability to make some other magisk things work (including lsposed) is due to this very same bug. I did look through the commit and... well... Would nuking everything in "/data/system/package_cache/*" to smithereens after installing a likely-affected magisk module (or other likely affected app) be unwise and unsafe? On a related note, do both the system apps (priv-app etc.) and the "normal person user apps" all have their package manager caches in that same "/data/system/package_cache" path ? EDIT yeah that was two questions sorry 😊 |
|
The only purpose of the cache is to save a couple seconds of boot time. It really isn't very useful. Some OS's, like GrapheneOS, disable the feature entirely. Deleting
find /data/system/package_cache -mindepth 1 -deleteThe script is guaranteed to run early enough during boot that nothing will be trying to read from the cache yet. EDIT: Adding |
|
Whoops, forgot to answer your second question.
Yes. The cache is for all apks across all users. The filename pattern is: In case you're curious, these are the components are:
|
So long story short I'm updating my CalyxOS on a redfin (Pixel 5)
I grab the OTA from OTA full section of here https://calyxos.org/get/ota/
Full link (stable)
https://release.calyxinstitute.org/redfin-ota_update-24505020.zip
(security-express produces same behavior)
when I try to run OTAtool (recent version) it displays following message
Enter passphrase for "/home/user/avbr/keyman-slave/ota.key":
Verifying OTA signature...
Device name: redfin
Fingerprint: google/redfin/redfin:14/UP1A.231105.001.B2/11260668:user/release-keys
Security patch: 2023-11-05
That's helluva old security patch!
And of course, needless to say, when I attempt to load it up in custota on the phone, it complains of security patch being downgraded and aborts.
What is going on here ?
The text was updated successfully, but these errors were encountered: