Mender rollback issue

[dat-tran-matternet]

I posted this question in meta-tegra project but it makes more sense to be here:

I noticed using the tegra-demo-distro (dunfell branch) the following issues with mender:

After a "mender install" and reboot, then a reboot WITHOUT "mender commit":

• Expected behavior: Mender to automatically roll back to the original rootfs because I didn't run "mender commit"
• Actual behavior: New partition is still active no matter how many times I reboot

After a "mender install" and reboot, then "mender rollback":

• Expected behavior: Mender to rollback to original rootfs after a reboot
• Actual behaviro: Mender states that it will roll back to original rootfs but new partition is still active after a reboot

Setup:

Project: tegra-demo-distro
Branch: dunfell (dc1dd88)
Machine: jetson-xavier-nx-devkit-tx2-nx
Distro: tegrademo-mender
Image: demo-image-base
Cboot only (no UBoot)
Mender-client: v2.6.1 and v3.2.1

Thanks

[dwalkes]

I think this is a side effect of the mender cboot integration and related to the discussion at OE4T/meta-mender-community#8 (comment) - which describes a deviation from the way this works with mender on other platforms and relates to the additional dependencies between boot slots and cboot managment we need to do on this platform. It's been a while so if any of that is unclear LMK and I will try to refresh my memory.

[dat-tran-matternet]

Thanks for your reply. Does this mean this issue is not resolved yet? I'm using the latest dunfell branch for everything and only using cboot.

[dwalkes]

I think the current implementation does not match the expected behavior you list (and Mender expects) for rollback, and I don't think there are any plans by anyone to change at this time. See the discussion at OE4T/meta-mender-community#8 (comment). I think the benefit would be to solve cases where boot is successful but the device can't reach the mender server, assuming the code at OE4T/meta-mender-community#8 (comment) exists somewhere I couldn't find definitively. We'd need to make sure to always keep the bootloader and rootfs slots in sync when handling the rollback scenario.

[madisongh]

We'd need to make sure to always keep the bootloader and rootfs slots in sync when handling the rollback scenario.

We might be able to emulate the U-Boot+Mender behavior now that we've switched from NVIDIA's bootloader tools to tegra-boot-tools. Part of the problem, originally, was that we had no equivalent of U-Boot's environment variables that we could use to communicate state between the bootloader and the Mender client. With tegra-boot-tools, we do have variables we can use for this, although the logic in Mender's u-boot script would either have to be hard-coded into cboot via a patch or possibly implemented in the script we run in the initramfs (so there would still be a bit of a difference).

That said, the current behavior does work adequately for the normal production use case with the Mender client running as a daemon. It's only using the Mender "standalone" mode (i.e., manually running mender install, mender rollback, mender commit) that you would be seeing this discrepancy. As Dan pointed out, if the system fails to boot (7 times during bootloader stages, or 3 times once the kernel is running), the system will roll back automatically. The Mender client should detect this, update its internal database and report the failure to the server. The A/B redundancy/failover support is built into the system already, and we were just trying to make the Mender client compatible with it - unlike other systems where the the Mender client is (with the help of its bootloader integration) implementing the feature for you.