Skip to content
This repository has been archived by the owner on May 30, 2023. It is now read-only.

sys-auth/polkit: bump to 0.119 and apply duktape patchset #1263

Merged
merged 5 commits into from
Oct 19, 2021
Merged

Conversation

wrl
Copy link
Contributor

@wrl wrl commented Sep 7, 2021

This PR does two things simultaneously:

Duktape is substantially smaller than Spidermonkey, which both reduces package build time and allows us to ship a smaller image (though I have not determined how much smaller). More importantly, Spidermonkey and its transitive dependencies have been a substantial problem for building arm64 images, and using duktape sidesteps this issue nicely. As such, this PR also resolves the outstanding arm64 polkit issue (ref #863 flatcar/Flatcar#474 flatcar/Flatcar#156).

Of note is that this patchset merge request is marked as WIP and the polkit maintainer has issues with it that are still outstanding. I am going to attempt to resolve them and push this work further along. For the time being, if we ship this duktape'd polkit, we need to be clear publicly that it potentially introduces issues with custom rules.

An option would be to continue depending on Spidermonkey for amd64 and only use duktape for arm64, but a counterpoint is that shipping duktape for both architectures allows for more testing and feedback.

CI 🔵 : http://jenkins.infra.kinvolk.io:8080/job/os/job/manifest/3866/cldsv/

@wrl wrl mentioned this pull request Sep 7, 2021
@wrl wrl self-assigned this Sep 7, 2021
@wrl wrl requested a review from a team September 7, 2021 11:22
@pothos
Copy link
Contributor

pothos commented Sep 7, 2021

Can you follow #863 (review) to make sure we can identify the downstream changes and also know that previous downstream changes are preserved?
When that's done we can start a CI build.

@wrl
Copy link
Contributor Author

wrl commented Sep 7, 2021

Now find all downstream patches for the package by running git log CATEGORY/PACKAGE. If everybody followed the process of resetting before importing an upstream update, you only have to look for the commits after the last update and port them to the new version. Otherwise you have to compare the files manually to their upstream versions from older portage revisions.

@pothos What constitutes a "downstream patch", exactly? To be honest with you, I have no idea what makes 5c4d184 a commit that needs to be forward-ported and I would appreciate any insight you can provide on this.

@jepio
Copy link
Contributor

jepio commented Sep 7, 2021

"Downstream patches" are changes that were applied to the ebuild in this repository compared to the version in https://github.com/gentoo/gentoo (the upstream). Many of the "downstream" changes are things like in 5c4d184: installing Flatcar specific configuration files or files in Flatcar specific locations (keeping in mind that: /usr is read-only and gets auto-updated).

@pothos
Copy link
Contributor

pothos commented Sep 7, 2021

The commits in git log sys-auth/polkit/ which result in the difference between polkit-0.113-r5.ebuild and the last upstream ebuild file this was based on: https://github.com/gentoo/gentoo/blob/e9aa53351ccea3a7e9ee29f8597e9e51ec5449f4/sys-auth/polkit/polkit-0.113-r4.ebuild

The example commit I took is quite interesting because it relates to how Flatcar is updated:

 polkit: fix config install paths, use systemd-tmpfiles

All configs should be installed to /usr and tmpfiles should be used to
create and fix directory permissions instead of the ebuild's postinst.

What is auto-updated is the /usr partition, which means that /etc and /var is managed by the user. Everything that can be moved from /etc to /usr/share is moved by this commit, but some things have to be set up on /etc and /var. A way to set up the required files or folders on /etc and /var from the auto-updated /usr partition is by shipping systemd-tempfile directives (systemd_dotmpfilesd uses the downstream sys-auth/polkit/files/polkit.conf file for this purpose).

@dongsupark
Copy link
Contributor

Exciting to see the duktape-based polkit! 👍

Ideally, we should move the native Gentoo ebuilds, i.e. without Flatcar patches, to portage-stable.
acct-group/polkitd, acct-user/polkitd, dev-lang/duktape belong to there.
Then we can keep only sys-auth/polkit in coreos-overlay.

@wrl
Copy link
Contributor Author

wrl commented Sep 7, 2021

@jepio @pothos Okay, is there any way to, say, just get a list of outstanding downstream patches that I have to port forward, or does this just entail manually inspecting commit-by-commit?

@pothos
Copy link
Contributor

pothos commented Sep 7, 2021

Getting the list easily requires that people follow this best practice but it wasn't done in the case of this package here. But if you update by resetting the package to upstream state and then adding the downstream changes in a separate commit (one big or many small as you like), it will be easy for the next person to get the list of downstream patches.

@pothos
Copy link
Contributor

pothos commented Sep 7, 2021

Or actually this package here followed this practice somehow already and you can consider all commits from fb5590c onwards as downstream changes but the patches that are included in the new polkit release can be dropped.

@krnowak
Copy link
Contributor

krnowak commented Sep 9, 2021

With my python2-eradicator hat on, I'd love this to get this in, as spidermonkey is still using python2.

@wrl wrl changed the title sys-auth/polkit: bump to 0.119 and apply duktape patchset WIP: sys-auth/polkit: bump to 0.119 and apply duktape patchset Sep 21, 2021
@wrl wrl force-pushed the wrl/polkit-0.119 branch 2 times, most recently from c5e253a to 5c0b98b Compare September 28, 2021 15:13
@pothos
Copy link
Contributor

pothos commented Sep 28, 2021

The second commit should say re-apply downstream patches and also include the duktape patch as it is a downstream patch compared to Gentoo.

@tormath1
Copy link
Contributor

CI is OK with arm64 but not with amd64, polkit pulls python through an intermediate dependency. 🤔

@jepio
Copy link
Contributor

jepio commented Sep 29, 2021

Looks like we'll have to disable introspection useflag

@krnowak
Copy link
Contributor

krnowak commented Sep 29, 2021

Looks like we'll have to disable introspection useflag

Not sure if that's an option, grepping for sys-auth/polkit gives me among other lines:

sys-auth/realmd/realmd-0.17.0.ebuild:DEPEND="sys-auth/polkit[introspection]

I'll need to check if realmd really needs introspection stuff from polkit.

@krnowak
Copy link
Contributor

krnowak commented Sep 29, 2021

I had a look at realmd's and polkit build systemd and I think that realmd's dependency on polkit with introspection is bogus - it should be fine with being just DEPEND="sys-auth/polkit, without the [introspection] part. And there's nothing in portage-stable that depends on polkit, so that ought to work.

@krnowak
Copy link
Contributor

krnowak commented Sep 29, 2021

And dropping introspection from polkit would be nice - it was giving me a headache when I tried to update gobject-introspection, because polkit then was trying to put or get some introspection things from /build/amd64-usr/build/amd64-usr/blah/blah/blah and I couldn't figure out why.

@wrl
Copy link
Contributor Author

wrl commented Sep 29, 2021

@pothos Considering that the duktape patch is pulled from Gitlab and is therefore version-dependent, does it make sense to have it be part of the roll-up patch? It seems like that's just adding complexity next time we have to bump the version (i.e. "reformat these patches for the new version, but also re-fetch this other patch from Gitlab".)

@pothos
Copy link
Contributor

pothos commented Oct 4, 2021

Yes, a separate commit would be nice which documents how to create it.

@wrl
Copy link
Contributor Author

wrl commented Oct 4, 2021

Have updated this PR with separate duktape patchset commit.

Note: this will not build with only the first commit ("bump to 0.119") applied because it depends on a version of Spidermonkey which we do not have in portage-stable.

@wrl wrl changed the title WIP: sys-auth/polkit: bump to 0.119 and apply duktape patchset sys-auth/polkit: bump to 0.119 and apply duktape patchset Oct 4, 2021
Comment on lines 17 to 19
# flatcar changes: remove `[introspection]` from `sys-auth/polkit`
# it's the only component preventing us to build polkit without `introspection`
# support.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could drop this comment - this package is our own, not based on anything from gentoo.

Copy link
Contributor

@tormath1 tormath1 Oct 18, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh good to know. Any reason to not push this ebuild to ::gentoo then ?

EDIT: comment removed and commit reworded in the rebasing of the branch onto main.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh good to know. Any reason to not push this ebuild to ::gentoo then ?

Probably no reason other than "no one got around to do it yet". :) I never contributed anything to gentoo, so I'd need to invest some time to see how it's done.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we could first drop a comment into this issue: https://bugs.gentoo.org/486230 to see what's going on and mentioning that ::coreos-overlay holds an ebuild without so much Flatcar customization (except maybe for the tmpfiles).
Otherwise, I'm used to push ebuilds on ::guru1 when it's not really needed by ::gentoo and often ebuilds from ::guru finally land on ::gentoo.

Footnotes

  1. https://github.com/gentoo/guru

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, adding the comment and the ebuild (probably changing tmpfiles to keepdir or something) to guru sound good as first steps. I can add it to my TODO list, if you won't be able to do it. That would still mean that we would need to maintain our own fork of the ebuild, so it probably won't be a big win.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, and changing DBus policy path from /etc/dbus-1/system.d to /usr/share/dbus-1/system.d is also a flatcarism.

@tormath1
Copy link
Contributor

tormath1 commented Oct 18, 2021

build fails for both boards:

!!! All ebuilds that could satisfy ">=dev-lang/python-2.7.5-r2:2.7[xml]" for /mnt/host/source/src/build/images/amd64-usr/developer-2021.10.18+dev-flatcar-master-3867-a1/rootfs/ have been masked.
!!! One of the following masked packages is required to complete your request:
- dev-lang/python-2.7.15::portage-stable (masked by: package.mask)
/mnt/host/source/src/third_party/coreos-overlay/profiles/coreos/targets/generic/prod/package.mask:
# We don't want to support interpreted languages, changes/updates we make
# would have a high risk of breaking users.


(dependency required by "dev-libs/gobject-introspection-1.40.0-r1::portage-stable" [binary])
(dependency required by "sys-auth/polkit-0.119-r2::coreos" [binary])
(dependency required by "sys-auth/realmd-0.17.0::coreos" [binary])
(dependency required by "coreos-base/coreos-0.0.1-r301::coreos" [binary])
(dependency required by "coreos-base/coreos" [argument])
For more information, see the MASKED PACKAGES section in the emerge
man page or refer to the Gentoo Handbook.

but we can clearly see that sys-auth/polkit is built without introspection in the packages-matrix:

[ebuild  N     ] sys-auth/polkit-0.119-r2::coreos to /build/amd64-usr/ USE="pam systemd (-elogind) -examples -gtk -introspection -kde -nls -selinux -test" 1355 KiB

Now comes the fun part, let's compare binaries between this PR and regular binaries:

This PR, we can clearly see that there is no introspection in the USE flag and realmd is built with sys-auth/polkit but without the required [introspection] :

$ qxpak --extract --stdout developer_boards_amd64-usr_2021.10.18+dev-flatcar-master-3867_pkgs_sys-auth_polkit-0.119-r2.tar.xpak USE
abi_x86_64 amd64 elibc_glibc kernel_linux pam systemd userland_GNU
$ qxpak --extract --stdout developer_boards_amd64-usr_2021.10.18+dev-flatcar-master-3867_pkgs_sys-auth_realmd-0.17.0.tar.xpak DEPEND
sys-auth/polkit sys-devel/gettext dev-libs/glib:2 net-nds/openldap virtual/krb5 sys-apps/systemd

comparing with current output from a nightly:

$ qxpak --extract --stdout developer_boards_amd64-usr_2021.10.16+dev-main-nightly-3856_pkgs_sys-auth_polkit-0.113-r5.tar.xpak USE
abi_x86_64 amd64 elibc_glibc introspection kernel_linux pam systemd userland_GNU
$ qxpak --extract --stdout developer_boards_amd64-usr_2021.10.16+dev-main-nightly-3856_pkgs_sys-auth_realmd-0.17.0.tar.xpak DEPEND
sys-auth/polkit[introspection] sys-devel/gettext dev-libs/glib:2 net-nds/openldap virtual/krb5 sys-apps/systemd

so the binaries look good but we still rely on the dev-libs/gobject-introspection lib, not sure why but if we check required lib for both, we can see:

$ qxpak --extract --stdout developer_boards_amd64-usr_2021.10.16+dev-main-nightly-3856_pkgs_sys-auth_realmd-0.17.0.tar.xpak REQUIRES
x86_64: libc.so.6 libgio-2.0.so.0 libglib-2.0.so.0 libgobject-2.0.so.0 libkrb5.so.3 liblber-2.4.so.2 libldap-2.4.so.2 libpolkit-gobject-1.so.0 libpthread.so.0 libresolv.so.2 libsystemd.so.0
$ qxpak --extract --stdout developer_boards_amd64-usr_2021.10.18+dev-flatcar-master-3867_pkgs_sys-auth_realmd-0.17.0.tar.xpak REQUIRES
x86_64: libc.so.6 libgio-2.0.so.0 libglib-2.0.so.0 libgobject-2.0.so.0 libkrb5.so.3 liblber-2.4.so.2 libldap-2.4.so.2 libpolkit-gobject-1.so.0 libpthread.so.0 libresolv.so.2 libsystemd.so.0

there is libpolkit-gobject-1.so.0 which IMHO should not be there in the case we disabled introspection.

I'm tempted to remove realmd from amd64 too - but not sure about the impact.

@krnowak
Copy link
Contributor

krnowak commented Oct 18, 2021

@tormath1: libpolkit-gobject-1.so.0 is always built, regardless of the introspection. Introspection was only used for generating gir and typelib files.

@krnowak
Copy link
Contributor

krnowak commented Oct 18, 2021

@tormath1: I suppose that the problem is that dev-libs/gobject-introspection is pulled in through DEPEND, and RDEPEND is defined in terms of DEPEND. So I think DEPEND ought to have dev-libs/gobject-introspection hidden behind the introspection use flag, just like BDEPEND does.

@krnowak
Copy link
Contributor

krnowak commented Oct 18, 2021

@tormath1: Actually, gentoo upstream dropped dev-libs/gobject-introspection from DEPEND. Maybe some syncing with gentoo would fix it?

@tormath1
Copy link
Contributor

I suppose that the problem is that dev-libs/gobject-introspection is pulled in through DEPEND

I missed that 🤦 haha - I just don't understand from where it comes, it seems that dev-libs/gobject-introspection has never been into the DEPEND section on the upstream, at the least for the -0.119 version.

Copy link
Contributor

@krnowak krnowak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fixup commit should rather go to sys-auth/polkit 0.119: apply duktape patchset commit - it's the one that brought gobject-introspection to DEPEND variable, possibly by mistake.

sys-auth/polkit/polkit-0.119-r2.ebuild Outdated Show resolved Hide resolved
wrl and others added 4 commits October 19, 2021 12:46
https://gitlab.freedesktop.org/polkit/polkit/-/merge_requests/35

this should be re-fetched from the above MR when forward-porting to
updated polkit versions.
commit 5c4d184
Author: Michael Marineau <michael.marineau@coreos.com>
Date:   Fri Aug 1 14:48:59 2014 -0700

    polkit: fix config install paths, use systemd-tmpfiles

    All configs should be installed to /usr and tmpfiles should be used to
    create and fix directory permissions instead of the ebuild's postinst.
we don't need to build realmd with introspection support for polkit.

Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
it was already the case for ARM64, we just extend it to AMD64.

Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
@tormath1
Copy link
Contributor

@krnowak thanks for reviewing - commits have been rebased / amended / pushed following your comments.
@wrl any comments regarding this: #1263 (comment) ?

Copy link
Contributor

@krnowak krnowak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Let's get this finally in. :)

@wrl wrl merged commit ef38fa6 into main Oct 19, 2021
dongsupark added a commit to dongsupark/portage-stable that referenced this pull request Nov 1, 2021
flatcar-archive/coreos-overlay#1263 made polkit
depend on duktape instead of spidermonkey.
So we can delete spidermonkey completely.
dongsupark added a commit to dongsupark/portage-stable that referenced this pull request Nov 1, 2021
flatcar-archive/coreos-overlay#1263 made polkit
depend on duktape instead of spidermonkey.
So we can delete spidermonkey completely.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants