libglib: split libraries from glib #103366

XuehaiPan · 2022-06-10T11:33:01Z

Have you followed the guidelines for contributing?
Have you ensured that your commits follow the commit style guide?
Have you checked that there aren't other open pull requests for the same formula update/change?
Have you built your formula locally with brew install --build-from-source <formula>, where <formula> is the name of the formula you're submitting?
Is your test running fine brew test <formula>, where <formula> is the name of the formula you're submitting?
Does your build pass brew audit --strict <formula> (after doing brew install --build-from-source <formula>)? If this is a new formula, does it pass brew audit --new <formula>?

Include glib and libglib formulae only in

libglib: split formula glib #102045

The remaining formulae that depend on glib will be migrated in separate PRs.

XuehaiPan · 2022-06-10T11:35:30Z

Formula/glib.rb

  def post_install
-    (HOMEBREW_PREFIX/"lib/gio/modules").mkpath
+    libglib = Formula["libglib"]
+    include.make_relative_symlink(libglib.opt_include)
+    lib.make_relative_symlink(libglib.opt_lib)
  end


Make symlinks to let the downstream formula access Formula["glib"].opt_include and Formula["glib"].opt_lib. These symlinks will be removed when the migration is done.

carlocab · 2022-06-10T13:50:32Z

This approach doesn't work, because it requires writing to paths outside glib's keg, which is disallowed by the sandbox. That's why the build fails on macOS.

XuehaiPan · 2022-06-12T17:29:39Z

Formula/glib.rb

+  def script
+    <<~EOS
+      import pickle as pkl
+      from mesonbuild.minstall import load_install_data
+      installdata = load_install_data('meson-private/install.dat')
+      for attrname in ('data', 'emptydir', 'headers', 'install_scripts', 'install_subdirs', 'man', 'symlinks', 'targets'):
+          attr = getattr(installdata, attrname)
+          attr = list(filter(lambda data: all('opt/libglib' not in dataattr
+                                              for _, dataattr in vars(data)
+                                              if isinstance(dataattr, str)),
+                            attr))
+          setattr(installdata, attrname, attr)
+      with open('meson-private/install.dat', mode='wb') as file:
+          pkl.dump(file, installdata)
+    EOS
+  end


This approach would be really hacky that modifies a binary file created by meson (created by pickle.dump from Python). I have read the source code of meson, I don't find any other way to do this.

The file targets can be ignored by changing the mtime of files (make them older than those files in libglib). But there are also some symlinks in which we cannot do the same things. Then it will raise PermissionError due to sandboxing.

The only way to not write things to libglib is overwriting the install data of meson.

XuehaiPan · 2022-06-13T12:48:08Z

Require CI-long-timeout.

Formula/libglib.rb

SMillerDev · 2022-06-14T06:51:14Z

Formula/libglib.rb

+    # TODO: This is a workaround for `brew audit --new-formula`.
+    #       Use `patch` rather than `inreplace` (see also `glib`).
+    # replace several hardcoded paths with homebrew counterparts


Please don't work around the audits. They are there for a reason.

Formula/libglib.rb

chenrui333 · 2022-06-15T16:46:34Z

some test failures

brew test --retry --verbose glib-networking
brew test --retry --verbose glib-openssl
brew test --retry --verbose libsoup
brew test --retry --verbose libsoup@2

iMichka · 2022-06-15T20:17:19Z

I'm in favour of the split. I wanted to work on something similar, inspired by https://uwekorn.com/2020/10/19/r-without-python.html.

I am wondering if we could reduce the scope of the split. What is our goal here? Reduce the complexity of the Python dependency tree? In this case, I think there are only 2-3 python scripts in bin that need to be extracted to a separate formula (glib and glib-utils like gentoo does, for example).

XuehaiPan · 2022-06-16T06:39:26Z

I am wondering if we could reduce the scope of the split. What is our goal here? Reduce the complexity of the Python dependency tree? In this case, I think there are only 2-3 python scripts in bin that need to be extracted to a separate formula (glib and glib-utils like gentoo does, for example).

@iMichka, do you mean ship binary executables in libglib and only keep the Python scripts in glib?

The current implementation of the split in this PR:

libglib: only ships headers and shared libraries (include and lib).
glib: ships all executables (bin), completions (etc/bash_completion.d), locale settings (share/locale), and Python scripts (share/*).

iMichka · 2022-06-16T06:50:10Z

@iMichka, do you mean ship binary executables in libglib and only keep the Python scripts in glib?

Yes.

Just to avoid any confusion, here is your proposal:
glib -> libglib (headers and libraries)
glib -> glib (all binaries, share, ...)

Here is my proposal:
glib -> glib-utils (Python binaries only)
glib -> glib (everything else)

SMillerDev · 2022-06-16T07:00:07Z

I think the libglib/glib split is easier and cleaner to maintain.

It might be better to add a libglib formula first, migrate and after link glib to it though.

XuehaiPan · 2022-06-16T07:05:53Z

Here is my proposal:
glib -> glib-utils (Python binaries only)
glib -> glib (everything else)

This seems easier to implement and requires less migration effort. There are only three places contain Python scripts (bin / share/glib-2.0/codegen / share/gdb). However, in the future release, the upstream may add more Python code. We should recursively glob the directory to find Python scripts rather than use arbitrary paths.

danielnachun · 2022-06-16T21:45:00Z

We've also been discussing this on Slack and I think I like what @iMichka proposed for glib/glib-utils. While we will have to pay attention if upstream does add Python code elsewhere, we can just use globbing as @XuehaiPan suggested to find and remove Python scripts in glib, and then make sure we install those in glib-utils.

If you look at the Gentoo ebuild for glib-utils it's very simple to reimplement as a formula in Ruby: https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-util/glib-utils/glib-utils-2.72.2.ebuild. Basically just a few inreplace statements instead of sed and some calls to xsltproc. Given how much of a headache glib is currently causing for Python migrations, I think this would be the simplest route to dealing with this.

XuehaiPan · 2022-06-17T14:32:31Z

Closed in favor of PR #103916.

BrewTestBot added the automerge-skip `brew pr-automerge` will skip this pull request label Jun 10, 2022

XuehaiPan commented Jun 10, 2022

View reviewed changes

XuehaiPan force-pushed the split-glib branch from 0bb3ab2 to ea6ad8f Compare June 10, 2022 11:43

XuehaiPan force-pushed the split-glib branch 3 times, most recently from 6a01942 to 8f92ad3 Compare June 12, 2022 17:26

XuehaiPan commented Jun 12, 2022

View reviewed changes

XuehaiPan force-pushed the split-glib branch 4 times, most recently from 6eae317 to 0b4384f Compare June 13, 2022 12:25

SMillerDev reviewed Jun 13, 2022

View reviewed changes

Formula/libglib.rb Outdated Show resolved Hide resolved

Formula/libglib.rb Outdated Show resolved Hide resolved

XuehaiPan force-pushed the split-glib branch from 0b4384f to b39a9db Compare June 14, 2022 06:23

SMillerDev reviewed Jun 14, 2022

View reviewed changes

XuehaiPan force-pushed the split-glib branch 2 times, most recently from 66b346a to 899d783 Compare June 14, 2022 09:52

alebcay added long build Set a long timeout for formula testing CI-long-timeout [DEPRECATED] Use longer GitHub Actions CI timeout. labels Jun 14, 2022

libglib: split libraries from glib

9b6d549

XuehaiPan force-pushed the split-glib branch from 899d783 to 85bee34 Compare June 15, 2022 13:57

XuehaiPan added 2 commits June 15, 2022 22:05

glib: depends on libglib

7cbedae

glib: sync version with libglib

3ab8cdc

XuehaiPan force-pushed the split-glib branch from 85bee34 to 3ab8cdc Compare June 15, 2022 14:05

chenrui333 added CI-no-fail-fast Continue CI tests despite failing GitHub Actions matrix builds. and removed CI-long-timeout [DEPRECATED] Use longer GitHub Actions CI timeout. labels Jun 15, 2022

danielnachun mentioned this pull request Jun 15, 2022

qt@5 5.15.5 #103229

Closed

6 tasks

XuehaiPan mentioned this pull request Jun 17, 2022

glib-utils: split Python code from glib #103916

Closed

6 tasks

XuehaiPan closed this Jun 17, 2022

XuehaiPan deleted the split-glib branch June 23, 2022 09:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

libglib: split libraries from glib #103366

libglib: split libraries from glib #103366

XuehaiPan commented Jun 10, 2022

XuehaiPan Jun 10, 2022 •

edited

Loading

carlocab commented Jun 10, 2022

XuehaiPan Jun 12, 2022 •

edited

Loading

XuehaiPan commented Jun 13, 2022

SMillerDev Jun 14, 2022

chenrui333 commented Jun 15, 2022

iMichka commented Jun 15, 2022 •

edited

Loading

XuehaiPan commented Jun 16, 2022

iMichka commented Jun 16, 2022 •

edited

Loading

SMillerDev commented Jun 16, 2022

XuehaiPan commented Jun 16, 2022 •

edited

Loading

danielnachun commented Jun 16, 2022 •

edited

Loading

XuehaiPan commented Jun 17, 2022

libglib: split libraries from glib #103366

libglib: split libraries from glib #103366

Conversation

XuehaiPan commented Jun 10, 2022

XuehaiPan Jun 10, 2022 • edited Loading

Choose a reason for hiding this comment

carlocab commented Jun 10, 2022

XuehaiPan Jun 12, 2022 • edited Loading

Choose a reason for hiding this comment

XuehaiPan commented Jun 13, 2022

SMillerDev Jun 14, 2022

Choose a reason for hiding this comment

chenrui333 commented Jun 15, 2022

iMichka commented Jun 15, 2022 • edited Loading

XuehaiPan commented Jun 16, 2022

iMichka commented Jun 16, 2022 • edited Loading

SMillerDev commented Jun 16, 2022

XuehaiPan commented Jun 16, 2022 • edited Loading

danielnachun commented Jun 16, 2022 • edited Loading

XuehaiPan commented Jun 17, 2022

XuehaiPan Jun 10, 2022 •

edited

Loading

XuehaiPan Jun 12, 2022 •

edited

Loading

iMichka commented Jun 15, 2022 •

edited

Loading

iMichka commented Jun 16, 2022 •

edited

Loading

XuehaiPan commented Jun 16, 2022 •

edited

Loading

danielnachun commented Jun 16, 2022 •

edited

Loading