build2: use lld from llvm 16 to link on darwin #243367

tjni · 2023-07-14T00:58:06Z

Description of changes

This seems to fix the build of build2 after the darwin stdenv rework in #240433. For some reason, the ar from cctools-llvm produces an artifact that breaks linking in this package with the following error:

ld: warning: ignoring file libbuild2/libbuild2.dylib.u.a, building for macOS-arm64 but attempting to link with file built for unknown-unsupported file format ( 0x21 0x3C 0x74 0x68 0x69 0x6E 0x3E 0x0A 0x2F 0x20 0x20 0x20 0x20 0x20 0x20 0x20 )

Also wouldn't mind a better solution than this.

Broken right now in staging-next so targeting it: #241951

/cc @reckenrode

Things done

reckenrode · 2023-07-14T02:12:25Z

Thanks for the ping. It’s failing due to two things:

build2 creates thin archives if it detects you’re using LLVM ar (or GNU ar). Its assume is your linker comes from the same source as ar and supports thin archives.
ld64 does not support thin archives.

I’m digging into this to see if there is a different fix.

pkgs/development/tools/build-managers/build2/default.nix

reckenrode · 2023-07-14T04:39:52Z

The following patch uses lld to link instead of ld64. I was able to build build2, bdep, bpkg, and libodb (albeit with clang 16 instead of clang 11). It’s not exactly pretty, but it appears things don’t necessarily use config.bin.ld, so I had to add it as a propagated build input. The setup hook hackery is probably required regardless of solution since the setting needs to be picked up by packages that use build2 to build.

diff --git a/pkgs/development/tools/build-managers/build2/bootstrap.nix b/pkgs/development/tools/build-managers/build2/bootstrap.nix
index 4f9879840b2..0fadb42c8a4 100644
--- a/pkgs/development/tools/build-managers/build2/bootstrap.nix
+++ b/pkgs/development/tools/build-managers/build2/bootstrap.nix
@@ -1,6 +1,7 @@
 { lib, stdenv
 , fetchurl
 , pkgs
+, buildPackages
 , fixDarwinDylibNames
 }:
 stdenv.mkDerivation rec {
@@ -25,6 +26,7 @@ stdenv.mkDerivation rec {
 
   propagatedBuildInputs = lib.optionals stdenv.targetPlatform.isDarwin [
     fixDarwinDylibNames
+    (lib.getBin buildPackages.llvmPackages.lld)
   ];
 
   doCheck = true;
@@ -40,5 +42,10 @@ stdenv.mkDerivation rec {
     runHook postInstall
   '';
 
+  postFixup = ''
+    substituteInPlace $out/nix-support/setup-hook \
+      --subst-var-by isTargetDarwin '${toString stdenv.targetPlatform.isDarwin}'
+  '';
+
   inherit (pkgs.build2) passthru;
 }
diff --git a/pkgs/development/tools/build-managers/build2/default.nix b/pkgs/development/tools/build-managers/build2/default.nix
index 562d3aa0c47..6f636150ba3 100644
--- a/pkgs/development/tools/build-managers/build2/default.nix
+++ b/pkgs/development/tools/build-managers/build2/default.nix
@@ -4,6 +4,7 @@
 , fixDarwinDylibNames
 , libbutl
 , libpkgconf
+, buildPackages
 , enableShared ? !stdenv.hostPlatform.isStatic
 , enableStatic ? !enableShared
 }:
@@ -57,6 +58,9 @@ stdenv.mkDerivation rec {
   # LC_LOAD_DYLIB entries containing @rpath, requiring manual fixup
   propagatedBuildInputs = lib.optionals stdenv.targetPlatform.isDarwin [
     fixDarwinDylibNames
+    # Build2 needs to use lld on Darwin because it creates thin archives when it detects `llvm-ar`,
+    # which ld64 does not support.
+    (lib.getBin buildPackages.llvmPackages.lld)
   ];
 
   postPatch = ''
@@ -73,6 +77,11 @@ stdenv.mkDerivation rec {
     install_name_tool -add_rpath "''${!outputLib}/lib" "''${!outputBin}/bin/b"
   '';
 
+  postFixup = ''
+    substituteInPlace $dev/nix-support/setup-hook \
+      --subst-var-by isTargetDarwin '${toString stdenv.targetPlatform.isDarwin}'
+  '';
+
   passthru = {
     bootstrap = build2;
     inherit configSharedStatic;
diff --git a/pkgs/development/tools/build-managers/build2/setup-hook.sh b/pkgs/development/tools/build-managers/build2/setup-hook.sh
index 16b592d3c10..1d19091ac98 100644
--- a/pkgs/development/tools/build-managers/build2/setup-hook.sh
+++ b/pkgs/development/tools/build-managers/build2/setup-hook.sh
@@ -19,6 +19,12 @@ build2ConfigurePhase() {
         $build2ConfigureFlags "${build2ConfigureFlagsArray[@]}"
     )
 
+    if [ -n "@isTargetDarwin@" ]; then
+        flagsArray+=("config.bin.ld=ld64-lld")
+        flagsArray+=("config.cc.coptions+=-fuse-ld=lld")
+        flagsArray+=("config.cc.loptions+=-headerpad_max_install_names")
+    fi
+
     echo 'configure flags' "${flagsArray[@]}"
 
     b configure "${flagsArray[@]}"

tjni · 2023-07-14T05:08:24Z

Aha, I see. Where does isTargetDarwin come from?

Please let me know if you'd like me to apply these changes, or if you'd like to create a separate PR.

reckenrode · 2023-07-14T05:13:24Z

Aha, I see. Where does isTargetDarwin come from?

It’s substituted in the fixupPhase based on the value of stdenv.targetPlatform.isDarwin.

Please let me know if you'd like me to apply these changes, or if you'd like to create a separate PR.

It’s up to you. This PR already has the context of the issue, so that might be easier, but I can also open a separate one.

tjni · 2023-07-14T05:19:21Z

Give me a few minutes to try to do this and test it.

tjni · 2023-07-14T05:42:29Z

Builds build2 and bdep gets past its original error. I had to make two changes that could use a second pair of eyes:

In bootstrap.nix, the setup hook is at $out/nix-support/setup-hook while it's at $dev/nix-support/setup-hook in the main derivation.
I added -fuse-ld=lld to config.cc.loptions instead of config.cc.coptions since I got a lot of warnings otherwise.

I currently do see these warnings when building bdep, not sure if they are relevant:

ld64.lld: warning: ignoring unknown argument: -headerpad_max_install_names
ld64.lld: warning: ignoring unknown argument: -platform_version
ld64.lld: warning: ignoring unknown argument: -no_uuid
ld64.lld: warning: -sdk_version is required when emitting min version load command.  Setting sdk version to match provided min version

reckenrode · 2023-07-14T06:21:29Z

In bootstrap.nix, the setup hook is at $out/nix-support/setup-hook while it's at $dev/nix-support/setup-hook in the main derivation.

The bootstrap derivation only has one output while the actual build also has a dev output. The dev output is where the setup hook will be put, so it has to be modified there.

I added -fuse-ld=lld to config.cc.loptions instead of config.cc.coptions since I got a lot of warnings otherwise

I thought I tried that, but I might have done something wrong. I agree with this change.

I currently do see these warnings when building bdep, not sure if they are relevant:

ld64.lld: warning: ignoring unknown argument: -headerpad_max_install_names
ld64.lld: warning: ignoring unknown argument: -platform_version
ld64.lld: warning: ignoring unknown argument: -no_uuid
ld64.lld: warning: -sdk_version is required when emitting min version load command.  Setting sdk version to match provided min version

-headerpad_max_install_names was added in LLVM 12. It adds extra padding to make sure install_name_tool will work on the resulting binary. lld has different behavior than ld64 (see llvm/llvm-project#53550).

-no_uuid is unimplemented in lld. Newer show a message saying as much. The UUID is supposed to be deterministic, so the lack of support should be harmless. It arguably should be included even when linking with ld64 (see #178366).

-platform_version was implemented in LLVM 12. It sets the deployment target.

-sdk_version does something similar to -platform_version but doesn’t let you specify the platform (e.g., macos, ios, etc).

tjni · 2023-07-14T07:09:40Z

Something is going wrong when building libbutl (in between build2's bootstrap and build2 proper):

ld64.lld: error: Unable to find framework for -framework CoreFoundation
clang-12: error: linker command failed with exit code 1 (use -v to see invocation)

regardless of which clang version I use, even if I add CoreFoundation to buildInputs. Late so will mark this as a draft and poke around tomorrow.

reckenrode · 2023-07-14T07:55:12Z

It appears lld 11 doesn’t have very good Mach-O support. It didn’t even get arm64 support until lld 13. Instead of using llvmPackages.lld, try using llvmPackages_16.lld.

tjni · 2023-07-14T08:36:04Z

Result of nixpkgs-review pr 243367 run on aarch64-darwin 1

22 packages built:

bdep
bdep.doc
bdep.man
bpkg
bpkg.doc
bpkg.man
build2
build2.dev
build2.doc
build2.man
libbpkg
libbpkg.dev
libbpkg.doc
libbutl
libbutl.dev
libbutl.doc
libodb
libodb-sqlite
libodb-sqlite.dev
libodb-sqlite.doc
libodb.dev
libodb.doc

tjni · 2023-07-14T08:41:21Z

Using lld from llvmPackages_16 does the trick! Thank you for all the help.

r-burns

Tested on x86_64-darwin. Nice work, thanks for the fix!

ofborg bot requested a review from r-burns July 14, 2023 01:18

ofborg bot added 10.rebuild-darwin: 1-10 10.rebuild-linux: 0 labels Jul 14, 2023

reckenrode reviewed Jul 14, 2023

View reviewed changes

pkgs/development/tools/build-managers/build2/default.nix Outdated Show resolved Hide resolved

tjni force-pushed the build2 branch 2 times, most recently from 18c43c2 to bd55ab9 Compare July 14, 2023 05:38

tjni changed the title ~~build2: use ${cctools-port}/bin/ar in build~~ build2: use lld to link on darwin Jul 14, 2023

ofborg bot added 10.rebuild-linux: 1-10 and removed 10.rebuild-linux: 0 labels Jul 14, 2023

vcunat added the 6.topic: darwin Running or building packages on Darwin label Jul 14, 2023

tjni marked this pull request as draft July 14, 2023 07:09

tjni force-pushed the build2 branch 2 times, most recently from 1a22934 to ccc403c Compare July 14, 2023 08:29

tjni changed the title ~~build2: use lld to link on darwin~~ build2: use lld from llvm 16 to link on darwin Jul 14, 2023

build2: use lld from llvm 16 to link on darwin

ee4f1fe

tjni force-pushed the build2 branch from ccc403c to ee4f1fe Compare July 14, 2023 08:40

tjni marked this pull request as ready for review July 14, 2023 08:41

r-burns approved these changes Jul 14, 2023

View reviewed changes

r-burns merged commit 2203291 into NixOS:staging-next Jul 14, 2023
8 checks passed

tjni deleted the build2 branch July 14, 2023 23:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build2: use lld from llvm 16 to link on darwin #243367

build2: use lld from llvm 16 to link on darwin #243367

tjni commented Jul 14, 2023 •

edited

Loading

reckenrode commented Jul 14, 2023

reckenrode commented Jul 14, 2023 •

edited

Loading

tjni commented Jul 14, 2023

reckenrode commented Jul 14, 2023

tjni commented Jul 14, 2023

tjni commented Jul 14, 2023

reckenrode commented Jul 14, 2023

tjni commented Jul 14, 2023

reckenrode commented Jul 14, 2023 •

edited

Loading

tjni commented Jul 14, 2023

tjni commented Jul 14, 2023

r-burns left a comment

build2: use lld from llvm 16 to link on darwin #243367

build2: use lld from llvm 16 to link on darwin #243367

Conversation

tjni commented Jul 14, 2023 • edited Loading

Description of changes

Things done

reckenrode commented Jul 14, 2023

reckenrode commented Jul 14, 2023 • edited Loading

tjni commented Jul 14, 2023

reckenrode commented Jul 14, 2023

tjni commented Jul 14, 2023

tjni commented Jul 14, 2023

reckenrode commented Jul 14, 2023

tjni commented Jul 14, 2023

reckenrode commented Jul 14, 2023 • edited Loading

tjni commented Jul 14, 2023

tjni commented Jul 14, 2023

r-burns left a comment

Choose a reason for hiding this comment

tjni commented Jul 14, 2023 •

edited

Loading

reckenrode commented Jul 14, 2023 •

edited

Loading

reckenrode commented Jul 14, 2023 •

edited

Loading