Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ELF binary built linked against fake loader #102993

Closed
slondr opened this issue Oct 13, 2022 · 24 comments · Fixed by #111698
Closed

ELF binary built linked against fake loader #102993

slondr opened this issue Oct 13, 2022 · 24 comments · Fixed by #111698
Labels
A-linkage Area: linking into static, shared libraries and binaries C-bug Category: This is a bug. O-musl Target: The musl libc T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.

Comments

@slondr
Copy link

slondr commented Oct 13, 2022

hi, I'm compiling a Rust program on my ARM AArch64 machine which is running linux with a musl libc. Rust wants to compile the binary with the interpreter set as /lib/ld-linux-aarch64.so.1 which does not exist on my system. I do have /lib/ld-musl-aarch64.so.1 however.

As a result when I try to execute the binary built by rust in this case I just get ENOENT. In fact when I strace ./binary there is only a single syscall printed, the execve on the binary itself, which returns ENOENT. This makes me sad...

Manually loading the binary (ie running exec /lib/ld-musl-aarch64.so.1 $PWD/binary) gets me a lot farther in the execution of my program which then crashes for other reasons, presumably my fault :) But that's out of scope for this issue.

If it helps I installed Rust via rustup.

@compiler-errors
Copy link
Member

See #82519 I think.

@bjorn3
Copy link
Member

bjorn3 commented Oct 14, 2022

What linker did you tell rustc to use for AArch64? You need to use a gcc or clang for cross-compiling to aarch64+musl, not one for aarch64+glibc.

@slondr
Copy link
Author

slondr commented Oct 14, 2022

@bjorn3 I am not cross-compiling, I'm just running cargo build

@bjorn3
Copy link
Member

bjorn3 commented Oct 14, 2022

My bad. What does rustc -vV show?

@slondr
Copy link
Author

slondr commented Oct 14, 2022

@bjorn3 Here you go:

rustc 1.64.0 (a55dd71d5 2022-09-19)
binary: rustc
commit-hash: a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52
commit-date: 2022-09-19
host: aarch64-unknown-linux-musl
release: 1.64.0
LLVM version: 14.0.6

@jyn514
Copy link
Member

jyn514 commented Oct 27, 2022

The issue tracker is not a support forum. For future questions please ask in one of these places:

https://discord.gg/rust-lang/
https://discord.gg/rust-lang-community/
https://users.rust-lang.org/
https://rust-lang.zulipchat.com/

I suspect your current issue is a problem with your default linker, not the compiler itself.

@jyn514 jyn514 closed this as not planned Won't fix, can't repro, duplicate, stale Oct 27, 2022
@slondr
Copy link
Author

slondr commented Oct 27, 2022

For the record, I only posted this issue here after discussing it in the rust matrix chat, where I was told (after some group investigation) that it looked like a bug in cargo or rustc.

@jyn514 jyn514 reopened this Oct 27, 2022
@jyn514
Copy link
Member

jyn514 commented Oct 27, 2022

Ok. You can use cargo build -v to show the invocations, and RUSTFLAGS=--print=link-args to see what's being passed to the linker. I'd still suggest trying to build a normal C program and make sure that works ok before building a rust program.

@jbg
Copy link

jbg commented Jan 23, 2023

I just hit this as well. To be clear, no cross-compiling is involved. Both host and target are aarch64-unknown-linux-musl.

It seems to be triggered by external C library deps that cause the binary to be dynamically linked (rather than static as would be the default for -musl targets). The resulting binary always has the wrong loader, and only works if the libc6-compat package is installed.

Here's an example of reproducing it by linking curl. In a clean alpine:3.17 container, reproduce as follows:

apk add build-base curl curl-dev pkgconf
curl https://sh.rustup.rs -sSf | sh -s -- -y
. ~/.cargo/env
cargo init --bin tester
cd tester
cargo add curl
echo 'use curl::easy::Easy; fn main() { Easy::new(); }' > src/main.rs
cargo build

Running the resulting binary fails:

# target/debug/tester
/bin/sh: target/debug/tester: not found

The loader is wrong:

# ldd target/debug/tester
	/lib/ld-linux-aarch64.so.1 (0xffffa0b61000)
	libcurl.so.4 => /usr/lib/libcurl.so.4 (0xffffa0ace000)
	libnghttp2.so.14 => /usr/lib/libnghttp2.so.14 (0xffffa0a8d000)
	libssl.so.3 => /lib/libssl.so.3 (0xffffa09e9000)
	libcrypto.so.3 => /lib/libcrypto.so.3 (0xffffa0663000)
	libbrotlidec.so.1 => /usr/lib/libbrotlidec.so.1 (0xffffa0642000)
	libz.so.1 => /lib/libz.so.1 (0xffffa0611000)
	libc.musl-aarch64.so.1 => /lib/ld-linux-aarch64.so.1 (0xffffa0b61000)
	libbrotlicommon.so.1 => /usr/lib/libbrotlicommon.so.1 (0xffffa05e0000)

# ls /lib/ld-linux-aarch64.so.1
ls: /lib/ld-linux-aarch64.so.1: No such file or directory

@jyn514
Copy link
Member

jyn514 commented Jan 23, 2023

@jbg do you have the same issue in a standalone C problem that links against curl? If so, this isn't a bug in the compiler itself. If not, you may want to talk to the owners of the curl crate on crates.io, there are various flags build scripts can pass that affect how the final program is linked. You can use cargo build -vvv curl to see build script output.

@bjorn3
Copy link
Member

bjorn3 commented Jan 23, 2023

It seems to be triggered by external C library deps that cause the binary to be dynamically linked (rather than static as would be the default for -musl targets). The resulting binary always has the wrong loader, and only works if the libc6-compat package is installed.

Does setting RUSTFLAGS="-Ctarget-feature=-crt-static" fix this issue?

@jbg
Copy link

jbg commented Jan 23, 2023

No, linking isn't broken in Alpine in general, it's only Rust that has this issue.

rust-builder-test:~$ cat > curl.c
#include <curl/curl.h>

int main(void) {
  CURL *curl = curl_easy_init();
  return 0;
}
rust-builder-test:~$ gcc curl.c -lcurl -o curl
rust-builder-test:~$ ldd curl
	/lib/ld-musl-aarch64.so.1 (0xffffbef04000)
	libcurl.so.4 => /usr/lib/libcurl.so.4 (0xffffbee50000)
	libc.musl-aarch64.so.1 => /lib/ld-musl-aarch64.so.1 (0xffffbef04000)
	libnghttp2.so.14 => /usr/lib/libnghttp2.so.14 (0xffffbee0f000)
	libssl.so.3 => /lib/libssl.so.3 (0xffffbed6b000)
	libcrypto.so.3 => /lib/libcrypto.so.3 (0xffffbe9e5000)
	libbrotlidec.so.1 => /usr/lib/libbrotlidec.so.1 (0xffffbe9c4000)
	libz.so.1 => /lib/libz.so.1 (0xffffbe993000)
	libbrotlicommon.so.1 => /usr/lib/libbrotlicommon.so.1 (0xffffbe962000)

The curl crate was just a convenient example. All Rust crates that dynamically link a C library seem to be affected. curl-sys seems to just use the pkg-config crate in its build.rs which outputs the usual rustc-link-lib lines.

@jbg
Copy link

jbg commented Jan 23, 2023

@bjorn3 yes, setting RUSTFLAGS="-Ctarget-feature=-crt-static" resolves it.

$ RUSTFLAGS="-Ctarget-feature=-crt-static" cargo build
[...]
    Finished dev [unoptimized + debuginfo] target(s) in 10.89s
$ ldd target/debug/tester
	/lib/ld-musl-aarch64.so.1 (0xffff928d0000)
	libcurl.so.4 => /usr/lib/libcurl.so.4 (0xffff927cc000)
	libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0xffff9279b000)
	libc.musl-aarch64.so.1 => /lib/ld-musl-aarch64.so.1 (0xffff928d0000)
	libnghttp2.so.14 => /usr/lib/libnghttp2.so.14 (0xffff9275a000)
	libssl.so.3 => /lib/libssl.so.3 (0xffff926b6000)
	libcrypto.so.3 => /lib/libcrypto.so.3 (0xffff92330000)
	libbrotlidec.so.1 => /usr/lib/libbrotlidec.so.1 (0xffff9230f000)
	libz.so.1 => /lib/libz.so.1 (0xffff922de000)
	libbrotlicommon.so.1 => /usr/lib/libbrotlicommon.so.1 (0xffff922ad000)

@bjorn3
Copy link
Member

bjorn3 commented Jan 23, 2023

Does gcc -static-pie curl.c -lcurl -o curl have the same issue as rustc? Rustc will pass -static-pie on MUSL unless -Ctarget-feature=-crt-static is used to disable static linking.

@jbg
Copy link

jbg commented Jan 23, 2023

curl-static is not installed, so linking fails with -static-pie (and must also fail if rust passes that when linking curl?)

$ gcc -static-pie curl.c -lcurl -o curl
/usr/lib/gcc/aarch64-alpine-linux-musl/12.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: cannot find -lcurl: No such file or directory
collect2: error: ld returned 1 exit status

After installing the various required static libraries and amending the command to link them, gcc still produces a binary with the correct loader:

$ gcc -static-pie curl.c -lcurl -lssl -lcrypto -lz -lnghttp2 -lbrotlidec -lbrotlicommon -o curl
$ ldd curl
	/lib/ld-musl-aarch64.so.1 (0xffff90fab000)

@jbg
Copy link

jbg commented Jan 24, 2023

The rust linker invocation when building the above example program with the curl crate is:

"cc" "/home/app/.rustup/toolchains/stable-aarch64-unknown-linux-musl/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crt1.o" "/home/app/.rustup/toolchains/stable-aarch64-unknown-linux-musl/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crti.o" "/home/app/.rustup/toolchains/stable-aarch64-unknown-linux-musl/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crtbegin.o" "/tmp/rustcvSxskF/symbols.o" "/home/app/tester/target/debug/deps/tester-215b1acbed4714d0.23v6dd6gnzxsr0s5.rcgu.o" "/home/app/tester/target/debug/deps/tester-215b1acbed4714d0.4dt189f4dyl9j8vw.rcgu.o" "/home/app/tester/target/debug/deps/tester-215b1acbed4714d0.4gbgwgnrj8n5169z.rcgu.o" "/home/app/tester/target/debug/deps/tester-215b1acbed4714d0.61safi306wbxgy0.rcgu.o" "/home/app/tester/target/debug/deps/tester-215b1acbed4714d0.mxi8xlc1ysy99fp.rcgu.o" "/home/app/tester/target/debug/deps/tester-215b1acbed4714d0.3c9i9azmr8v9rblc.rcgu.o" "-Wl,--as-needed" "-L" "/home/app/tester/target/debug/deps" "-L" "/usr/lib" "-L" "/lib" "-L" "/home/app/.rustup/toolchains/stable-aarch64-unknown-linux-musl/lib/rustlib/aarch64-unknown-linux-musl/lib" "-Wl,-Bstatic" "/home/app/tester/target/debug/deps/libcurl-2f6194c65c51f1e9.rlib" "/home/app/tester/target/debug/deps/libsocket2-0083cc04d93ebdab.rlib" "/home/app/tester/target/debug/deps/libcurl_sys-196fe6cf0eeb8504.rlib" "/home/app/tester/target/debug/deps/liblibc-a9a6a976b0a6ad02.rlib" "/home/app/.rustup/toolchains/stable-aarch64-unknown-linux-musl/lib/rustlib/aarch64-unknown-linux-musl/lib/libstd-ae17b16940af2365.rlib" "/home/app/.rustup/toolchains/stable-aarch64-unknown-linux-musl/lib/rustlib/aarch64-unknown-linux-musl/lib/libpanic_unwind-2b0142bfc7026d5f.rlib" "/home/app/.rustup/toolchains/stable-aarch64-unknown-linux-musl/lib/rustlib/aarch64-unknown-linux-musl/lib/libobject-7a0ca05b7d8fab32.rlib" "/home/app/.rustup/toolchains/stable-aarch64-unknown-linux-musl/lib/rustlib/aarch64-unknown-linux-musl/lib/libmemchr-9a064688bd82d012.rlib" "/home/app/.rustup/toolchains/stable-aarch64-unknown-linux-musl/lib/rustlib/aarch64-unknown-linux-musl/lib/libaddr2line-e4e5a05a96e35b9c.rlib" "/home/app/.rustup/toolchains/stable-aarch64-unknown-linux-musl/lib/rustlib/aarch64-unknown-linux-musl/lib/libgimli-6f278cc924fd0ba8.rlib" "/home/app/.rustup/toolchains/stable-aarch64-unknown-linux-musl/lib/rustlib/aarch64-unknown-linux-musl/lib/librustc_demangle-97a3515a22332f3b.rlib" "/home/app/.rustup/toolchains/stable-aarch64-unknown-linux-musl/lib/rustlib/aarch64-unknown-linux-musl/lib/libstd_detect-0ec64aafbef4d0f1.rlib" "/home/app/.rustup/toolchains/stable-aarch64-unknown-linux-musl/lib/rustlib/aarch64-unknown-linux-musl/lib/libcfg_if-b1cd2eea3d76a265.rlib" "/home/app/.rustup/toolchains/stable-aarch64-unknown-linux-musl/lib/rustlib/aarch64-unknown-linux-musl/lib/libhashbrown-bd335344f313ebfd.rlib" "/home/app/.rustup/toolchains/stable-aarch64-unknown-linux-musl/lib/rustlib/aarch64-unknown-linux-musl/lib/libminiz_oxide-45416aec543333a9.rlib" "/home/app/.rustup/toolchains/stable-aarch64-unknown-linux-musl/lib/rustlib/aarch64-unknown-linux-musl/lib/libadler-108cbf5f7b726a2f.rlib" "/home/app/.rustup/toolchains/stable-aarch64-unknown-linux-musl/lib/rustlib/aarch64-unknown-linux-musl/lib/librustc_std_workspace_alloc-1c1fe643ea1c8e21.rlib" "/home/app/.rustup/toolchains/stable-aarch64-unknown-linux-musl/lib/rustlib/aarch64-unknown-linux-musl/lib/libunwind-39a97ef386de6c62.rlib" "-lunwind" "/home/app/.rustup/toolchains/stable-aarch64-unknown-linux-musl/lib/rustlib/aarch64-unknown-linux-musl/lib/libcfg_if-7a73fff29ca7856e.rlib" "/home/app/.rustup/toolchains/stable-aarch64-unknown-linux-musl/lib/rustlib/aarch64-unknown-linux-musl/lib/liblibc-b944c66ee70b3134.rlib" "-lc" "/home/app/.rustup/toolchains/stable-aarch64-unknown-linux-musl/lib/rustlib/aarch64-unknown-linux-musl/lib/liballoc-8ea6846373ecca10.rlib" "/home/app/.rustup/toolchains/stable-aarch64-unknown-linux-musl/lib/rustlib/aarch64-unknown-linux-musl/lib/librustc_std_workspace_core-06b5cf25321492b2.rlib" "/home/app/.rustup/toolchains/stable-aarch64-unknown-linux-musl/lib/rustlib/aarch64-unknown-linux-musl/lib/libcore-7d43051c8e116420.rlib" "/home/app/.rustup/toolchains/stable-aarch64-unknown-linux-musl/lib/rustlib/aarch64-unknown-linux-musl/lib/libcompiler_builtins-06fe9ef48f62ded5.rlib" "-Wl,-Bdynamic" "-lcurl" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-nostartfiles" "-L" "/home/app/.rustup/toolchains/stable-aarch64-unknown-linux-musl/lib/rustlib/aarch64-unknown-linux-musl/lib" "-L" "/home/app/.rustup/toolchains/stable-aarch64-unknown-linux-musl/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained" "-o" "/home/app/tester/target/debug/deps/tester-215b1acbed4714d0" "-Wl,--gc-sections" "-static" "-no-pie" "-Wl,-zrelro,-znow" "-nodefaultlibs" "/home/app/.rustup/toolchains/stable-aarch64-unknown-linux-musl/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crtend.o" "/home/app/.rustup/toolchains/stable-aarch64-unknown-linux-musl/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crtn.o"

Guessing at the problematic bits, I was able to recreate the same issue with the curl.c example like this:

$ gcc -c -o curl.o curl.c
$ gcc -o curl curl.o -Wl,-Bdynamic -lcurl -static
$ ldd curl
	/lib/ld-linux-aarch64.so.1 (0xffffb387c000)
	libcurl.so.4 => /usr/lib/libcurl.so.4 (0xffffb37e9000)
	libc.musl-aarch64.so.1 => /lib/ld-linux-aarch64.so.1 (0xffffb387c000)
	libnghttp2.so.14 => /usr/lib/libnghttp2.so.14 (0xffffb37a8000)
	libssl.so.3 => /lib/libssl.so.3 (0xffffb3704000)
	libcrypto.so.3 => /lib/libcrypto.so.3 (0xffffb337e000)
	libbrotlidec.so.1 => /usr/lib/libbrotlidec.so.1 (0xffffb335d000)
	libz.so.1 => /lib/libz.so.1 (0xffffb332c000)
	libbrotlicommon.so.1 => /usr/lib/libbrotlicommon.so.1 (0xffffb32fb000)

@jbg
Copy link

jbg commented Jan 24, 2023

FWIW, folks on #musl IRC say that rust is invoking the linker in a broken way here by using -Bdynamic together with -static. The root cause of the wrong loader is that because of -static, gcc is not passing -dynamic-linker to ld and thus ld is using its default, but even if the loader was correct, the binary would be somewhat broken and have two copies of libc.

Maybe trying to dynamically link should be an error if the crt-static target feature is enabled?

@bjorn3
Copy link
Member

bjorn3 commented Jan 24, 2023

That nakes sense.

Maybe trying to dynamically link should be an error if the crt-static target feature is enabled?

At the very least it should he a warning pointing you to -Ctarget-feature=-static-crt. Not sure if ot should be an error. There may be targets that support this combination I guess.

@jbg
Copy link

jbg commented Jan 24, 2023

Is it possible for the default of crt-static to change based on the host as well as based on the target?

I can understand the motivation for making crt-static the default for the *-unknown-linux-musl targets because they have usually been used to build self-contained static binaries, often cross-compiling from a glibc host. But when you are e.g. running Alpine and trying to build programs for the local system, it would be much more sensible for it to dynamically link libc so that dynamic linking can work as normal. The Alpine packages for rust actually patch this to remove the crt-static target feature from the -musl targets, but rustup does not.

@bjorn3
Copy link
Member

bjorn3 commented Jan 24, 2023

There is an MCP to move *-musl to dynamic linking by default, but it isn't actually implemented yet and will need some care to notify everyone currently using musl for static linking. rust-lang/compiler-team#422

@jbg
Copy link

jbg commented Jan 24, 2023

#59302 is relevant

@jyn514 jyn514 added A-linkage Area: linking into static, shared libraries and binaries T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. O-musl Target: The musl libc C-bug Category: This is a bug. labels Apr 9, 2023
@Amanieu
Copy link
Member

Amanieu commented May 17, 2023

The root cause seems to be here:

NativeLibKind::Unspecified => {
if link_dynamic {
cmd.link_dylib(name, verbatim, true);
}
}

If no kind is specified in the #[link] attribute then we already attempt to link a dynamic library.

@WhyNotHugo
Copy link

Is there any reason why Rust can't link binaries dynamically by default on *-unknown-linux-musl like it does on *-unknown-linux-gnu?

It seems weird that the behaviour for one libc is different to the other, and the norm on Linux is to dynamically link libraries.

@bors bors closed this as completed in f383703 Jun 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-linkage Area: linking into static, shared libraries and binaries C-bug Category: This is a bug. O-musl Target: The musl libc T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants