println! segfaults with nightly-2018-06-24 on macOS 10.10

[SimonSapin]

Update: this is due to LLVM assuming 16-byte alignment for a #[thread_local] static that is in fact 8-bytes aligned. Bisected PR might not be directly related, and might only have moved the static from a location that happened to be 16-bytes aligned by chance?

---

$ echo 'fn main() { println!("") }' > a.rs
$ rustup run --install nightly-2018-06-24 rustc a.rs
$ ./a
Segmentation fault: 11
$ lldb a
(lldb) target create "a"
Current executable set to 'a' (x86_64).
(lldb) run
Process 26778 launched: '/private/tmp/aa/a' (x86_64)
Process 26778 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame #0: 0x000000010000a565 a`std::io::stdio::_print::h8f28cebade0a06e8 + 197
a`std::io::stdio::_print::h8f28cebade0a06e8:
->  0x10000a565 <+197>: movdqa 0x10(%rax), %xmm0
    0x10000a56a <+202>: xorps  %xmm1, %xmm1
    0x10000a56d <+205>: movaps %xmm1, 0x10(%rax)
    0x10000a571 <+209>: movl   $0x1, %ecx
Target 0: (a) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
  * frame #0: 0x000000010000a565 a`std::io::stdio::_print::h8f28cebade0a06e8 + 197
    frame #1: 0x0000000100000b2f a`a::main::h1162ee7c6f50d0b5 + 63
    frame #2: 0x0000000100000ada a`std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::hdecedfffbbfc4ed3 + 10
    frame #3: 0x000000010000e8c8 a`std::panicking::try::do_call::h586091eb2188e424 (.llvm.4887162513922290480) + 24
    frame #4: 0x000000010001af0f a`__rust_maybe_catch_panic + 31
    frame #5: 0x0000000100007712 a`std::rt::lang_start_internal::h5f9decd422205a6f + 242
    frame #6: 0x0000000100000ab5 a`std::rt::lang_start::ha7a5b042430cd1ae + 53
    frame #7: 0x0000000100000b65 a`main + 37
    frame #8: 0x00007fffcd189235 libdyld.dylib`start + 1

Reproduced on three CI builders with macOS 10.10.5, but not on two with 10.11.6.

Not reproduced with the previous Nightly, nightly-2018-06-23. Commit range: cbc4c83...60efbde. Of the PRs merged in this range, eliminating those I’m fairly certain are unrelated leaves #51696 and #51723 which I think are unrelated but am less confident about, and #51580 which touched 69 files in 19 commits.

(Found while upgrading Servo to today’s Nightly: servo/servo#21089 (comment).)

[sfackler]

Could possibly be #50586?

[kennytm]

Can't repro on 10.13.

Could you use cargo-bisect-rustc, bisect-rust or rustup-toolchain-install-master to bisect and find out the offending commit?

[SimonSapin]

Based on reading surrounding assembly, EXC_BAD_ACCESS happens when reading 0x10(%rax) where %rax is the pointer returned by the call to std::io::stdio::LOCAL_STDOUT::__getit::__KEY::h4ce6bf5c2028f9bc which is:

#[thread_local]
static __KEY: $crate::thread::__FastLocalKeyInner<$t> = …;

So yeah, thread locals on old macOS suggest the same underlying issue as #50867 and possibly #50586.

[SimonSapin]

movdqa requires 16-bytes alignment but is used to read form address 0x0000000100300228 which is only 8-bytes-aligned. This matches the description of #50586 (comment)

[SimonSapin]

With rustup-toolchain-install-master, this reproduces with 56e8f29 but not 2ea922a, which makes #51580 the PR that uncovered this bug (but likely not the root cause).

[SimonSapin]

To try and reduce further, the code below is somewhat similar to part of std::io::stdio::_print. It does not segfault (maybe this static happens to be sufficently-aligned?), but the optimized assembly does contain a similar movdqa and the LLVM IR contains load instructions that I think (?) are annotated to assume 32-bytes alignment (in both Nightly versions).

Is there an LLVM bug to open about reducing that assumption?

use std::cell::RefCell;
use std::io::Write;

thread_local! {
    static FOO: RefCell<Option<Box<Write>>> = RefCell::new(None);
}

pub fn foo() -> bool {
    FOO.with(|c| c.borrow().is_some())
}

        movq    __ZN1a3FOO7__getit5__KEY17h0d854c627114cb56E@TLVP(%rip), %rdi
        callq   *(%rdi)
        movdqa  16(%rax), %xmm0

  %2 = load i8, i8* getelementptr inbounds (<{ [40 x i8] }>, <{ [40 x i8] }>* @_ZN1a3FOO7__getit5__KEY17h0d854c627114cb56E, i64 0, i32 0, i64 32), align 32, !range !28

  %t.0.copyload11.i.i.i.i.i.i.i.i = load <4 x i64>, <4 x i64>* bitcast (<{ [40 x i8] }>* @_ZN1a3FOO7__getit5__KEY17h0d854c627114cb56E to <4 x i64>*), align 32, !alias.scope !37, !noalias !42

[landersson]

Hi, I've also run into this... Is there a good workaround? I read somewhere that enabling LTO will solve the problem, but I'm guessing that will increase compile times significantly.

[SimonSapin]

An arguably better workaround is to build on macOS 10.11 / XCode 8, or more recent. (That built program should still be able to run on some earlier OS versions.) But of course that can be more involved than flipping an LTO switch.

[landersson]

Ok, I was under the impression that XCode 8 and later doesn't support OSX 10.11.

I just tried installing the Xcode 8.3 command line tools, but the installer refuses to go ahead, saying it needs 10.12.

Has anyone been able to make this work on 10.11 by upgrading XCode?

[SimonSapin]

Per wikipedia, XCode 8.2.1 runs on macOS 10.11.5.

[kennytm]

Is it possible to test this with Xcode 9.3? Given how many times this TLS problem appears it is worth adding the test case to CI, however can't just switch our CI images back to Xcode 7 as that would be too slow for normal tests.

[SimonSapin]

Per #50867 (comment) the root cause is a bug in XCode which is fixed in XCode 8.

I think that the two realistic outcomes are:

• Try to find a reproducible test case that does not involve rustc or std, and file an LLVM bug to reduce the alignment assumption
• Or document in https://forge.rust-lang.org/platform-support.html#tier-1 some old toolchain versions as unsupported or not Tier-1

[landersson]

XCode 8.2.1 is listed as running on 10.11.5, but does only include OS X SDK version 10.12.2, so I would be reluctant to remove 7.3 and install 8.2.1 if I wanted to keep using xcode to build for 10.11.

Anyway, what just worked for me on 10.11 is to install the XCode 8.2 Command Line Tools (CLT) only, downloaded from here:

https://download.developer.apple.com/Developer_Tools/Command_Line_Tools_macOS_10.11_for_Xcode_8.2/Command_Line_Tools_macOS_10.11_for_Xcode_8.2.dmg

(you may need an apple dev account)

Running the installer will keep the current CLT installation and install the new one in /Library/Developer/CommandLineTools/

I then ran:

sudo xcode-select -s /Library/Developer/CommandLineTools/

After this, I'm able to run my rust test program (using regex) using both rust 1.27 and rust nightly in debug and release without getting a segfault.

You may need to run sudo xcode-select -r to revert back to the original CLT before being able to build with XCode again, although macports did seem to be able to build a simple command line app using the new CLT.

[nikomatsakis]

Tagging as P-high but @kennytm seems to have this in hand 🎉

[SimonSapin]

If this is a linker bug, using a different linker naively seems like it could be a solution. Shipping lld with rustc and having at least an easy opt-in for it, if not enabling it by default, could be another way forward: #39915.

[kennytm]

@SimonSapin It is a dynamic linker bug (the linker that runs when the program starts). Switching to LLD won't fix it.

[SimonSapin]

When building a program on 10.10.5 / XCode 7.2.1 and running it on a machine with a more recent OS (and presumably a more recent dynamic linker?) I still reproduce the issue.