Failed to activate YJIT for 3.2.0-dev on macOS Ventura (M1: Apple Silicon)

[hachi8833]

Unable to activate YJIT for the latest 3.2.0-dev in the following:

• M1 Macbook Pro 2021
• macOS Ventura 13.0.1
• rbenv 1.2.0-16-gc4395e5
• CommandLine Tools: 14.1.0.0.1.1666437224
• rustc 1.62.1 (e092d0b6b 2022-07-16)
• Homebrew: 3.6.12
  • openssl@3: stable 3.0.7 (bottled) [keg-only]
  • readline: stable 8.2.1 (bottled) [keg-only]
  • libyaml: stable 0.2.5 (bottled)
  • bison: stable 3.8.2 (bottled) [keg-only]

I also checked https://github.com/rbenv/ruby-build/wiki#macos and #1961 (comment) for the setup.

Regarding 3.2.0-dev (v20221121), if and only if specifying --disable-yjit, the build is successful. Adding (or not adding) --disable-shared or --enable-shared to RUBY_CONFIGURE_OPTS does not affect the success.

But without specifying --disable-yjit, such as specifying (or not specifying) --disable-shared or --enable-shared or --enable-yjit or --enable-yjit=dev, the build fails. In other words, I cannot build 3.2.0-dev unless disabling YJIT.

The following is one of the case:

export RUBY_CONFIGURE_OPTS="--with-opt-dir="$(brew --prefix openssl)" --disable-install-doc --disable-install-rdoc"

$ rbenv install 3.2.0-dev --verbose
...
generating table from enc/trans/big5.c
generating table from enc/trans/cesu_8.c
generating table from enc/trans/chinese.c
generating table from enc/trans/ebcdic.c
generating table from enc/trans/emoji.c
generating table from enc/trans/emoji_iso2022_kddi.c
generating table from enc/trans/emoji_sjis_docomo.c
generating table from enc/trans/emoji_sjis_kddi.c
generating enc/trans/cesu_8.c ...
generating table from enc/trans/emoji_sjis_softbank.c
generating enc/trans/ebcdic.c ...
linking miniruby
generating enc/trans/emoji.c ...
generating table from enc/trans/escape.c
generating enc/trans/emoji_sjis_docomo.c ...
ld: warning: ignoring file yjit/target/release/libyjit.a, building for macOS-arm64 but attempting to link with file built for macOS-x86_64
generating enc/trans/chinese.c ...
Undefined symbols for architecture arm64:
  "_rb_yjit_before_ractor_spawn", referenced from:
      _ractor_create in ractor.o
  "_rb_yjit_bop_redefined", referenced from:
      _rb_vm_check_redefinition_opt_method in vm.o
  "_rb_yjit_call_threshold", referenced from:
      _rb_vm_exec in vm.o
      _vm_sendish in vm.o
  "_rb_yjit_cme_invalidate", referenced from:
      _clear_method_cache_by_id_in_class in vm.o
  "_rb_yjit_code_gc", referenced from:
      _mjit_compile_invokebuiltin_for_rb_yjit_code_gc in yjit.o
      _Init_builtin_yjit.yjit_table in yjit.o
  "_rb_yjit_constant_ic_update", referenced from:
      _vm_exec_core in vm.o
  "_rb_yjit_constant_state_changed", referenced from:
      _rb_clear_constant_cache_for_id in vm.o
  "_rb_yjit_disasm_iseq", referenced from:
      _mjit_compile_invokebuiltin_for_rb_yjit_disasm_iseq in yjit.o
      _Init_builtin_yjit.yjit_table in yjit.o
  "_rb_yjit_enabled_p", referenced from:
      _rb_jit_cont_finish in cont.o
      _rb_jit_cont_init in cont.o
      _fiber_initialize in cont.o
      _rb_threadptr_root_fiber_setup in cont.o
      _cont_free in cont.o
      _cont_capture in cont.o
      _clear_method_cache_by_id_in_class in vm.o
      ...
  "_rb_yjit_get_exit_locations", referenced from:
      _mjit_compile_invokebuiltin_for_rb_yjit_get_exit_locations in yjit.o
      _Init_builtin_yjit.yjit_table in yjit.o
  "_rb_yjit_get_stats", referenced from:
      _mjit_compile_invokebuiltin_for_rb_yjit_get_stats in yjit.o
      _Init_builtin_yjit.yjit_table in yjit.o
  "_rb_yjit_init_rust", referenced from:
      _rb_yjit_init in yjit.o
  "_rb_yjit_insns_compiled", referenced from:
      _mjit_compile_invokebuiltin_for_rb_yjit_insns_compiled in yjit.o
      _Init_builtin_yjit.yjit_table in yjit.o
  "_rb_yjit_iseq_free", referenced from:
      _rb_iseq_free in iseq.o
  "_rb_yjit_iseq_gen_entry_point", referenced from:
      _rb_yjit_compile_iseq in yjit.o
  "_rb_yjit_iseq_mark", referenced from:
      _rb_iseq_mark in iseq.o
  "_rb_yjit_iseq_update_references", referenced from:
      _rb_iseq_update_references in iseq.o
  "_rb_yjit_parse_option", referenced from:
      _proc_options in ruby.o
  "_rb_yjit_reset_stats_bang", referenced from:
      _mjit_compile_invokebuiltin_for_rb_yjit_reset_stats_bang in yjit.o
      _Init_builtin_yjit.yjit_table in yjit.o
  "_rb_yjit_root_mark", referenced from:
      _yjit_root_type in yjit.o
  "_rb_yjit_simulate_oom_bang", referenced from:
      _mjit_compile_invokebuiltin_for_rb_yjit_simulate_oom_bang in yjit.o
      _Init_builtin_yjit.yjit_table in yjit.o
  "_rb_yjit_stats_enabled_p", referenced from:
      _mjit_compile_invokebuiltin_for_rb_yjit_stats_enabled_p in yjit.o
      _Init_builtin_yjit.yjit_table in yjit.o
  "_rb_yjit_trace_exit_locations_enabled_p", referenced from:
      _mjit_compile_invokebuiltin_for_rb_yjit_trace_exit_locations_enabled_p in yjit.o
      _Init_builtin_yjit.yjit_table in yjit.o
  "_rb_yjit_tracing_invalidate_all", referenced from:
      _tracepoint_enable_m in vm_trace.o
      _update_global_event_hook in vm_trace.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [miniruby] Error 1
make: *** Waiting for unfinished jobs....
generating enc/trans/emoji_iso2022_kddi.c ...
generating table from enc/trans/gb18030.c
generating table from enc/trans/gbk.c
generating table from enc/trans/iso2022.c
generating enc/trans/emoji_sjis_kddi.c ...
generating table from enc/trans/japanese.c
generating table from enc/trans/japanese_euc.c
generating enc/trans/emoji_sjis_softbank.c ...
generating enc/trans/big5.c ...
generating table from enc/trans/japanese_sjis.c
generating table from enc/trans/korean.c
generating enc/trans/escape.c ...
generating table from enc/trans/single_byte.c
generating enc/trans/iso2022.c ...
generating enc/trans/japanese.c ...
generating table from enc/trans/utf8_mac.c
generating enc/trans/gbk.c ...
generating table from enc/trans/utf_16_32.c
generating enc/trans/gb18030.c ...
generating enc/trans/japanese_sjis.c ...
generating enc/trans/japanese_euc.c ...
generating enc/trans/utf_16_32.c ...
generating enc/trans/korean.c ...
generating enc/trans/single_byte.c ...
generating enc/trans/utf8_mac.c ...

[hachi8833]

Configuration summary from the log:

Configuration summary for ruby version 3.2.0

   * Installation prefix: /Users/hachi8833/.anyenv/envs/rbenv/versions/3.2.0\
                          -dev
   * exec prefix:         ${prefix}
   * arch:                arm64-darwin22
   * site arch:           ${arch}
   * RUBY_BASE_NAME:      ruby
   * ruby lib prefix:     ${libdir}/${RUBY_BASE_NAME}
   * site libraries path: ${rubylibprefix}/${sitearch}
   * vendor path:         ${rubylibprefix}/vendor_ruby
   * target OS:           darwin22
   * compiler:            clang
   * with thread:         pthread
   * with coroutine:      arm64
   * enable shared libs:  no
   * dynamic library ext: bundle
   * CFLAGS:              -fdeclspec ${optflags} ${debugflags} ${warnflags}
   * LDFLAGS:             -L. \
                          -L/Users/hachi8833/.anyenv/envs/rbenv/versions/3.2\
                          .0-dev/lib -L/opt/homebrew/opt/openssl@3/lib \
                          -fstack-protector-strong \
                          -L/opt/homebrew/opt/gmp/lib \
                          -L/opt/homebrew/opt/openssl@3/lib
   * DLDFLAGS:            -L/Users/hachi8833/.anyenv/envs/rbenv/versions/3.2\
                          .0-dev/lib -L/opt/homebrew/opt/openssl@3/lib \
                          -L/opt/homebrew/opt/gmp/lib \
                          -L/opt/homebrew/opt/openssl@3/lib \
                          -Wl,-multiply_defined,suppress
   * optflags:            -O3 -fno-fast-math
   * debugflags:          -ggdb3
   * warnflags:           -Wall -Wextra -Wextra-tokens \
                          -Wdeprecated-declarations -Wdivision-by-zero \
                          -Wdiv-by-zero -Wimplicit-function-declaration \
                          -Wimplicit-int -Wmisleading-indentation \
                          -Wpointer-arith -Wshorten-64-to-32 \
                          -Wwrite-strings -Wold-style-definition \
                          -Wmissing-noreturn -Wno-cast-function-type \
                          -Wno-constant-logical-operand -Wno-long-long \
                          -Wno-missing-field-initializers \
                          -Wno-overlength-strings -Wno-parentheses-equality \
                          -Wno-self-assign -Wno-tautological-compare \
                          -Wno-unused-parameter -Wno-unused-value \
                          -Wunused-variable -Wundef
   * strip command:       strip -A -n
   * install doc:         no
   * MJIT support:        yes
   * YJIT support:        yes
   * man page type:       doc
   * BASERUBY -v:         ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) \
                          [arm64-darwin21]

---
	BASERUBY = /Users/hachi8833/.anyenv/envs/rbenv/shims/ruby --disable=gems
	CC = clang
	LD = clang
	LDSHARED = clang -dynamiclib
	CFLAGS = -fdeclspec -O3 -fno-fast-math -ggdb3 -Wall -Wextra -Wextra-tokens -Wdeprecated-declarations -Wdivision-by-zero -Wdiv-by-zero -Wimplicit-function-declaration -Wimplicit-int -Wmisleading-indentation -Wpointer-arith -Wshorten-64-to-32 -Wwrite-strings -Wold-style-definition -Wmissing-noreturn -Wno-cast-function-type -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wunused-variable -Wundef   -pipe 
	XCFLAGS = -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fno-strict-overflow -fvisibility=hidden -DRUBY_EXPORT -fPIE -I. -I.ext/include/arm64-darwin22 -I./include -I. -I./enc/unicode/14.0.0 
	CPPFLAGS = -I/Users/hachi8833/.anyenv/envs/rbenv/versions/3.2.0-dev/include -I/opt/homebrew/opt/openssl@3/include -I/opt/homebrew/opt/gmp/include -I/opt/homebrew/opt/openssl@3/include -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -D_DARWIN_UNLIMITED_SELECT -D_REENTRANT   
	DLDFLAGS = -L/Users/hachi8833/.anyenv/envs/rbenv/versions/3.2.0-dev/lib -L/opt/homebrew/opt/openssl@3/lib -L/opt/homebrew/opt/gmp/lib -L/opt/homebrew/opt/openssl@3/lib -Wl,-multiply_defined,suppress -fstack-protector-strong -Wl,-pie -framework CoreFoundation  
	SOLIBS = yjit/target/release/libyjit.a -lgmp -ldl -lobjc -lpthread 
	LANG = ja_JP.UTF-8
	LC_ALL = 
	LC_CTYPE = 
	MFLAGS = - --jobserver-fds=6,7 -j
	RUSTC = rustc
	YJIT_RUSTC_ARGS = --crate-name=yjit --crate-type=staticlib --edition=2021 -C opt-level=3 -C overflow-checks=on '--out-dir=/var/folders/6_/412tm5md0svdq66fxwwgn0tw0000gn/T/ruby-build.20221122202816.98147.TB2j8e/ruby-master/yjit/target/release/' ./yjit/src/lib.rs

[mislav]

Hi, not sure what went wrong, but keep in mind that Ruby 3.2 is in a preview stage. If you can reproduce this failure in a fresh checkout of Ruby 3.2 by running ./configure --prefix=/path/to/ruby && make && make install, this should be reported to the Ruby project rather than here. Thank you!

[hachi8833]

Thank you for the suggestion! I'd try the build from source and report to ruby/ruby

[hsbt]

I didn't see this error recently. I'll handle this on our tracker that is https://bugs.ruby-lang.org/. Thanks.

[hachi8833]

FYI: Thanks to maximecb, building from source was successful!

https://bugs.ruby-lang.org/issues/19146
https://bugs.ruby-lang.org/projects/ruby-master/repository/git/revisions/a81c89b7c877cc0ddf40a8d4672ca90090e116c6

[hachi8833]

FYI: After that, I found that building 3.2.0-dev with rbenv+ruby-build still fails, while source build is now certainly successful:

# successful via source build:
$ ./autogen.sh;mkdir build && cd build;mkdir ~/.rubies;../configure --prefix="${HOME}/.rubies/ruby-master" --disable-install-doc --disable-install-rdoc;make install
.
.
.
$ RUBYOPT='--yjit' ~/.rubies/ruby-master/bin/ruby -v
ruby 3.2.0dev (2022-11-26T06:45:33Z master 26794b95cf) +YJIT [arm64-darwin22.1.0]

I also confirmed that rbenv+ruby-build and export RUBY_CONFIGURE_OPTS="--disable-install-doc --disable-install-rdoc" is successful:

# successful unless specifying --enable-yjit
$ export RUBY_CONFIGURE_OPTS="--disable-install-doc --disable-install-rdoc"
$ rbenv install 3.2.0-dev --verbose
.
.
.
# RUBYOPT='--yjit' ruby -v
ruby: warning: Ruby was built without YJIT support
ruby 3.2.0dev (2022-11-26T06:45:33Z master 26794b95cf) [arm64-darwin22]

But if specifying --enable-yjit, the build fails as noted first.

# fails if specifying --enable-yjit
$ export RUBY_CONFIGURE_OPTS="--enable-yjit --disable-install-doc --disable-install-rdoc"
$ rbenv install 3.2.0-dev --verbose
.
.
.
ld: warning: ignoring file yjit/target/release/libyjit.a, building for macOS-arm64 but attempting to link with file built for macOS-x86_64
generating enc/trans/chinese.c ...
Undefined symbols for architecture arm64:
  "_rb_yjit_before_ractor_spawn", referenced from:
      _ractor_create in ractor.o
.
.
.

(Note that Rosetta 2 has been installed in my environment and the following rustup toolchain setup is needed, as maximecb suggested above)

$ rustup toolchain list
stable-aarch64-apple-darwin (override)
stable-x86_64-apple-darwin (default)

Attached the comparison the configuration summaries between source build and rbenv+ruby-build:

• left side: source build
• right side: rbenv+ruby-build

compare_source_rbenv.pdf

I noticed:

• the versions in arch and target OS are slightly different🤔
  • arch:
    • source build: arm64-darwin22.1.0
    • rbenv+ruby-build: arm64-darwin22
  • target OS:
    • source build: darwin22.1.0
    • rbenv+ruby-build: darwin22
• RBS has not been built rbenv+ruby-build 🤔
• Current Ruby 3.2.0-dev (master 26794b95cf) does not need --enable-yjit option to enable YJIT, and it looks like rbenv+ruby-build still requires this option to enable YJIT.

I recognize that 3.2.0-dev is just for a preview, and I'm not sure this might be fixed in the final release or some other adjustments are needed.
Thus I note this info just for your interests by now.
If this issue I noted is not a big deal for rbenv, just ignore this 🙏. Anyway I can try 3.2.0-dev via source build.
Thank you,

[mislav]

Thank you for sharing your experience in detail.

Note that the latest ruby-build does not explicitly detect nor try to enable YJIT anymore: #2091

[hachi8833]

FYI: I finally found the cause of the failure: I incorrectly performed the following two types of installations:

• brew install rustup-init; rustup-init
• curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh (via https://www.rust-lang.org/ja/tools/install)

That was my failure.

By running rustup self uninstall to uninstall the second one, I got successful with building 3.2.0-dev with rbenv+ruby-build.

$ rbenv install 3.2.0-dev
$ ruby -v --yjit
ruby 3.2.0dev (2022-11-30T08:20:38Z master c8bfbbc25e) +YJIT [arm64-darwin22]

Sorry for disturbing you🙏

[jzmalik123]

Worked for me. Thanks