Install openssl whenever the system openssl version is not supported by a Ruby

[znz]

Ubuntu 22.04 includes openssl 3.0.2, and libssl-dev is also 3.0.2 only.
But Ruby < 3.1 does not support openssl 3.
So I want to install openssl 1 automatically.

I just tested rbenv install 3.0.4 and rbenv install 2.7.6 with this on Ubuntu 22.04.
Sorry if this makes trouble on other platforms.

[eregon]

Thank you for the PR, I agree this is useful and needed, given ruby-core seems unwilling to add openssl 3 support to Ruby 3.0 and 2.7: https://bugs.ruby-lang.org/issues/18658#note-8
I will also need this to build Ruby <= 3.0 on the upcoming ubuntu 22.04 github action runners.

[mislav]

@znz I appreciate the effort that went into this, and I do agree that the current situation on Ubuntu is unfriendly for our users. However, I'd be hard to convince that this is the way to go.

Our current has_mac_broken_openssl approach was initially made because Ruby was usually absolutely uninstallable on macOS due to its patched system OpenSSL version. It was a way to ease Ruby installation for macOS users, but it opened a whole can of worms:

• It drastically slowed down installation of every Ruby version because every rbenv install call would compile its own contained openssl instance;
• It made the ruby-build project responsible for debugging when that compilation failed, which we (ruby-build maintainers) didn't always have capacity nor knowledge to do;
• It wouldn't result in a great long-term experience for our users because, since every Ruby version would have its own openssl instance, a new allow-listed certificate added to the system's certificate bundle wouldn't propagate to or be known by Ruby versions;
• Since the user wasn't responsible or even aware of openssl having been built on their behalf for each Ruby version, they typically also wouldn't apply openssl security upgrades.

Basically, all of this would also happens on other platforms too if we started to extend this approach outside of just macOS. However, what I would ideally like to see is that ruby-build stops building or maintaining its own openssl versions. Instead, a ruby-build process should ideally just quickly error out if a compatible openssl version wasn't found on the system, with instructions on how the user should take initiative to fix it. For example, macOS users should ideally just install openssl@1.1 via Homebrew and link their Rubies to that:

brew install openssl@1.1
export RUBY_CONFIGURE_OPTS="--with-openssl-dir=$(brew --prefix openssl@1.1)"

We could optionally start to print these instructions, or equivalent instructions on other systems, with the idea being that the user should be responsible for this. Sure, that would result in users having more to do themselves, but I think it would be in their interest that they are in control.

@eregon @hsbt Thoughts?

[eregon]

It drastically slowed down installation of every Ruby version

Certainly better than failing. Maybe we could install openssl in a well-known place which could be reused if the same openssl version is used by multiple Ruby versions.

It made the ruby-build project responsible for debugging when that compilation failed

This definitely sucks, but I think we'd get almost as much reports even without that. Failing to compile ruby feeks like 90% from openssl issues, and telling people to install manually wouldn't improve that, I don't see it likely people would report to upstream openssl, and they likely wouldn't be able to help with e.g. how ruby & ruby-build use that openssl.
IMHO many such issues should be filed to https://bugs.ruby-lang.org/ mostly, because errors are too obscure and it's too easy to pick a mix of openssl headers and libs from different versions (e.g., postmodern/ruby-install#412 (comment)), but that's not what happens.

It wouldn't result in a great long-term experience for our users because, since every Ruby version would have its own openssl instance, a new allow-listed certificate added to the system's certificate bundle wouldn't propagate to or be known by Ruby versions;

That's a non-issue with this PR, because it symlinks the system certificates.

with instructions on how the user should take initiative to fix it.

The problem is this is very annoying to do on Ubuntu 22.04 and other recent distros which default to openssl 3.
AFAIK it is not possible to install an openssl 1.1 with the package manager (apt), the package manager only knows openssl 3 and no other openssl version. It's always been like that, there can only be one system openssl, i.e., one version in /usr/include/openssl/opensslv.h (in some old distros one could choose between 1.0 and 1.1, now it's just 3 and that's it).
So the only solution would be to compile manually from source, which seems worse to ask users to do manually vs ruby-build doing it.
Maybe ruby-build could fail by default in a system-openssl3 but need 1.1 situation, but have a flag to opt-in to make this more explicit?

The situation is already a pain and already existed for Ruby <= 2.3 (which doesn't compile with openssl 1.1 and needs 1.0), but those are old so it not so commonly a problem. But here it's simply not possible to build Ruby 3.0 on Ubuntu 22.04 with ruby-build, unless you manually download & compile openssl and link the certificates.

There is another issue to not use the system openssl not mentioned above:

• When a Ruby is not compiled against the system openssl (i.e. installed by the package manager), it means that Ruby does not work with other packages installed by that package manager if those packages depend on openssl. As an example database packages typically depend on openssl and are typically installed by the packager manager so mysql/postgres/etc typically don't work in such cases, because one ends up with two libssl loaded in the same process and that either breaks or segfaults. The workaround would be to manually compile these databases against the openssl Ruby was built against, which seems very difficult.

That reason is IMHO why Ruby 3.0 should support openssl 3, otherwise it's not possible/very difficult to use Ruby 3.0 with database drivers on Ubuntu 22.04.

BTW on macOS I think ruby-build should use Homebrew's openssl by default to avoid this issue, because that's the "system libssl" of macOS for most users (i.e., they'd install mysql/postgres via brew and I think those link against the Homebrew openssl). But I don't know the details of openssl on macOS anymore, it seems it ships with LibreSSL nowadays?

[eregon]

Related discussion: #1940

[mislav]

Maybe we could install openssl in a well-known place which could be reused

That would be great! If ruby-build is invoked in the context of rbenv, maybe we could have a shared openssl installation under ~/.rbenv/openssl or something like that?

That's a non-issue with this PR, because it symlinks the system certificates.

Ah, true; thanks for pointing that out. I guess this wasn't possible on macOS because there were no physical files to link to.

That reason is IMHO why Ruby 3.0 should support openssl 3

Agreed—in the ideal world. I don't have the expertise to know what would actually be involved with that.

@eregon In the interest on making this changeset minimal, how would you feel about building this cross-platform openssl functionality into the old functions mac_openssl and has_mac_broken_openssl, so that we don't have to do rename all across all current build formulae? Sure, the functions would then become a slight misnomer (since they would apply to all systems, not just macOS), but we could address that in the future.

[eregon]

I guess you meant to ping @znz.
IMHO it's good to change the definitions for clarity.
I think we should give it a fully general name like:

install_package "openssl-1.0.2u" ..." openssl --if system_openssl_version_does_not_match

This then gives us all the flexibility in the check, and we will (most likely) never need to rename that again.

[eregon]

That would be great! If ruby-build is invoked in the context of rbenv, maybe we could have a shared openssl installation under ~/.rbenv/openssl or something like that?

Maybe simply ~/.ruby-build/openssl?
I also thought of ~/.rubies/openssl too (IMHO the best place to install Rubies in a Ruby-switcher-agnostic way).
It could also be configurable.

[mislav]

No, I meant to ping you as a maintainer.

I think we should give it a fully general name like:

install_package "openssl-1.0.2u" ..." openssl --if system_openssl_version_does_not_match

Sure, that's descriptive, but I find it verbose and scary-looking. How about something generic and simpler, since this is going to be repeated across hundreds of build definitions:

openssl --if needs_openssl

[mislav]

Maybe simply ~/.ruby-build/openssl?
I also thought of ~/.rubies/openssl too

I'd be open to that, but it would break the unspoken contract of rbenv never writing to locations outside of ~/.rbenv, and ruby-build never writing anything to locations outside of the build target directory.

It could also be configurable.

Like most features of ruby-build, I definitely agree it should be configurable via an environment variable.

[eregon]

openssl --if needs_openssl sounds fine to me, it's a bit less clear (Ruby always "needs" openssl otherwise it's barely usable) but that probably doesn't matter too much, the behavior regarding openssl should anyway be documented in the README or so.

I'd be open to that, but it would break the unspoken contract of [...] ruby-build never writing anything to locations outside of the build target directory.

Right, but that is unavoidable if we want to share the openssl installation between different Ruby versions.
Maybe the default should remain "inside the ruby installation prefix" then to honor that contract, although it would mean non-rbenv users would miss on the build time gains.

For rbenv we could just have rbenv pass the path for where to put openssl (SOME_ENV_VAR=~/.rbenv).

BTW, I was thinking we'd install openssl e.g. in ~/.rbenv/openssl-1.0.2u/, so include the version in it, always and append /openssl-$VERSION to the given path.

[mislav]

Maybe the default should remain "inside the ruby installation prefix" then to honor that contract, although it would mean non-rbenv users would miss on the build time gains.

Non-rbenv users could set something like RUBY_BUILD_OPENSSL_DIR to keep openssl in a shared location. We do the same with speeding up downloads: rbenv users have ruby-build caching downloads to ~/.rbenv/cache (if it exists) and non-rbenv users have to explicitly specify RUBY_BUILD_CACHE_PATH.

[dentarg]

However, what I would ideally like to see is that ruby-build stops building or maintaining its own openssl versions.

I like that ruby-build builds OpenSSL for each Ruby version I install.

I used to use ruby-install to build/install Ruby on my Mac. Often I discovered issues with Ruby versions previously built after a brew upgrade upgraded OpenSSL. I think I had to re-build the older Rubies to get them usable again. I've never run into that since switching to ruby-build.

[mislav]

Often I discovered issues with Ruby versions previously built after a brew upgrade upgraded OpenSSL. I think I had to re-build the older Rubies to get them usable again.

That used to happen to me as well. I think this was because Homebrew by default does aggressive pruning of older versions of software whenever it installs a newer version. Once I've configured export HOMEBREW_NO_INSTALL_CLEANUP=true globally on my system, my rubies were never broken even though they were linked to brew --prefix openssl@1.1

[eregon]

For CRuby 3.1+, OpenSSL 3 is supported, so there we should not install a custom openssl but use the system openssl.
So this could remain --if has_broken_mac_openssl.

I'm thinking we should have needs_openssl3 (3.0 or older, so not 4+), needs_openssl11 (1.1 or older, so not 3+), needs_openssl10 (1.0 or older, so not 1.1+).
For CRuby 3.1+ we'd use needs_openssl3, for other CRuby versions which install_package "openssl-1.1 it'd be needs_openssl11 and for install_package "openssl-1.0 it'd be needs_openssl10.
That could be done in this PR or later, but we should not install openssl for CRuby 3.1+ on non-macOS.

[eregon]

@mislav Could you do another review?
For me this looks good, modulo the comments above.

I think we also need to choose, either:

• This approach (install openssl for the user when system openssl is not compatible, this PR handles system openssl 3, but I think we should generalize it soon after). Pros is it's convenient and known to work relatively well. Cons is the Ruby built this way won't work with system packages depending on the system openssl (e.g., some databases do).
• Avoid installing openssl as much as possible, and rather automatically use workarounds like https://bugs.ruby-lang.org/issues/18658#note-4 (so build without openssl, then install a proper openssl gem). I'm not sure if workarounds like that exist for e.g. system openssl 1.1 and Ruby needs openssl 1.0 like in CRuby < 2.4 though. Pros is using the "system openssl", always.

[eregon]

@mislav Any thought on the above?

[mislav]

Sorry for the delay! This is looking good. Some style suggestions and clarifying questions about Ruby 3.2

[mislav]

This looks great; thank you for your work and patience!

As far as I'm concerned, this can ship. One final thought I have about all this is forwards compatibility: it seems that with this change, we've kicked the can down the street (by redefining the semantics that were previously only around OpenSSL v1 to be OpenSSL v3-aware), but if anything significantly changes with how OSs package OpenSSL in the future (e.g. they start shipping OpenSSL v4), we'll have a set of problems again because we've hardcoded that needs_openssl in ruby-build means that it specifically needs OpenSSL v1 (and that it can't be Apple-patched OpenSSL/LibreSSL).

Ideally, in the future, needs_openssl would understand the precise version that a Ruby version needs. To a human, that version requirement can be inferred from the formula itself:

install_package "openssl-<ver>" "https://www.openssl.org/source/openssl-<ver>.tar.gz#<sha>" openssl --if <condition>

except that when the condition is evaluated, it doesn't have access to the openssl-<ver> bit. We could figure this out in subsequent changes.

[yob]

For future travelers that find this and need to get ruby 3.0 installed on a system with openssl 3.0, here's one possible workaround:

https://gist.github.com/yob/08d53a003181aa0fcce9812b1b533870

tl;dr trick ruby-build into install ruby without openssl, manually download the openssl 3.0 gem using wget and install it.

Rather than install openssl 1.1 to compile ruby, is it possible to teach ruby-build an opt-in version of the above?

[eregon]

Rather than install openssl 1.1 to compile ruby, is it possible to teach ruby-build an opt-in version of the above?

Discussed in #1974 (comment) and https://bugs.ruby-lang.org/issues/18658#note-11.
So not really, because that technique does not work for Ruby < 2.6.
And also IMHO it feels quite hacky and not the right way to build Ruby (CRuby should really backport openssl 3 support to 2.7/3.0 IMHO: ruby/openssl#517, TruffleRuby did that).

[eregon]

I will try to rebase this and finish reviewing this PR. I'll squash all commits because otherwise it seems impossible to rebase with the recent openssl update. Original commits at master...eregon:ruby-build:install-openssl_1-when-openssl_is_3-bak

[eregon]

I've pushed a commit so the logic is intuitive and general:

Install openssl whenever the system openssl version is not supported by that Ruby / the system openssl version cannot be used for that Ruby.

And for that I also made all supported versions explicit with needs_openssl_101_111, etc.
I tested locally and the version tests work well.

@mislav Could you review my Bash code added in ruby-build?
Then I think this should be ready to be merged and released.

[eregon]

Actually, let me merge this and release it now, to make sure we don't get conflicts again.

[mislav]

Thanks @znz for your hard work and @eregon for review, your patience, and additional fixes 🙇