Support OpenSSL 3.0

[rhenium]

OpenSSL 3.0 is scheduled to be released later this year. It is a major version bump from 1.1 and contains architecture changes that affect Ruby/OpenSSL.

From https://www.openssl.org/policies/releasestrat.html:

The following alpha and beta releases for OpenSSL 3.0 are currently scheduled. Note that these dates are subject to change and alpha or beta releases may be inserted or removed as required:

alpha1, 2020-03-31: Basic functionality plus basic FIPS module
alpha2, 2020-04-21: Complete external provider support (serialization, support for new algs, support for providers which only include operations in a class)
alpha3, 2020-05-21: Aiming to test the API completeness before beta1 freezes it)
beta1, 2020-06-02: Code complete (API stable, feature freeze)
betaN: Other beta releases TBD
Final: 2020 early Q4

The design is outlined in the web page:

https://www.openssl.org/docs/OpenSSL300Design.html

Unlike OpenSSL 1.0 -> 1.1, not so many changes are required to make it just compile, but a lot of deprecation warnings are generated while compiling and many test cases are currently failing when compiled against OpenSSL's master.

• OpenSSL::HMAC needs a rewrite with the EVP API as it currently uses the low-level HMAC_*() functions.
• Subclasses of OpenSSL::PKey, such as RSA or DSA, provide access to those low-level functions. The following methods need rewrite:
  • PKey::*#generate (and an overload of .new)
    • This can probably be implemented in pure-Ruby with PKey.generate_parameters and PKey.generate_key.
  • Low-level sign/verify methods that take prehashed values.
    • RSA#{private,public}_{encrypt,decrypt}
    • DSA#syssign and #sysverify
    • EC#dsa_sign_asn1 and #dsa_verify_asn1
• The ENGINE API is deprecated in favor of "Provider"s.

[rhenium]

Updated TODO list as of alpha14:

• OpenSSL::HMAC, Use the EVP API

  • No incompatibilities expected for Ruby programs.
  • Done by hmac: migrate from the low-level HMAC API to the EVP API #371.

• OpenSSL::PKey, Parameters/key generation with the EVP API

  • No incompatibilities expected.
  • Done by pkey: use high level EVP interface to generate parameters and keys #397.
    • Added OpenSSL::PKey.generate_key and OpenSSL::PKey.generate_parameters.
    • Rewritten OpenSSL::PKey::{RSA,DSA,DH}.{new,generate}.

• OpenSSL::PKey::*, Low-level methods for signature/encryption/key agreement

  • RSA#{private,public}_{encrypt,decrypt}, DSA#syssign, DSA#sysverify, EC#dsa_sign_asn1, and EC#dsa_verify_asn1.
  • No incompatibilities expected.
  • Done by pkey: implement PKey#encrypt, #decrypt, #sign_raw, #verify_raw, and #verify_recover #382.
    • Added OpenSSL::PKey.{encrypt,decrypt,sign_raw,verify_raw,verify_recover}.

• OpenSSL::PKey::*, Getters for parameters/key components

  • RSA#{n,e,d,p,...}, etc.
  • No incompatibilities expected.
  • Needs a rewrite with EVP_PKEY_get_params() family (only exists in 3.0)

• OpenSSL::PKey::*, Setters for parameters/key components

  • {RSA,DSA,DH}#set_* and EC#{private_key=,public_key=,group=}
  • Feature removed without replacement. Keys are now immutable once created - all components must be specified at once.
  • Use cases definitely exist, a new interface is required.
    • EVP_PKEY_fromdata() requires the caller to specify what the pkey is: parameters only, public key only, or private key?
    • OpenSSL::PKey.new_private_key("RSA", n: 123, e: 456, d: 789)?

• OpenSSL::PKey::{RSA,DSA}, Private/public key decoders for unpopular/non-standard formats

  • OpenSSL::PKey::DSA.new accepts PEM encoded "DSAPublicKey" format ("DSA PUBLIC KEY" header).
  • OpenSSL::PKey::RSA.new accepts DER/PEM encoded "RSAPublicKey" format ("RSA PUBLIC KEY" header).
  • ruby-openssl has no ability to export keys in such formats as of [Bug #4422].
  • Would it be OK to drop them? We need to write our decoder using OpenSSL::ASN1.decode if we want to keep it.

• OpenSSL::PKey::*, Methods that need a rewrite, which can be done with existing (<= 1.1.1) API

  • #to_text
  • #public_key (the method to create a copy with private components removed)
  • DH#params_ok? and EC#check_key
  • Done in Use EVP API in more places #436.

• OpenSSL::PKey::*, Methods that need a rewrite, which can be done in a compatible way, but requires new functions in 3.0.0

  • #params
  • #initialize_copy - annoying because EVP_PKEY_dup() only exists in OpenSSL 3.0.
  • #private? and #public?

• OpenSSL::PKey::{DH,EC}, #generate_key!

  • Feature removed without replacement. Keys are now immutable once created.
  • Print deprecation warning and suggest using OpenSSL::PKey.generate_key(pkey_obj_without_key_components).

• OpenSSL::PKey::EC::Group and OpenSSL::PKey::EC::Point

  • Effectively removed without replacement, though most of the functions are not explicitly marked as deprecated.
  • Because the low-level key generation is deprecated and the EVP API can't create keys using an EC_GROUP object, there is no use for this class.
  • Information that used to be obtained through EC_GROUP object (curve's name or parameters) are directly obtained from EVP_PKEY object with EVP_PKEY_get_params().

• OpenSSL::Engine

  • The entire ENGINE API is deprecated in favor of Providers, but it's new and is a completely different concept.
  • ENGINE should work if we ignore the deprecation warnings, but I doubt if there will be any ENGINE implementations that work with OpenSSL 3.0 anyway.

• OpenSSL::SSL, tmp_dh_callback

  • The callback used on the server side for ephemeral DH. Called during the handshake, but there is no reason for it to be a callback (anymore).
  • Replaced by SSL_CTX_set_tmp_dh() or SSL_CTX_set_dh_auto().
  • Added SSLContext#tmp_dh= by ssl: add SSLContext#tmp_dh=  #459.

[zzak]

This is awesome, thanks for your hard work @rhenium!

[ojab]

Not sure if it's a known issue, but

require 'openssl'

ec_key = OpenSSL::PKey::EC.generate('prime256v1')
cipher = OpenSSL::Cipher.new('aes-128-cbc-hmac-sha256')
ec_key.export(cipher, 'key_password')

leads to

$ ruby /tmp/2.rb 
/tmp/2.rb:5:in `export': PEM_write_bio_PrivateKey_traditional: cipher operation failed (OpenSSL::PKey::PKeyError)
	from /tmp/2.rb:5:in `<main>'

on Fedora-37 (openssl-3.0.5) and openssl gem from master (1ddbf28)

[akostadinov]

@ojab , shouldn't you be using an EC algorithm when using an EC key?

[ojab]

@akostadinov Nope, aes-128-cbc-hmac-sha256 is encrypting generic string (it's EC key in the example above, just an extract from the existing code). Could be as well:

require 'openssl'

data = "Very, very confidential data"
cipher = OpenSSL::Cipher.new('aes-128-cbc-hmac-sha256')
cipher.encrypt

key = cipher.random_key
iv = cipher.random_iv

encrypted = cipher.update(data)
encrypted += cipher.final
p encrypted

which also fails with cipher operation failed on cipher#update

[ojab]

But that's a good point, because

require 'openssl'

ec_key = OpenSSL::PKey::EC.generate('prime256v1')
cipher = OpenSSL::Cipher.new('aes-256-gcm')
pem = ec_key.export(cipher, 'key_password')

p OpenSSL::PKey.read(pem, 'key_password')

fails with

/tmp/3.rb:7:in `read': Could not parse PKey: bad decrypt (OpenSSL::PKey::PKeyError)
	from /tmp/3.rb:7:in `<main>'

Maybe it's covered by unchecked points above, but dunno.

EDIT: okay, gcm is hard and I can't get it working even with openssl-1.

[collimarco]

Do you have any suggestions to update the Ruby web-push library to use OpenSSL v3?

I can replace OpenSSL::PKey::EC.new with OpenSSL::PKey::EC.generate, but then I find it hard to update this file to v3:

https://github.com/pushpad/web-push/blob/master/lib/web_push/vapid_key.rb

The gem has good test coverage and is already updated to use the openssl gem v3, but tests pass only on with the C library v1.1 (and not with C library v3).

Any suggestions?

[collimarco]

# input: the base64 string of the public key and the base64 string of private key
# output: an OpenSSL::PKey::EC with that keys

public_key = OpenSSL::PKey::EC::Point.new(group, to_big_num(public_key_base64))
private_key = to_big_num(private_key_base64)

# ... then how do you generate the OpenSSL::PKey::EC from that (in OpenSSL 3)?

This is the only workaround that we have found:
https://github.com/pushpad/web-push/pull/2/files#diff-e8e36e8a5282b2fbf5503e699e222f3d5413c86cdbb94ecdd13148ea0c59e5f5R90

Which is quite complex compared to the straightforward assignment that you could do with OpenSSL 1.1.

Is there a simpler alternative to create an OpenSSL::PKey::EC from existing keys (in base64) in OpenSSL 3?

[collimarco]

OpenSSL::PKey::, Setters for parameters/key components
{RSA,DSA,DH}#set_ and EC#{private_key=,public_key=,group=}
Feature removed without replacement. Keys are now immutable once created - all components must be specified at once.

Would it make sense to replace these methods with a new method like set_keys or new_from_keys that assigns both keys at the same time (since they are immutable)? Would you consider adding it to Ruby? Or what is the new recommended way for assigning the existing keys?

[rhenium]

Would it make sense to replace these methods with a new method like set_keys or new_from_keys that assigns both keys at the same time (since they are immutable)?

#555 is working on it, which provides access to EVP_PKEY_fromdata() added in OpenSSL 3.0. It currently (as of openssl gem v3.1) has to be done by making the ASN.1 encoding as in this PR:

This is the only workaround that we have found:
https://github.com/pushpad/web-push/pull/2/files#diff-e8e36e8a5282b2fbf5503e699e222f3d5413c86cdbb94ecdd13148ea0c59e5f5R90

[junaruga]

As this is related to supporting OpenSSL 3, I just put the issue tickets and PR to support FIPS mode on OpenSSL 3

• OpenSSL 3: OpenSSL.fips_mode returns false in FIPS enabled environment #605, PR
• Support OpenSSL 3 FIPS #603

[mlarraz]

FYI OpenSSL 1.1.1 (the last version before 3.0) is set to be EOL in less than 2 months.

I first want to say I appreciate all the work that has gone into adding 3.0 support so far (as well as all the other work on this gem).

From my limited perspective, it looks like full support is unlikely to be completed before the EOL deadline. Is this a reasonable assumption? If so, it seems like something many users will want to know ahead of time.

[hlein]

Although the occasional OpenSSL-3.x issue is still spotted and fixed, according to https://github.com/ruby/openssl/blob/master/History.md OpenSSL 3.0 has been supported since v3.0.0 in Dec 24, 2021.

Meanwhile projects that consume this library are looking at this issue still being open as a reason they can't support OpenSSL 3.x yet. Can this issue be marked closed?

[thomthom]

We also got confused by this open issue and the incomplete 3.0 milestone: https://github.com/ruby/openssl/milestone/2
We didn't move to OpenSSL 3 because we got the impression it wasn't ready for production yet.

[eregon]

Agreed, @rhenium are you OK to close this issue and the milestone?
And if there any remaining TODO for OpenSSL 3.0 then I would suggest to file a new issue with just that.