OpenSSL 3 FIPS mode - creating encrypted RSA key pair fails with PEM_write_bio_PrivateKey_traditional: initialization error (OpenSSL::PKey::PKeyError)

[tarnowsc]

I'm trying to execute the first example from docs: https://docs.ruby-lang.org/en/master/OpenSSL.html#module-OpenSSL-label-Examples

And I'm not able to execute the following code:

require 'openssl'

key = OpenSSL::PKey::RSA.new 2048
cipher = OpenSSL::Cipher.new 'aes-256-cbc'
pass_phrase = 'my secure pass phrase goes here'

key_secure = key.export cipher, pass_phrase

The error I'm getting is:

irb(main):013:0> key_secure = key.export cipher, pass_phrase
(irb):13:in `export': PEM_write_bio_PrivateKey_traditional: initialization error (OpenSSL::PKey::PKeyError)
        from (irb):13:in `<main>'                    
        from /var/lib/ruby/lib/ruby/gems/3.2.0/gems/irb-1.6.2/exe/irb:11:in `<top (required)>'
        from /var/lib/ruby/bin/irb:25:in `load'      
        from /var/lib/ruby/bin/irb:25:in `<main>'

The same code works fine without the FIPS mode.

I'm using Ubuntu 22.04LTS with the following versions:

root@b6b9c4cc9cb7:/opt/conjur-server# ruby --version      
ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]
root@b6b9c4cc9cb7:/opt/conjur-server# openssl version     
OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)

...

irb(main):014:0> OpenSSL::VERSION
=> "3.1.0"

[junaruga]

I was reproduce this issue on the current master branch < 97fb410> with OpenSSL 3.0.9 FIPS built from the source.

The reproducing steps:

Install the OpenSSL seeing the Documents - How to debug Ruby OpenSSL binding.

Check the enabled FIPS and fips and base providers by this testing program.

$ LD_LIBRARY_PATH=${HOME}/.local/openssl-3.0.9-fips-debug/lib/ \
  OPENSSL_CONF=/home/jaruga/.local/openssl-3.0.9-fips-debug/ssl/openssl_fips.cnf \
  ./fips_mode
Loaded providers:
  fips
  base
FIPS mode enabled: 1

$ ruby -v
ruby 3.3.0dev (2023-05-30T12:39:26Z master 30b960ba34) [x86_64-linux]

bundle install --standalone

Compile

$ MAKEFLAGS="V=1" \
  RUBY_OPENSSL_EXTCFLAGS="-O0 -g3 -ggdb3 -gdwarf-5" \
  bundle exec rake compile -- \
  --enable-debug \
  --with-openssl-dir=$HOME/.local/openssl-3.0.9-fips-debug

Check the compiled Ruby binding is linked to the OpenSSL 3.0.9, and FIPS is enabled.

$ ldd lib/openssl.so 
	linux-vdso.so.1 (0x00007ffc551b0000)
	libruby.so.3.3 => /home/jaruga/.local/ruby-30b960ba34-debug/lib/libruby.so.3.3 (0x00007f59b7800000)
	libssl.so.3 => /home/jaruga/.local/openssl-3.0.9-fips-debug/lib/libssl.so.3 (0x00007f59b7eae000)
	libcrypto.so.3 => /home/jaruga/.local/openssl-3.0.9-fips-debug/lib/libcrypto.so.3 (0x00007f59b7200000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f59b7db5000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f59b7022000)
	libz.so.1 => /lib64/libz.so.1 (0x00007f59b77e6000)
	libgmp.so.10 => /lib64/libgmp.so.10 (0x00007f59b7741000)
	libcrypt.so.2 => /lib64/libcrypt.so.2 (0x00007f59b7708000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f59b6ffe000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f59b7ff7000)

$ bundle exec rake debug
/home/jaruga/.local/ruby-30b960ba34-debug/bin/ruby -I./lib -ropenssl -ve'puts OpenSSL::OPENSSL_VERSION, OpenSSL::OPENSSL_LIBRARY_VERSION'
ruby 3.3.0dev (2023-05-30T12:39:26Z master 30b960ba34) [x86_64-linux]
OpenSSL 3.0.9 30 May 2023
OpenSSL 3.0.9 30 May 2023

$ OPENSSL_CONF=${HOME}/.local/openssl-3.0.9-fips-debug/ssl/openssl_fips.cnf \
  bundle exec ruby -I ./lib -e 'require "openssl"; p OpenSSL.fips_mode'
true

Here is your script.

$ cat test.rb 
require 'openssl'

key = OpenSSL::PKey::RSA.new 2048
cipher = OpenSSL::Cipher.new 'aes-256-cbc'
pass_phrase = 'my secure pass phrase goes here'

key_secure = key.export cipher, pass_phrase

Your reproducing script fails on FIPS as reported.

$ OPENSSL_CONF=${HOME}/.local/openssl-3.0.9-fips-debug/ssl/openssl_fips.cnf \
  bundle exec ruby -I ./lib test.rb
test.rb:7:in `export': PEM_write_bio_PrivateKey_traditional: initialization error (OpenSSL::PKey::PKeyError)
  from test.rb:7:in `<main>'

Your script succeeds as reported.

$ bundle exec ruby -I ./lib test.rb

$ echo $?
0

[rhenium]

I think this might be unfixable. OpenSSL's traditional PEM encryption format uses MD5 to derive encryption keys from the password, but MD5 is prohibited under the FIPS mode, hence it fails.

We should document this. There is also room for improvement in the exception message. Running the code with OpenSSL.debug = true shows more information:

t643.rb:9: warning: error on stack: error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported (Global default library context, Algorithm (MD5 : 100), Properties ())
t643.rb:9: warning: error on stack: error:03000086:digital envelope routines:evp_md_init_internal:initialization error (Global default library context, Algorithm (MD5 : 100), Properties ())

[junaruga]

@rhenium Thanks for the info. In that case, I think we can improve the logic to print the error message to tell users the reason with a better error message, and I think that can be a solution to close this issue ticket.

As a note OpenSSL 3.0 FIPS supports FIPS 140-2, and OpenSSL 3.1 FIPS supports FIPS 140-3. I learned it recently. :)

We should document this. There is also room for improvement in the exception message. Running the code with OpenSSL.debug = true shows more information:

Those are nice tips. For another improvement, perhaps, we might be able to create an issue template for this repository, adding the info as a reporting way.

[junaruga]

I plan to list up all the supported encryptions in Ruby OpenSSL binding, then to compare the specification with OpenSSL 3.0 FIPS (FIPS 140-2) and OpenSSL 3.1 FIPS (FIPS 140-3) manually to decide how to implement.

[rhenium]

#645 updates the RDoc for exporting PKeys.

[junaruga]

@tarnowsc I suppose that the following code with OpenSSL::PKey::RSA#private_to_pem is a right way to achieve what you want to do.

key_secure = key.private_to_pem cipher, pass_phrase

instead of the following code.

key_secure = key.export cipher, pass_phrase

The #645 is to explain the details.

[tarnowsc]

yes, thanks for your support, the output of those two commands is not exactly identical, but it should do the trick:

irb(main):007:0> key_secure = key.export cipher, pass_phrase
=> "-----BEGIN RSA PRIVATE KEY-----\nProc-Type: 4,ENCRYPTED\nDEK-Info: AES-256-CBC,B52746B87D40786BA0D8DC738CEF2072\n\ni/oWxCNgdtv7MutETCgnVSq6ZZSuYB8cnnJUa31Vx5DqGqegxLNcDxlEWka4ZOiO\nTnDYWE5RZJgSzJCdKCi+ZxneTklXxCfXIe4N1uiOgvNUXIRAgThBoUD...
irb(main):008:0> 
irb(main):009:0> key_secure = key.private_to_pem cipher, pass_phrase
=> "-----BEGIN ENCRYPTED PRIVATE KEY-----\nMIIFLTBXBgkqhkiG9w0BBQ0wSjApBgkqhkiG9w0BBQwwHAQIFuomdujf3bACAggA\nMAwGCCqGSIb3DQIJBQAwHQYJYIZIAWUDBAEqBBDQ92vVYiciFltnl5o33sxvBIIE\n0EX2XuxV7CuTSPN56kSSJ5XRkoOtsyNSIKY0/dzgqRwl+uKJn2aN52QIvjbh74Tw\...