❗️omnibus build is broken

[robbkidd]

Describe the problem

Builds of the omnibus package fail with:

ArgumentError "\x80\x00\x00\x00d\x9E\xD3\xD9" is not an octal string

Software Version

The master branch, starting with tag v3.3.6. There is no v3.3.6 version currently because the package cannot be built.

Replication Case

See https://buildkite.com/chef/chef-supermarket-master-omnibus-adhoc/builds/9#job-20de4140-a25d-4193-a7c5-d379eccea99f for an example failed run.

Stacktrace/Details

[Builder: supermarket-cookbooks] I | 2019-06-22T14:55:18+00:00 | $ berks vendor /opt/supermarket/embedded/cookbooks
    D | 2019-06-22T14:55:20+00:00 | Resolving cookbook dependencies...
    D | 2019-06-22T14:55:20+00:00 | Fetching 'omnibus-supermarket' from source at .
    D | 2019-06-22T14:55:20+00:00 | Fetching cookbook index from https://supermarket.chef.io...
    D | 2019-06-22T14:55:23+00:00 | Using build-essential (8.2.1)
    D | 2019-06-22T14:55:23+00:00 | Using enterprise (0.15.1)
    D | 2019-06-22T14:55:23+00:00 | Using fipsify (0.1.0)
    D | 2019-06-22T14:55:23+00:00 | Using mingw (2.1.0)
    D | 2019-06-22T14:55:23+00:00 | Using omnibus-supermarket (1.1.1) from source at .
    D | 2019-06-22T14:55:23+00:00 | Installing line (2.3.2)
    D | 2019-06-22T14:55:23+00:00 | Using packagecloud (1.0.1)
    D | 2019-06-22T14:55:23+00:00 | Using runit (5.1.1)
    D | 2019-06-22T14:55:23+00:00 | Using seven_zip (3.1.1)
    D | 2019-06-22T14:55:23+00:00 | Using unicorn (2.1.2)
    D | 2019-06-22T14:55:23+00:00 | Using windows (6.0.0)
    D | 2019-06-22T14:55:23+00:00 | Using yum-epel (3.3.0)
    D | 2019-06-22T14:55:25+00:00 | ArgumentError "\x80\x00\x00\x00d\x9E\xD3\xD9" is not an octal string

This occurs during the install of supermarket-cookbooks. Supermarket's omnibus-internal cookbook still uses berkshelf and seems to be pulling in a version that uses RubyGem's TarReader to expand cookbooks (see #1806 and #1810 for ugly details about why this happens). The chain of what's going on seems to be as follows:

• berks tries to vendor all the cookbooks the internal cookbook depends on
• the internal cookbook depends on the unicorn cookbook (so far as I can tell, only for its unicorn.rb.erb template; no resources are used)
• the unicorn cookbook depends on the windows cookbook
• the windows cookbook has extended headers for large uids/guids present

Possible Solution

Some ideas:

1. upgrade berks to a version that doesn't use RubyGems TarReader?
2. make the internal cookbook policyfile-driven to update the dependency resolver/downloader to something that uses mixlib-archive instead of RubyGems TarReader
3. confirm the internal cookbook only uses the unicorn cookbook for the unicorn.rb.erb template and, if so, copy it into the internal cookbook, manage it ourselves, and drop the dependency on the unicorn cookbook

[robbkidd]

Updated the internal cookbook to not depend on the unicorn-and-therefore-windows cookbook and still get the error. Unsure of which remaining cookbook has the extended headers. Seems the extract-with-RubyGems-TarReader is down in berkshelf's CommunityRest class.

...
Using yum-epel (3.3.0)
ArgumentError "\x80\x00\x00\x00d\x9E\xD3\xD9" is not an octal string
  /opt/omnibus-toolchain/embedded/lib/ruby/site_ruby/2.6.0/rubygems/package/tar_header.rb:130:in `strict_oct'
  /opt/omnibus-toolchain/embedded/lib/ruby/site_ruby/2.6.0/rubygems/package/tar_header.rb:110:in `from'
  /opt/omnibus-toolchain/embedded/lib/ruby/site_ruby/2.6.0/rubygems/package/tar_reader.rb:59:in `each'
  /opt/omnibus-toolchain/embedded/lib/ruby/gems/2.6.0/gems/mixlib-archive-1.0.1/lib/mixlib/archive/tar.rb:28:in `block in extract'
  /opt/omnibus-toolchain/embedded/lib/ruby/site_ruby/2.6.0/rubygems/package/tar_reader.rb:29:in `new'
  /opt/omnibus-toolchain/embedded/lib/ruby/gems/2.6.0/gems/mixlib-archive-1.0.1/lib/mixlib/archive/tar.rb:146:in `reader'
  /opt/omnibus-toolchain/embedded/lib/ruby/gems/2.6.0/gems/mixlib-archive-1.0.1/lib/mixlib/archive/tar.rb:26:in `extract'
  /opt/omnibus-toolchain/embedded/lib/ruby/gems/2.6.0/gems/mixlib-archive-1.0.1/lib/mixlib/archive.rb:47:in `extract'
  /opt/omnibus-toolchain/embedded/lib/ruby/gems/2.6.0/gems/berkshelf-7.0.8/lib/berkshelf/community_rest.rb:15:in `unpack'
...

[arhue]

I get the same error. Needs a fix quickly.

[robbkidd]

@arhue You get the same error in what context? Are you building your own omnibus package of Supermarket?

[robbkidd]

Hoping for a release of RubyGems 3.0.6 which should include a fix for large uid/gids in tarfiles, but that release is currently held up (at least partially) by another unrelated PR.

More context:

The error is coming from berkshelf's use of RubyGem's TarReader to unpack cookbook tarballs. RubyGem's TarReader could not handle large values in uid or gid header entries. In other places where this error appeared, we moved to using Chef-maintained ffi-libarchive in the places in the code that dive into tarballs. So far, we've decided against using ffi-libarchive in berkshelf because libarchive would introduce a C dependency which would make using berkshelf as a gem more complicated.