Manually building and installing docker-api gem

[tyler-ball]

Description

The ruby cleanup omnibus software dep removes the gem installed by
bundler. I am not sure how Ohai does not have this issue. Reverting
to a pattern I understand to fix our build for now.

In the future it would be nice to only have to manage this gem via
the Gemfile and without the docker-api omnibus software definition.

Related Issue

https://buildkite.com/chef/chef-chef-workstation-master-omnibus-release/builds/356#eb929cfb-d3cb-4862-9990-75c1a99c09d0

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Chore (non-breaking change that does not add functionality or fix an issue)

Checklist:

• I have read the CONTRIBUTING document.
• I have run the pre-merge tests locally and they pass.
• I have updated the documentation accordingly.
• I have added tests to cover my changes.
• All new and existing tests passed.
• All commits have been signed-off for the Developer Certificate of Origin.

Aha! Link: https://chef.aha.io/features/SH-1579

[tyler-ball]

@tas50 I think I figured out how Ohai doesn't get removed by ruby-cleanup even though it is installed via the Gemfile from a branch: https://github.com/chef/omnibus-software/blob/6124b6d64b3540e9ded3e4e532d24fbc33d686e9/config/software/chef.rb#L51

It is doing the same thing I'm doing here

[lamont-granquist]

The client uses the ohai software definition to install ohai from git.

If you're installing from git it gets installed into the bundler gitsha'd directory and not into the proper location where normal rubygems can see it. Bundler sees it is from a git source and therefore untrusted so refuses to pollute your prestine set of gems from rubygems with that prerelease gem. That later gets cleaned up by the omnibus ruby cleanup scripts, but it never would have worked anyway.

So this is the right thing to do for git sources.

IMO this is screaming out that we need to either:

1. fork it to chef-docker-api gem, fully take it over, and release it to rubygems.

2. stop using the ruby API entirely and shell out to the docker command line.

I am mostly in favor of #2 (up until it is shown that the docker command line API is as big of a tirefire as dnf/yum or ruby is anyway). Until its shown that the CLI there is painfully slow, the benefits of a ruby library solution over shelling out to the command line are mostly philosophical rather than practical. The practical matter is that it ties you to that API gem being updated properly which is the net-ssh problem all over again.

[lamont-granquist]

We also have to do this, but as long as you're not pushing anything to the docker-api gem the version in the Gemfile.lock that appbundler reads and the master version here shouldn't differ so you shouldn't need it. But if you're actively developing in the docker-api gem you'll need this as well:

https://github.com/chef/chef/blob/master/omnibus_overrides.rb#L27-L33

[lamont-granquist]

Again, this is due to bundler and that stupid directory it installs git gems into. The gem build; gem install workaround is correct since that sends the gem into the right lib location. The problem is that may also pull all kinds of duplicated gems into the shipping artifact depending on the pins in the Gemfile.lock -- if you have any dependent gems in docker-api which are held back by other pins (e.g. inspec) that is how you get duplicated gems.

And I think that will cause Tim at least to favor one of the other two approaches.

[tyler-ball]

I ended up having to remove the git sourcing from the Gemfile because that caused appbundling to fail with the error

---- Begin output of bundle lock ----
--
  | STDOUT: Fetching gem metadata from http://k8s-artifactory-production.es.chef.co/artifactory/api/gems/rubygems-proxy/........
  | Fetching gem metadata from http://k8s-artifactory-production.es.chef.co/artifactory/api/gems/rubygems-proxy/..
  | Could not find gem 'docker-api' in source at
  | `/opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/docker-api-1.34.2`.
  | The source does not contain any versions of 'docker-api'

I don't know how that was failing because I added a debug line to verify that the gem was being correctly installed from the omnibus software definition:

[Builder: docker-api] I | 2020-04-23T22:56:40+00:00 | $ /opt/chef-workstation/embedded/bin/gem install docker-api-*.gem --no-document --ignore-dependencies
                      D | 2020-04-23T22:56:41+00:00 | Successfully installed docker-api-1.34.2
                      D | 2020-04-23T22:56:41+00:00 | 1 gem installed
[Builder: docker-api] I | 2020-04-23T22:56:41+00:00 | gem `install docker-api-*.gem --no-document --ignore-dependencies': 0.2902s
[Builder: docker-api] I | 2020-04-23T22:56:41+00:00 | $ ls -al /opt/chef-workstation/embedded/lib/ruby/gems/2.6.0/gems/
                      D | 2020-04-23T22:56:41+00:00 | total 56
                      D | 2020-04-23T22:56:41+00:00 | drwxr-xr-x 14 buildkite-agent buildkite-agent 4096 Apr 23 22:56 .
                      D | 2020-04-23T22:56:41+00:00 | drwxr-xr-x  8 buildkite-agent buildkite-agent 4096 Apr 23 22:53 ..
                      D | 2020-04-23T22:56:41+00:00 | drwxr-xr-x  5 buildkite-agent buildkite-agent 4096 Apr 23 22:53 bundler-1.17.2
                      D | 2020-04-23T22:56:41+00:00 | drwxr-xr-x  7 buildkite-agent buildkite-agent 4096 Apr 23 22:53 did_you_mean-1.3.0
                      D | 2020-04-23T22:56:41+00:00 | drwxr-xr-x  3 buildkite-agent buildkite-agent 4096 Apr 23 22:56 docker-api-1.34.2

[tyler-ball]

@lamont-granquist I know that doing a gem build; gem install from an omnibus definition can end up installing duplicate gems, so I added --ignore-dependencies here. Its not a good long-term solution, but I'm out of time to troubleshoot this. We currently only have 1 difference from master in our fork but if we start making other changes to that gem we will need to re-address how we do this.