Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build process and CI improvements #409

Merged
merged 42 commits into from
May 29, 2018
Merged

Build process and CI improvements #409

merged 42 commits into from
May 29, 2018

Commits on May 12, 2018

  1. Fix openssl key generation under OpenSSL 1.1+

    openssl 1.1.0+ can't have some empty fields specified, so instead don't
    specify them at all. See #408
    GUI committed May 12, 2018
    Configuration menu
    Copy the full SHA
    1e5ca7d View commit details
    Browse the repository at this point in the history

Commits on May 13, 2018

  1. Update Traffic Server v7. Ubuntu 18.04, Debian 9 builds. Rework builds.

    - Add Ubuntu 18.04 (Bionic) and Debian 9 (Stretch) to our package
      building process.
    - Remove Ubuntu 12.04 (Precise) and Debian 7 (Wheezy) from our package
      builds. Ubuntu 12.04 has reached its end of support and Debian 7 will
      at the end of this month.
    - Upgrade from Traffic Server v5 to v7. This stemmed from needing to
      build under Ubuntu 18.04 where OpenSSL v1.1 is the default, and we
      need at least Traffic Server v6 to build against OpenSSL 1.1.0. There
      might still be some keepalive issues to sort out (which is why we were
      holding back), but since Traffic Server v5 isn't supported any longer,
      this update is due.
    - Rework the build process. While this is a bit messy to get wrapped in
      with these other updates, this pulls over various improvements to the
      cmake build process that were in the lapis-postgres branch. I keep
      hitting issues that would benefit from these build improvements, and
      maintaining things on both branches is becoming more of a hassle, so
      we'll manually pull over the bulk of these build process improvements.
      The improvements mainly revolve around better splitting up the build
      process into discrete targets. This better organizes the build files,
      but also provides easier ways to perform the build in specific steps,
      which helps with the Docker build process and better caching.
    - Make various improvement to the Docker development environment so
      performing builds and re-builds are easier/more reliable. Now the
      symlinking happens in an entrypoint script, so that it still takes
      place on "run" commands. Also improve the build "configure" script so
      it always performs builds in the /build directory. All of these
      changes better ensure builds happens as they did on initial build even
      if you're manually running these commands on an existing docker dev
      container.
    - Some other misc improvements pulled over from the lapis-postgres
      branch. Again not quite ideal to be manually pulling these over, but
      this should help get some of these more general improvements live
      before lapis-postgres is merged in.
        - Switch from luatz to icu-date for more proper timezone handling.
        - Bring over some improvements to startup logic and default yaml
          config logic.
    GUI committed May 13, 2018
    Configuration menu
    Copy the full SHA
    4ac3d48 View commit details
    Browse the repository at this point in the history
  2. Switch from ngx_txid to lua-resty-txid for generating request IDs.

    The IDs should be functionally equivalent, this is just a pure-Lua
    implementation to reduce our need for custom nginx modules.
    GUI committed May 13, 2018
    Configuration menu
    Copy the full SHA
    9d2ebd4 View commit details
    Browse the repository at this point in the history
  3. More build fixes and tweaks.

    - Ensure test dependencies are installed in CI environment.
    - Use system libxml and libxslt libraries for nokogiri (to make security
      updates easier and speed up installs).
    - Optimize bundle installations so that a common bundle directory is
      used across bundles for better caching.
    - Fix unnecessary make re-builds caused by using the dynamic PATH
      environment variable. Instead use a hard-coded base PATH.
    GUI committed May 13, 2018
    Configuration menu
    Copy the full SHA
    ff277c6 View commit details
    Browse the repository at this point in the history
  4. Remove experimental Kylin analytics code. We may revisit this.

    We may revisit this, but since it's probably not on the immediate
    horizon, go ahead and remove this code, so we're not trying to maintain
    it or its dependencies.
    GUI committed May 13, 2018
    Configuration menu
    Copy the full SHA
    77d50d0 View commit details
    Browse the repository at this point in the history
  5. Switch to CircleCI 2.0 setup.

    This mostly ports over the setup used on the lapis-postgres branch where
    we had started this migration.
    GUI committed May 13, 2018
    Configuration menu
    Copy the full SHA
    9b2a812 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    dff55d1 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    18126a6 View commit details
    Browse the repository at this point in the history
  8. Fix eslint in CI environment.

    GUI committed May 13, 2018
    Configuration menu
    Copy the full SHA
    f4d0060 View commit details
    Browse the repository at this point in the history
  9. More CI tweaks.

    GUI committed May 13, 2018
    Configuration menu
    Copy the full SHA
    43128e9 View commit details
    Browse the repository at this point in the history

Commits on May 14, 2018

  1. Fix shell linting.

    GUI committed May 14, 2018
    Configuration menu
    Copy the full SHA
    850e70d View commit details
    Browse the repository at this point in the history
  2. Fix CI tests to pick up all files.

    globstar isn't apparently set by defualt in CircleCI, so `**` wasn't
    picking up files in nested directories.
    GUI committed May 14, 2018
    Configuration menu
    Copy the full SHA
    9c773e4 View commit details
    Browse the repository at this point in the history

Commits on May 17, 2018

  1. Remove defunct clean make task.

    We don't actually have any archives in this directory anymore, so this
    clean task doesn't actually do anything.
    GUI committed May 17, 2018
    Configuration menu
    Copy the full SHA
    1ee8c8b View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    bbc138d View commit details
    Browse the repository at this point in the history

Commits on May 18, 2018

  1. Rollback to TrafficServer 5 in this branch. We'll upgrade separately.

    We'll do the upgrades to v7 in a separate branch.
    GUI committed May 18, 2018
    Configuration menu
    Copy the full SHA
    35e5a5e View commit details
    Browse the repository at this point in the history
  2. Fix leak test due to switch from ngx_txid to lua-resty-txid.

    There are no longer any urandom descriptors opened with lua-resty-txid,
    so we no longer need to check to ensure it's greater than 0. But still
    keep the basic test intact to ensure we don't accidentally leak things
    in the future.
    GUI committed May 18, 2018
    Configuration menu
    Copy the full SHA
    c74bbb4 View commit details
    Browse the repository at this point in the history
  3. Fix elasticsearch index naming, and pull in elasticsearch improvements.

    This fixes the elasticsearch index names missing the "v" part of the
    "v1" version number in the name after the various version config changes
    in this branch.
    
    This also pulls in some of the elasticsearch refactoring to add the
    helper library for handling connections and queries (originally on the
    lapis-postgres branch).
    GUI committed May 18, 2018
    Configuration menu
    Copy the full SHA
    cd0c913 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    e36151e View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    d564e68 View commit details
    Browse the repository at this point in the history

Commits on May 19, 2018

  1. Go back to docker environment for CI tests, and disable IPv6 test.

    The machine executor doesn't have caching, which means testing would be
    very slow given our current reliance on the cache for caching our build
    files.
    
    So instead, go back to the docker test environment, which lacks IPv6
    support, so disable our one IPv6 test in the CI environment (not ideal,
    but for now seems like the simplest option).
    GUI committed May 19, 2018
    Configuration menu
    Copy the full SHA
    41aa90f View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    83e588a View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    f25b84f View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    a224137 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    07bc323 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    007b280 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    e97d03c View commit details
    Browse the repository at this point in the history
  8. Improve CI caching setup.

    GUI committed May 19, 2018
    Configuration menu
    Copy the full SHA
    247d4bf View commit details
    Browse the repository at this point in the history

Commits on May 20, 2018

  1. Configuration menu
    Copy the full SHA
    03869d3 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    d644097 View commit details
    Browse the repository at this point in the history

Commits on May 22, 2018

  1. Configuration menu
    Copy the full SHA
    0af3d55 View commit details
    Browse the repository at this point in the history
  2. Update dependencies.

    Also disable and remove liblogging dependency for rsyslog, since it
    looks like it's becoming deprecated in a future version, and since we
    don't actually use it, we might as well get rid of an extra dependency.
    GUI committed May 22, 2018
    Configuration menu
    Copy the full SHA
    6d20aee View commit details
    Browse the repository at this point in the history
  3. Flip the CI cache naming around to prevent conflicts/mismatches.

    Since CircleCI's matching is based on prefixes, with the previous
    approach it mean the cache for the `updates-trafficserver` branch could
    inadvertently get used when building the `updates` branch. So reorder
    the naming convention around so conflicts like this aren't as likely
    (unless a branch name happened to contain the system arch name).
    GUI committed May 22, 2018
    Configuration menu
    Copy the full SHA
    025101f View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    e076732 View commit details
    Browse the repository at this point in the history

Commits on May 23, 2018

  1. Configuration menu
    Copy the full SHA
    4873104 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    acce469 View commit details
    Browse the repository at this point in the history

Commits on May 24, 2018

  1. Configuration menu
    Copy the full SHA
    aeaaf98 View commit details
    Browse the repository at this point in the history

Commits on May 27, 2018

  1. Configuration menu
    Copy the full SHA
    1108e96 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    9afdbde View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    6cab6dd View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    1f02a0f View commit details
    Browse the repository at this point in the history

Commits on May 28, 2018

  1. Configuration menu
    Copy the full SHA
    c6f960f View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    ed0e2c7 View commit details
    Browse the repository at this point in the history