Sprint: CI/CD/Testing Infra/Automation

[flyingzumwalt]

Objectives

• Setting up Jenkins with configuration from SCM
• Be faster than Travis on go-ipfs commit to green status
  • Measure time from commit to green status
  • Make sure it's fast
• Go/JS-ipfs unit and integration tests
• Artifacts/Pipelines flow
• Replace Teamcity/TravisCI with Jenkins
• Simulate failures of the Testing Infra
• Runs tests on
  • Linux
  • OSX
  • Windows
• Trusted machine running at someones home for Continuous Delivery
• TAP viewer on Jenkins

Team for this Sprint

• @victorbjelkholm (PM)
• @lgierth
• @Kubuxu?

Dependencies

N/A

Relevant

ipfs/notes#211
ipfs/notes#202

[RichardLitt]

I believe that this includes making sure that all repos can have testing, do. Which means I can do some work to make sure that project-repos works well for tracking this.

[victorb]

Waffle board for tracking the work: https://waffle.io/ipfs/jenkins
Work happening in this repository: https://github.com/ipfs/jenkins

[victorb]

Sprint CI - Standup 2016-01-31

lgierth

done

• looked into metrics
• adding plugin to jenkins, pending review
• wired up with grafana & production prometheus via cjdns

working on

• not sure what to work on next
• write documentation for how to wrire up with grafana

blockers

• waiting for jenkins deploy to production

questions

Kubuxu

done

• try to figure out docker and jenkins
• identifying optimizations

working on

• getting go-ipfs to run tests in jenkins

blockers

• making jenkins run docker

questions

victorbjelkholm

done

• set up jenkins from git, everybody able to run dev server

working on

• today started on production deploy
• some issues with permissoins and layers, using volumes instead
• figuring out deploy flow

blockers

questions

Notes

• chat about docker-in-docker or jenkins native
  • setup jenkins as dind + --privileged and always checkout Jenkinsfile from master branch
• production deploy flow
  • Victors whiteboard flow got thumbs up from everyone, moving on to implementation

[victorb]

Sprint CI - Standup 2017-02-01

lgierth

done

• not a terribly lot
• variable/templating tool for provsn

working on

• inspecting how jenkins runs

blockers

• monitoring blocked by production jenkins

questions

Kubuxu

done

• prepping for exam

working on

• starting on dockerfile for go-ipfs

blockers

• stuck on how to deal with gx on dist.ipfs.io for our container, solution: statically linked musl build of gx + go-ipfs
  • check with jeromy to verify tradeoffs and stuff

questions

victorbjelkholm

done

• invited @Kubuxu and @lgierth to quay.io (for image uploads)

working on

• production deployment of jenkins, and docs

blockers

• code review on PRs
• need advice for https

questions

Notes

[victorb]

Sprint CI - Standup 2017-02-02

lgierth

done

• wired the metrics to production prometheus
• nginx wired, domain ci.ipfs.team

working on

• getting SSL for Jenkins

blockers

questions

Kubuxu

done

• docker container for go-ipfs running
• simple pipeline for these

working on

• separate stages for parts of the build and tests
• parallelization

blockers

questions

victorbjelkholm

done

• we merged a lot of stuff
• improved some production docs
• jenkins now binds to localhost to make space for nginx
• always restart container

working on

• secrets, e.g. api keys for github and codecov
• login with github

blockers

questions

Notes

[victorb]

Sprint CI - Standup 2017-02-03

lgierth

done

• not a terribly lot

working on

• ssl for ci.ipfs.team
• failure modes
• look into dashboard weirdness

blockers

• none

questions

Kubuxu

done

working on

• continue with go-ipfs tests

blockers

• might have issues with jenkins

questions

victorbjelkholm

done

• js-ipfs running nodejs and browser tests in ci.ipfs.team

working on

• finishing js-ipfs tests
• starting working on js-ipfs-api tests
• after gonna move on to the integration tests

blockers

• start-prod.sh is a bit wonky if you changed start-prod.sh

questions

Notes

[victorb]

Sprint CI - Standup 2017-02-06

lgierth

done

• got ssl certificates!

working on

• gonna upload to jenkins host
• help out with orbit.chat

blockers

• none

questions

Kubuxu

done

• go-ipfs tests are running successful!
• added docker plugin

working on

• artifacts/coverage
• tests crashing for some reasons sometimes

blockers

• study for exams

questions

• docker plugin

victorbjelkholm

done

• finish js-ipfs test
• started working on js-ipfs-api

working on

• refactoring docker and jenkinsfiles
• integration

blockers

questions

Notes

[ghost]

Sprint CI - Standup 2017-02-07

lgierth

done

• SSL is DONE

working on

• HTTP to HTTPs redirect
• Windows hosting
• Simple Job (build and go test for now) running on Windows

blockers

questions

Kubuxu

done

• sharness test running - and it runs fast
• simple artefacts

working on

• coverage reporitng

blockers

• exams

questions

victorbjelkholm

12:19+0000 <@victorbjelkholm> Kubuxu: lgierth I'm not gonna be able to make it for the standup today, so here is my update: Yesterday done: Not very much, refinements with js-ipfs/js-ipfs-api and starting looking and integration test. Today: Finalizing PR and getting them merged. Hopefully get integration tests running. Blockers: None

Notes

[ghost]

Sprint CI - Standup 2017-02-08

Kubuxu

done

• almost nothing, studying for exam

working on

• looking more into coverage, hopefully today

blockers

questions

lgierth

done

• wrap up of ssl
• went through email backlog
• looking into ovh for windows boxes

working on

• the linux workers
• @Kubuxu suggests EG-32 with SSD 480G

blockers

questions

victorbjelkholm

done

• not very productive yesterday

working on

• multibranch, working on CI statuses
• right now using an ipfsbot token, should create dedicated user
• moving pipeline config away into a separate repo, so that built repos only specify which pipeline to use, but not how that pipeline works (security)

blockers

questions

Notes

• go-ipfs weirdness with load

[victorb]

Sprint CI - Standup 2017-02-09

Kubuxu

done

• exams stuff on a different hour

working on

• coverage, leaving for now
• multibranch for go-ipfs

blockers

• credentials for branch indexing a bit in the way

questions

lgierth

done

• ordering boxes from ovh is annoying

working on

• getting workers up and running, 2 boxes, one windows and one linux, each running 4 workers each

blockers

• waiting for paypal credential...

questions

victorbjelkholm

done

• finished js-ipfs/js-ipfs-api multibranch job
• had oom issues with chrome, added swap space on the host currently running that job (master)

working on

• getting js-ipfs/js-ipfs-api PRs merged
• will help @Kubuxu with the github credentials
• will help @lgierth setting up new linux worker host

blockers

• going to ikea later today

questions

Notes

[victorb]

Sprint CI - Standup 2017-02-10

Kubuxu

done

working on

blockers

questions

lgierth

done

• ssl for orbit
• patch for go-ipfs
• browser sprint preparation

working on

• waiting for hetzner for instances
• setting up windows worker

blockers

questions

victorbjelkholm

done

working on

• a few PRs in js-land merged
• talking to david about how to ci/ci js-* with jenkins
• writing three-stage proposal for jenkins usage
• linux worker box

blockers

questions

Notes

[victorb]

Report from CI Sprint

Now we've reached the end of the sprint of trying to improve our infrastructure around testing, specifically spinning up a Jenkins instance for all our testing needs instead of CircleCI and TravisCI. We managed to finish about half of the sprint, meaning we got a deployed instance setup with units tests for go-ipfs/js-ipfs/js-ipfs-api. Missing tasks include tuning the monitoring, settings up integrations tests, moving projects over to use Jenkins in master and other tasks dedicated to replace the functionality CircleCI and TravisCI. You can see a list of unfinished tasks here: https://github.com/ipfs/jenkins/milestone/1

What we accomplished

• Setting up instance of Jenkins https://ci.ipfs.team/
• go-ipfs unit tests: https://ci.ipfs.team/job/go-ipfs/
• js-ipfs unit tests: https://ci.ipfs.team/job/js-ipfs/
• js-ipfs-api unit tests: https://ci.ipfs.team/job/js-ipfs-api/
• Github authentication for jobs
• Monitoring dashboard for Jenkins: http://metrics.ipfs.team/dashboard/db/jenkins?from=now-1h&to=now

Retrospective

This friday we also had a retrospective talking about what could have been better in this sprint. We identified two core issues with how the sprint was laid out. Splitting up tasks is something we could have done a lot better and was amplified because my preparation time as a PM was lacking due to participating in another sprint. With more planning time in beforehand, I could have spent more time correctly identifying the tasks and splitting them up. A bit of prioritisation was also missing, meaning we spent some time on tasks that we should have pushed down and worked on other things.

Next Steps

Steps for moving forward have been posted in a issue over at ipfs/jenkins: ipfs-inactive/jenkins#39

If you have any feedback regarding moving forward with the CI work, please let us know so we can make the process of testing our projects as good as possible.

As a side-note, I will continue working with the remaining tasks for this week and thereafter jump into the other sprint.