-
Notifications
You must be signed in to change notification settings - Fork 24
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
This commit adds the structure of the libraries and a few funcions. Most of the code in this commit is geared towards making sure this repo stays tested and documented in future. The following things are checked: * All functions are documented * All functions are tested * All scripts are linted See Readme for a list of the functions that are included, and for more information on testing with BATS.
- Loading branch information
1 parent
a620e57
commit d8b85ec
Showing
35 changed files
with
1,025 additions
and
2 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
[submodule "test-utils/bats"] | ||
path = test-utils/bats | ||
url = https://github.com/bats-core/bats | ||
[submodule "test-utils/bats-assert-1"] | ||
path = test-utils/bats-assert-1 | ||
url = https://github.com/jasonkarns/bats-assert-1 | ||
[submodule "test-utils/bats-support"] | ||
path = test-utils/bats-support | ||
url = https://github.com/ztombol/bats-support |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,51 @@ | ||
#!/usr/bin/env groovy | ||
|
||
pipeline { | ||
agent { label 'executor-v2' } | ||
|
||
options { | ||
timestamps() | ||
buildDiscarder(logRotator(numToKeepStr: '30')) | ||
} | ||
|
||
triggers { | ||
cron(getDailyCronString()) | ||
} | ||
|
||
environment { | ||
BATS_OUTPUT_FORMAT="junit" | ||
} | ||
|
||
stages { | ||
stage('Bash Linting') { | ||
environment { | ||
BATS_SUITE="BATS-Lint" | ||
} | ||
steps { | ||
sh './tests-for-this-repo/run-bats-tests.sh tests-for-this-repo/lint.bats' | ||
} | ||
} | ||
|
||
stage('Python Linting') { | ||
steps { | ||
sh './tests-for-this-repo/python-lint.sh' | ||
} | ||
} | ||
|
||
stage('Bash Tests') { | ||
environment { | ||
BATS_SUITE="BATS-Tests" | ||
} | ||
steps { | ||
sh './tests-for-this-repo/run-bats-tests.sh $(ls tests-for-this-repo/*.bats|grep -v lint.bats)' | ||
} | ||
} | ||
} | ||
|
||
post { | ||
always { | ||
junit '*-junit.xml' | ||
cleanupAndNotify(currentBuild.currentResult) | ||
} | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,197 @@ | ||
# bashlib | ||
Common bash functions for use in test pipelines | ||
# bash-lib | ||
``` | ||
_______________ _______________ | ||
.' .' .| | ||
.' .' .' | | ||
.'_______________.'______________ .' | | ||
| ___ _____ ___ || ___ _____ ___ | | | ||
||_=_|__=__|_=_||||_=_|__=__|_=_|| | | ||
______||_____===_____||||_____===_____|| | __________ | ||
.' ||_____===_____||||_____===_____|| .' .'| | ||
.' ||_____===_____||||_____===_____|| .' .' | | ||
.'___________|_______________||_______________|.'__________.' | | ||
|.----------.|.-----___-----.||.-----___-----.|| |_____.----------. | ||
|] |||_____________||||_____________||| .' [ | | ||
|| ||.-----___-----.||.-----___-----.||.' | | | ||
|| |||_____________||||_____________|||==========| | | ||
|| ||.-----___-----.||.-----___-----.|| |_____| | | ||
|] o|||_____________||||_____________||| .' [ 'o| | ||
|| ||.-----___-----.||.-----___-----.||.' | | | ||
|| ||| ||||_____________|||==========| | | ||
|| ||| |||.-----___-----.|| |_____| | | ||
|] ||| |||| ||| .' [ | | ||
||__________|||_____________||||_____________|||.'________|__________| | ||
''----------'''------------------------------'''----------'' | ||
(o)LGB (o) | ||
``` | ||
|
||
The place to store functions that are used in pipelines for multiple repos. | ||
|
||
Please add whatever is useful to you, but keep it tidy so its still useful to everyone else :) | ||
|
||
## Usage | ||
Firstly acquire a clone of bash-lib: | ||
1. If submodules are acceptable in your workflow, I recommend adding a submodule `git submodule add git@github.com:conjurinc/bash-lib bash-lib` | ||
1. Otherise you can use emulate submodules with a bash function eg:<sup>[1](#footnote-1)</sup> | ||
```bash | ||
function init_bash_lib(){ | ||
# shellcheck disable=SC2086,SC2046 | ||
d="$(cd $(dirname ${BASH_SOURCE[0]}); pwd)" | ||
ref_path="${d}/.bash_lib_ref" | ||
pushd ${d} | ||
git clone --recurse-submodules git@github.com:conjurinc/bash-lib | ||
pushd bash-lib | ||
if [[ -e "${ref_path}" ]] | ||
then | ||
git checkout "$(cat ${ref_path})" | ||
else | ||
git rev-parse HEAD > "${ref_path}" | ||
echo "Please commit ${ref_path} to ensure a consistent version is used" | ||
fi | ||
. init.sh | ||
popd | ||
popd | ||
} | ||
``` | ||
|
||
It is highly important that however you acquire bash-lib, you always **use it at | ||
a pinned verison**. Hopefully bash-lib will end up widely used, and we do not | ||
want to break every project when updating bash-lib. Each project must update the pin | ||
at a time that suits them to avoid unexpected breakages. | ||
|
||
Once you have bash-lib cloned in your project, you source two things: | ||
|
||
1. Source `bash-lib/init.sh`. This ensures submodules are initalised and sets the BASH_LIB env var to the absolute path to the bash-lib dir. This makes it easy to source libraries from other scripts. | ||
1. Source `${BASH_LIB}/lib-name/lib.sh` for any libraries you are interested in. | ||
|
||
You are now ready to use bash-lib functions :) | ||
|
||
## Structure | ||
The `/init.sh` script sets up everything required to use the library, most | ||
importantly the `BASH_LIB` variable which gives the absolute path to the root | ||
of the library and should be used for sourcing the modules. | ||
|
||
The repo is organised into libraries, each library is a directory that has a | ||
lib.sh file. Sourcing the lib.sh for a library should expose all the functions | ||
that library offers. The lib.sh file may source or reference other supporting | ||
files within it's directory. | ||
|
||
``` | ||
/init.sh | ||
/lib-name/ | ||
lib.sh # main library file | ||
supporting-file.yaml # a supporting file | ||
/tests-for-this-repo # self tests for this repo. | ||
``` | ||
## Style Guide | ||
Follow the [google shell style guide](https://google.github.io/styleguide/shell.xml#Naming_Conventions). | ||
TL;DR: | ||
1. Use snake_case function and variable names | ||
1. Use `function` when declaring functions. | ||
|
||
|
||
## Contents | ||
|
||
<!-- html table due to markdown's lack of support for lists within tables --> | ||
<table> | ||
<thead> | ||
<tr> | ||
<th>Library</th> | ||
<th>Description</th> | ||
<th>Functions</th> | ||
</tr> | ||
</thead> | ||
<tbody> | ||
<tr> | ||
<td><a href="filehandling/lib.sh">filehandling</a></td> | ||
<td>Functions relating to file and path handling | ||
<td> | ||
<ol> | ||
<li> abs_path: Ensure a path is absolute</li> | ||
</ol> | ||
</td> | ||
</tr> | ||
<tr> | ||
<td><a href="git/lib.sh">git</a></td> | ||
<td>Git helpers</td> | ||
<td> | ||
<ol> | ||
<li>repo_root: find the root of the current git repo</li> | ||
<li>all_files_in_repo: list files tracked by git</li> | ||
</ol> | ||
</td> | ||
</tr> | ||
<td><a href="helpers/lib.sh">helpers</a></td> | ||
<td>Bash scripting helpers</td> | ||
<td> | ||
<ol> | ||
<li>die: print message and exit 1</li> | ||
<li>spushd/spopd: safe verisons of pushd & popd that call die if the push/pop fails, they also drop stdout. </li> | ||
</ol> | ||
</td> | ||
</tr> | ||
<tr> | ||
<td><a href="k8s/lib.sh">k8s</a></td> | ||
<td>Utils for connecting to K8s</td> | ||
<td> | ||
<ol> | ||
<li>build_gke_image: Build docker image for running kubectl commands against GKE</li> | ||
<li>delete_gke_image: Delete image from GKE</li> | ||
<li>run_docker_gke_command: Run command in gke-utils container, already authenticated to k8s cluster.</li> | ||
</ol> | ||
</td> | ||
</tr> | ||
<tr> | ||
<td><a href="logging/lib.sh">logging</a></td> | ||
<td>Helpers related to login</td> | ||
<td> | ||
<ol> | ||
<li>announce: echo message in ascii banner to distinguish it from other log messages.</li> | ||
</ol> | ||
</td> | ||
</tr> | ||
<tr> | ||
<td><a href="test-utils/lib.sh">test-utils</a></td> | ||
<td>Helpers for executing tests</td> | ||
<td> | ||
<ol> | ||
<li>shellcheck_script: execute shellcheck against a script, uses docker.</li> | ||
<li>find_scripts: find git tracked files with .sh extension</li> | ||
<li>tap2junit: Convert a subset of <a href="http://testanything.org/">TAP</a> to JUnit XML. Retains logs for errors</li> | ||
</ol> | ||
</td> | ||
</tr> | ||
</tbody> | ||
</table> | ||
|
||
## Testing | ||
Tests are written using [BATS](https://github.com/bats-core/bats). Each lib should have a `lib-name.bats` file in [tests-for-this-repo](/tests-for-this-repo). | ||
Asserts are provided by [bats-assert-1](https://github.com/jasonkarns/bats-assert-1). The value in these is that they provide useful debugging output when the assertion fails, eg expected x got y. | ||
|
||
Example: | ||
```bash | ||
# source support and assert libraries | ||
. "${BASH_LIB}/test-utils/bats-support/load.bash" | ||
. "${BASH_LIB}/test-utils/bats-assert-1/load.bash" | ||
|
||
# source the library under test | ||
. "${BASH_LIB}/git/lib.sh" | ||
|
||
# define a test that calls a library function | ||
@test "it does the thing" { | ||
some_prep_work | ||
# run is a wrapper that catches failures so that assertsions can be run, | ||
# otherwise the test would immediately fail. | ||
run does_the_thing | ||
assert_success | ||
assert_output "thing done" | ||
} | ||
``` | ||
|
||
Test fixtures should go in /tests-for-this-repo/fixtures/lib-name. | ||
|
||
|
||
### Footnotes | ||
<a name="footnote-1">1</a>: Early Readers/Reviewers may notice that the | ||
submodule alternative clone function fails. This is because most of the | ||
content hasn't been merged to master yet (its still in the initial_libs branch) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
#!/bin/bash | ||
|
||
: "${BASH_LIB:?BASH_LIB must be set. Please source bash-lib/init.sh before other scripts from bash-lib.}" | ||
. "${BASH_LIB}/helpers/lib.sh" | ||
|
||
#https://stackoverflow.com/a/23002317 | ||
function abs_path() { | ||
# generate absolute path from relative path | ||
# $1 : relative filename | ||
# return : absolute path | ||
if [ -d "$1" ]; then | ||
# dir | ||
(spushd "$1"; pwd) | ||
elif [ -f "$1" ]; then | ||
# file | ||
if [[ $1 = /* ]]; then | ||
echo "$1" | ||
elif [[ $1 == */* ]]; then | ||
echo "$(spushd "${1%/*}"; pwd)/${1##*/}" | ||
else | ||
echo "$(pwd)/$1" | ||
fi | ||
fi | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
#!/bin/bash | ||
|
||
: "${BASH_LIB:?BASH_LIB must be set. Please source bash-lib/init.sh before other scripts from bash-lib.}" | ||
|
||
# Get the top level of a git repo | ||
function repo_root(){ | ||
git rev-parse --show-toplevel | ||
} | ||
|
||
# List files tracked by git | ||
function all_files_in_repo(){ | ||
git ls-tree -r HEAD --name-only | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
#!/bin/bash | ||
|
||
: "${BASH_LIB:?BASH_LIB must be set. Please source bash-lib/init.sh before other scripts from bash-lib.}" | ||
|
||
function die(){ | ||
echo "${@}" | ||
exit 1 | ||
} | ||
|
||
#safe pushd | ||
function spushd(){ | ||
pushd "${1}" >/dev/null || die "pushd ${1} failed :(" | ||
} | ||
|
||
#safe popd | ||
function spopd(){ | ||
popd >/dev/null || die "popd failed :(" | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,37 @@ | ||
#!/bin/bash | ||
|
||
## Initialisation Functions for the | ||
## Conjurinc Bash Library | ||
|
||
# Shell Otions | ||
set -euo pipefail | ||
|
||
# This script should be sourced before any of | ||
# the other scripts in this repo. Other scripts | ||
# make use of ${BASH_LIB} to find each other. | ||
|
||
# Get the relative path to the repo root | ||
# shellcheck disable=SC2086 | ||
BASH_LIB_RELATIVE="$(dirname ${BASH_SOURCE[0]})" | ||
|
||
# Must be set in order to load the filehandling | ||
# module. Will be updated when abs_path is available. | ||
BASH_LIB="${BASH_LIB_RELATIVE}" | ||
|
||
# Load the filehandling module for the abspath | ||
# function | ||
. "${BASH_LIB_RELATIVE}/filehandling/lib.sh" | ||
|
||
# Export the absolute path | ||
# shellcheck disable=SC2086 | ||
BASH_LIB="$(abs_path ${BASH_LIB_RELATIVE})" | ||
export BASH_LIB | ||
|
||
. "${BASH_LIB}/helpers/lib.sh" | ||
|
||
# Update Submodules | ||
spushd "${BASH_LIB}" | ||
git submodule update --init --recursive | ||
spopd | ||
|
||
export BATS_CMD="${BASH_LIB}/test-utils/bats/bin/bats" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
FROM google/cloud-sdk | ||
|
||
RUN mkdir -p /src /scripts | ||
WORKDIR /src | ||
COPY platform_login.sh /scripts | ||
|
||
# Install Docker client | ||
RUN apt-get update -y && \ | ||
apt-get install -y apt-transport-https ca-certificates curl gnupg2 software-properties-common wget && \ | ||
curl -fsSL https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")/gpg | apt-key add - && \ | ||
add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") $(lsb_release -cs) stable" && \ | ||
apt-get update && \ | ||
apt-get install -y docker-ce && \ | ||
rm -rf /var/lib/apt/lists/* | ||
|
||
# Install kubectl CLI | ||
ARG KUBECTL_CLI_URL | ||
RUN wget -O /usr/local/bin/kubectl ${KUBECTL_CLI_URL:-https://storage.googleapis.com/kubernetes-release/release/v1.7.6/bin/linux/amd64/kubectl} && \ | ||
chmod +x /usr/local/bin/kubectl |
Oops, something went wrong.