-
Notifications
You must be signed in to change notification settings - Fork 24
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #1 from conjurinc/initial_libs
Initialise bash-lib repo
- Loading branch information
Showing
152 changed files
with
9,350 additions
and
2 deletions.
There are no files selected for viewing
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
# Git Subtrees | ||
|
||
# The advantage of subtrees is that users don't have to care about them - its | ||
# just a single repo. The disadvantage is that git doesn't track the metadata | ||
# as it does for submodules. | ||
|
||
# This file provides an enumeration of the subtrees in this repo, and the URLs | ||
# they came from. | ||
|
||
# subtree_path remote_url remote_name | ||
test-utils/bats https://github.com/bats-core/bats bats | ||
test-utils/bats-support https://github.com/ztombol/bats-support bats-support | ||
test-utils/bats-assert-1 https://github.com/jasonkarns/bats-assert-1 bats-assert-1 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,47 @@ | ||
#!/usr/bin/env groovy | ||
|
||
pipeline { | ||
agent { label 'executor-v2' } | ||
|
||
options { | ||
timestamps() | ||
buildDiscarder(logRotator(numToKeepStr: '30')) | ||
} | ||
|
||
triggers { | ||
cron(getDailyCronString()) | ||
} | ||
|
||
environment { | ||
BATS_OUTPUT_FORMAT="junit" | ||
} | ||
|
||
stages { | ||
|
||
stage('BATS Tests') { | ||
steps { | ||
sh './tests-for-this-repo/run-bats-tests' | ||
} | ||
} | ||
|
||
stage('Python Linting') { | ||
steps { | ||
sh './tests-for-this-repo/run-python-lint' | ||
} | ||
} | ||
|
||
stage('Secrets Leak Check') { | ||
steps { | ||
sh './tests-for-this-repo/run-gitleaks' | ||
} | ||
} | ||
|
||
} | ||
|
||
post { | ||
always { | ||
junit '*-junit.xml' | ||
cleanupAndNotify(currentBuild.currentResult) | ||
} | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,196 @@ | ||
# bashlib | ||
Common bash functions for use in test pipelines | ||
# bash-lib | ||
``` | ||
_______________ _______________ | ||
.' .' .| | ||
.' .' .' | | ||
.'_______________.'______________ .' | | ||
| ___ _____ ___ || ___ _____ ___ | | | ||
||_=_|__=__|_=_||||_=_|__=__|_=_|| | | ||
______||_____===_____||||_____===_____|| | __________ | ||
.' ||_____===_____||||_____===_____|| .' .'| | ||
.' ||_____===_____||||_____===_____|| .' .' | | ||
.'___________|_______________||_______________|.'__________.' | | ||
|.----------.|.-----___-----.||.-----___-----.|| |_____.----------. | ||
|] |||_____________||||_____________||| .' [ | | ||
|| ||.-----___-----.||.-----___-----.||.' | | | ||
|| |||_____________||||_____________|||==========| | | ||
|| ||.-----___-----.||.-----___-----.|| |_____| | | ||
|] o|||_____________||||_____________||| .' [ 'o| | ||
|| ||.-----___-----.||.-----___-----.||.' | | | ||
|| ||| ||||_____________|||==========| | | ||
|| ||| |||.-----___-----.|| |_____| | | ||
|] ||| |||| ||| .' [ | | ||
||__________|||_____________||||_____________|||.'________|__________| | ||
''----------'''------------------------------'''----------'' | ||
(o)LGB (o) | ||
``` | ||
|
||
The place to store functions that are used in pipelines for multiple repos. | ||
|
||
Please add whatever is useful to you, but keep it tidy so its still useful to everyone else :) | ||
|
||
## Usage | ||
|
||
Add bash-lib into your project in the way that best fits your workflow. The only requirement is that you **pin the version of | ||
bash-lib that you use**. This is important so that changes to bash-lib do not have the power to break all projects that use | ||
bash-lib. Your project can then test updates to bash-lib and roll forward periodicly. | ||
|
||
Options: | ||
* Add a submodule: they are an easy way to integrate bash-lib and automatically use a single SHA until manually updated. Submodules add a pointer from a mount point in your repo to the external repo (bash-lib), and require workflow changes to ensure that pointer is derferenced during clone, checkout and some other opertaions. | ||
* Add a subtree: This repo uses subtrees to pull in test dependencies. Subtrees copy an external repo into a subdirectory of the host repo, no workflow changes are required. Subtrees naturally keep a single version of bash-lib until explicitly updated. Note that subtree merge commits do not rebase well :warning:, so best to keep subtree updates in separate PRs from normal commits. | ||
* Clone bash-lib in your deployment process, bash-lib doesn't have to be within your repo, just needs to be somewhere where your scripts can source [init](init). This is where it's most important that you implement a mechanism to always use the same SHA, as a **clone will track master by default, which is not an allowed use of bash-lib**. | ||
|
||
Once you have bash-lib cloned in your project, you source two things: | ||
|
||
1. Source `bash-lib/init`. This ensures submodules are initalised and sets the BASH_LIB_DIR env var to the absolute path to the bash-lib dir. This makes it easy to source libraries from other scripts. | ||
2. Source `${BASH_LIB_DIR}/lib-name/lib` for any libraries you are interested in. | ||
|
||
You are now ready to use bash-lib functions :) | ||
|
||
## Structure | ||
The `/init` script sets up everything required to use the library, most | ||
importantly the `BASH_LIB_DIR` variable which gives the absolute path to the root | ||
of the library and should be used for sourcing the modules. | ||
|
||
The repo is organized into libraries, each library is a directory that has a | ||
lib file. Sourcing the lib for a library should expose all the functions | ||
that library offers. The lib file may source or reference other supporting | ||
files within it's directory. | ||
|
||
``` | ||
. | ||
├── libname | ||
│ ├── lib | ||
│ └── supporting-file | ||
├── init # init script, source this first | ||
├── run-tests # top level test script, executes all tests | ||
├── secrets.yml # secrets required for executing tests | ||
├── test-utils | ||
│ ├── bats # subtree | ||
│ ├── bats-assert-1 # subtree | ||
│ ├── bats-support # subtree | ||
│ ├── lib | ||
│ └── tap2junit | ||
└── tests-for-this-repo | ||
├── filehandling.bats | ||
├── fixtures # | ||
│ └── libname # Dir containing test fixtures for a library | ||
├── tap2junit | ||
├── libname.bats # contains tests for libname/lib | ||
├── python-lint # supporting files for python lint | ||
├── run-bats-tests # script to run bats tests | ||
├── run-gitleaks # script to check for leaked secrets | ||
└── run-python-lint # script to run python lint | ||
``` | ||
## Style Guide | ||
Follow the [google shell style guide](https://google.github.io/styleguide/shell.xml#Naming_Conventions). | ||
TL;DR: | ||
1. Use snake_case function and variable names | ||
1. Use `function` when declaring functions. | ||
|
||
|
||
## Contents | ||
|
||
<!-- html table due to markdown's lack of support for lists within tables --> | ||
<table> | ||
<thead> | ||
<tr> | ||
<th>Library</th> | ||
<th>Description</th> | ||
<th>Functions</th> | ||
</tr> | ||
</thead> | ||
<tbody> | ||
<tr> | ||
<td><a href="filehandling/lib">filehandling</a></td> | ||
<td>Functions relating to file and path handling | ||
<td> | ||
<ol> | ||
<li> <b>abs_path</b>: Ensure a path is absolute</li> | ||
</ol> | ||
</td> | ||
</tr> | ||
<tr> | ||
<td><a href="git/lib">git</a></td> | ||
<td>Git helpers</td> | ||
<td> | ||
<ol> | ||
<li><b>repo_root</b>: Find the root of the current git repo.</li> | ||
<li><b>all_files_in_repo</b>: List files tracked by git.</li> | ||
<li><b>remote_latest_tag</b>: Returns the symbolic name of the latest tag from a remote.</li> | ||
<li><b>remote_latest_tagged_commit</b>: Returns the SHA of the most recently tagged commit in a remote repo (<code>tag^{}</code>).</li> | ||
<li><b>remote_sha_for_ref</b>: Returns the SHA for a given ref from a named remote.</li> | ||
<li><b>remote_tag_for_sha</b>: Returns the tag corresponding to a SHA from a named remote - if there is one.</li> | ||
<li>tracked_files_excluding_subtrees: List files tracked by git, but excluding any files that are in paths listed in <code>.gittrees</code>.</li> | ||
<li><b>cat_gittrees</b>: Returns the contents of .gittrees from the top level of the repo, excluding any comments. Fails if .gittrees is not present.</li> | ||
</ol> | ||
</td> | ||
</tr> | ||
<td><a href="helpers/lib">helpers</a></td> | ||
<td>Bash scripting helpers</td> | ||
<td> | ||
<ol> | ||
<li><b>die</b>: print message and exit 1</li> | ||
<li><b>spushd/spopd</b>: Safe verisons of pushd & popd that call die if the push/pop fails, they also drop stdout. </li> | ||
</ol> | ||
</td> | ||
</tr> | ||
<tr> | ||
<td><a href="k8s/lib">k8s</a></td> | ||
<td>Utils for connecting to K8s</td> | ||
<td> | ||
<ol> | ||
<li><b>build_gke_image</b>: Build docker image for running kubectl commands against GKE.</li> | ||
<li><b>delete_gke_image</b>: Delete image from GKE.</li> | ||
<li><b>run_docker_gke_command</b>: Run command in gke-utils container, already authenticated to k8s cluster.</li> | ||
</ol> | ||
</td> | ||
</tr> | ||
<tr> | ||
<td><a href="logging/lib">logging</a></td> | ||
<td>Helpers related to login</td> | ||
<td> | ||
<ol> | ||
<li><b>announce</b>: Echo message in ascii banner to distinguish it from other log messages.</li> | ||
</ol> | ||
</td> | ||
</tr> | ||
<tr> | ||
<td><a href="test-utils/lib">test-utils</a></td> | ||
<td>Helpers for executing tests</td> | ||
<td> | ||
<ol> | ||
<li><b>shellcheck_script</b>: Execute shellcheck against a script, uses docker.</li> | ||
<li><b>find_scripts</b>: Find git tracked files with extension.</li> | ||
<li><b>tap2junit</b>: Convert a subset of <a href="http://testanything.org/">TAP</a> to JUnit XML. Retains logs for errors.</li> | ||
</ol> | ||
</td> | ||
</tr> | ||
</tbody> | ||
</table> | ||
|
||
## Testing | ||
Tests are written using [BATS](https://github.com/bats-core/bats). Each libould have a `lib-name.bats` file in [tests-for-this-repo](/tests-for-this-repo). | ||
Asserts are provided by [bats-assert-1](https://github.com/jasonkarns/bats-assert-1). The value in these is that they provide useful debugging output when the assertion fails, eg expected x got y. | ||
|
||
Example: | ||
```bash | ||
# source support and assert libraries | ||
. "${BASH_LIB_DIR}/test-utils/bats-support/load.bash" | ||
. "${BASH_LIB_DIR}/test-utils/bats-assert-1/load.bash" | ||
|
||
# source the library under test | ||
. "${BASH_LIB_DIR}/git/lib" | ||
|
||
# define a test that calls a library function | ||
@test "it does the thing" { | ||
some_prep_work | ||
# run is a wrapper that catches failures so that assertsions can be run, | ||
# otherwise the test would immediately fail. | ||
run does_the_thing | ||
assert_success | ||
assert_output "thing done" | ||
} | ||
``` | ||
|
||
Test fixtures should go in /tests-for-this-repo/[fixtures](tests-for-this-repo/fixtures)/lib-name. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
#!/bin/bash | ||
|
||
: "${BASH_LIB_DIR:?BASH_LIB_DIR must be set. Please source bash-lib/init before other scripts from bash-lib.}" | ||
. "${BASH_LIB_DIR}/helpers/lib" | ||
|
||
#https://stackoverflow.com/a/23002317 | ||
function abs_path() { | ||
# generate absolute path from relative path | ||
# $1 : relative filename | ||
# return : absolute path | ||
if [ -d "$1" ]; then | ||
# dir | ||
(spushd "$1"; pwd) | ||
elif [ -f "$1" ]; then | ||
# file | ||
if [[ $1 = /* ]]; then | ||
echo "$1" | ||
elif [[ $1 == */* ]]; then | ||
echo "$(spushd "${1%/*}"; pwd)/${1##*/}" | ||
else | ||
echo "$(pwd)/$1" | ||
fi | ||
fi | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,79 @@ | ||
#!/bin/bash | ||
|
||
: "${BASH_LIB_DIR:?BASH_LIB_DIR must be set. Please source bash-lib/init before other scripts from bash-lib.}" | ||
. "${BASH_LIB_DIR}/helpers/lib" | ||
|
||
# Get the top level of a git repo | ||
function repo_root(){ | ||
git rev-parse --show-toplevel | ||
} | ||
|
||
# List files tracked by git | ||
function all_files_in_repo(){ | ||
git ls-tree -r HEAD --name-only | ||
} | ||
|
||
# Find the latest tag available at a repo url | ||
# Returns tag name, not sha | ||
function remote_latest_tag(){ | ||
local -r remote_url="${1}" | ||
# In ls-remote the ^{} suffix refers to a peeled/dereferenced object. | ||
# eg refs/tags/v0.0.1^{} shows the SHA of the commit that was tagged, | ||
# not the SHA of the tag itself. | ||
# Adding --refs hides peeled tags | ||
git ls-remote --tags --refs --quiet \ | ||
"${remote_url}" \ | ||
| tail -n 1 \ | ||
| cut -f 2 \ | ||
| sed -e 's+refs/tags/++' | ||
} | ||
|
||
# Find the SHA of the latests commit to be tagged in a remote repo | ||
function remote_latest_tagged_commit(){ | ||
local -r remote="${1}" | ||
local -r tag="$(remote_latest_tag "${remote}")" | ||
git ls-remote "${remote}" | awk "/refs\/tags\/${tag}\^/{print \$1}" | ||
} | ||
|
||
function remote_sha_for_ref(){ | ||
local -r remote="${1}" | ||
local -r ref="${2}" | ||
|
||
# First try adding ^{} to the ref, incase it's a tag | ||
# and needs peeling. If nothing is found for that, | ||
# try without. | ||
peeled_ref=$( | ||
git ls-remote "${remote}" \ | ||
| awk "/${ref}[^$]/{print \$1}" | ||
) | ||
|
||
if [[ -n "${peeled_ref}" ]]; then | ||
echo "${peeled_ref}" | ||
else | ||
git ls-remote "${remote}" \ | ||
| awk "/${ref}/{print \$1}" | ||
fi | ||
} | ||
|
||
function remote_tag_for_sha(){ | ||
local -r remote="${1}" | ||
local -r sha="${2}" | ||
git ls-remote "${remote}" \ | ||
| awk -F'/' "/${sha}.*tag/{ gsub(/\^\{\}\$/, \"\"); print \$3 }" | ||
} | ||
|
||
|
||
## Minimal git subtree functionality required for tests to pass | ||
# full subtree functionality is not ready for merge. | ||
function cat_gittrees(){ | ||
local -r git_trees="$(repo_root)/.gittrees" | ||
local -r subtrees_file_format=".gittrees should contain one subtree per line,\ | ||
space seperated with three fields: subtree_path renmote_url remote_name" | ||
[[ -e "${git_trees}" ]] || die ".gittrees file ${git_trees} not found. ${subtrees_file_format}" | ||
grep -E -v '^\s*$|^\s*#' "$(repo_root)/.gittrees" | ||
} | ||
|
||
function tracked_files_excluding_subtrees(){ | ||
subtrees="$(cat_gittrees | awk '{print $1}' | paste -sd '|' -)" | ||
all_files_in_repo | grep -E -v "${subtrees}" | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
#!/bin/bash | ||
|
||
: "${BASH_LIB_DIR:?BASH_LIB_DIR must be set. Please source bash-lib/init before other scripts from bash-lib.}" | ||
|
||
function die(){ | ||
echo "${@}" | ||
exit 1 | ||
} | ||
|
||
#safe pushd | ||
function spushd(){ | ||
if ! pushd "${1}" >/dev/null; then | ||
die "pushd ${1} failed :(" | ||
fi | ||
} | ||
|
||
#safe popd | ||
function spopd(){ | ||
popd >/dev/null || die "popd failed :(" | ||
} |
Oops, something went wrong.