docs: auto-deploy documentation from the Travis-CI build #2222

oliver-sanders · 2018-08-16T12:11:19Z

If building a git tag (as opposed to a branch) then append the built docs to the current gh-pages branch and push back the result, GitHub will take care of the rest.

Documentation is built incrementally atop what has gone before so changes are not destructive, i.e we don't need to do a force push.

Every now and again we will want to remove older versions from the documentation, this can be done manually:

Check out the gh-pages branch and delete the relevant commit.
If the relevant commit is the first one, delete the relevant directory and squash the commit with the next one.

Example in motion:

Example of Travis building a regular branch
- TB output: https://travis-ci.org/oliver-sanders/rose/builds/416776038
Example of Travis building a tag (release)
- TB output: https://travis-ci.org/oliver-sanders/rose/builds/416776730
- gh-pages branch: http://oliver-sanders.github.io/rose/doc/html/index.html

Before this can work on metomi/rose

We need a GitHub o-auth token with the permission "public_repo". This then has to be provided to Travis-Ci (via the dashboard) as the environment variable GH_TOKEN.

etc/travis-scripts

matthewrmshin · 2018-08-21T08:16:34Z

etc/travis-scripts

+    if ! rose make-docs --venv --dev --strict clean html slides pdf \
+            ${DEFAULT_VERSION:-} >out 2>&1; then
+        # output is a bit long, only output it if the docs fail to build
+        cat out


Contains both 1 & 2, we could cat to &2 but Travis doesn't distinguish so it makes no difference either way.

(Agreed. However it would look neater to redirect error to standard error before an exit 1. Sorry, but my brain is wired to expect this pattern.)

bin/rose-make-docs

etc/travis-scripts

oliver-sanders · 2018-08-22T11:02:54Z

De-conflicted.

matthewrmshin · 2018-09-10T10:40:33Z

@sadielbartholomew please review, but do not merged until we have sorted out the admin.

sadielbartholomew

Looks good & is functional, as described, as far as I can see & validate (obviously I cannot test deployment itself). A few minor points to consider while the admin is settled, but nothing essential.

sadielbartholomew · 2018-09-10T13:54:06Z

etc/deploy-docs

@@ -58,19 +58,20 @@ initial_deployment () {  # Create a new gh-pages branch from scratch.

 subsequent_deployment () {  # Add new version to documentation.
    # update gh-pages branch
-    git pull "${REMOTE}" gh-pages -f
+    git fetch "${REMOTE}" gh-pages -f


Why is the merge stage no longer required, out of interest?

Because we branch from the upstream version by name:

git checkout -B gh-pages upstream/gh-pages

sadielbartholomew · 2018-09-10T14:10:56Z

bin/rose-make-docs

@@ -40,6 +40,7 @@
 #     --default-version=VERSION
 #         By default the current version is symlinked as the default version,
 #         provide an alternative version to override this.
+#         If set to 'none' then the default version will remain unchanged.


Throughout travis-scripts --default-version none is used as opposed to --default-version=none as documented. I've tested these out & only the former syntax seems to work, so ideally we can update this docstring accordingly to prevent confusion.

Well spotted!

sadielbartholomew · 2018-09-10T14:17:18Z

etc/travis-scripts

+export PATH="${PWD}/bin:${HOME}/fcm-master/bin:${HOME}/cylc-master/bin:$PATH"
+
+TMP_DOCS_DIR="${TMPDIR:-/var/tmp}/temp-docs-dir"
+DOCUMENTATION_FILES=!(doc|404*|CHANGES.md|_config.yml)


Since (as briefly discussed in person) a bash -n travis-scripts vaildation is confused by this extended globbing line, consider appending a comment here identifying this as such (if I may be pedantic).

No idea why bash -n doesn't like the extglobs, perhaps it doesn't support them. For proper linting of shell script use something like shellcheck.

sadielbartholomew · 2018-09-10T14:29:47Z

etc/travis-scripts

+    # Cylc
+    if grep 'cylc' <<< "${args[@]}"; then
+        # install Cylc
+        wget 'https://github.com/cylc/cylc/archive/master.tar.gz' -O \


Now we have migrated from the monolithic .yml file, it might be useful to move some/all of the site link definitions (or at least the bases e.g. https://github.com/[cylc/cylc/]) from which we extract files etc. to the head of the file so that they are immediately explicit? Then again it might be clearer to define them within the functions they are used only... I'll leave it for consideration.

https://github.com/cylc/cylc/ is only used once so I don't think we would have much to gain, of have I misunderstood?

There is not much duplication, true, but the links are constants that are fundamental to the scripts, so putting them at the head of the script might be more appropriate & would make it clear where we are sourcing the codebase installations from. There is not much to gain now, admittedly, but may be useful for future reference. We can leave it be, though.

oliver-sanders · 2018-09-24T14:49:27Z

~~Update this against #2235 if it is merged first.~~
Deconflicted

oliver-sanders · 2018-10-31T13:58:27Z

Time to create a metomi "actor" account which we can use to perform actions on the metomi repositories?

oliver-sanders · 2018-11-13T17:05:14Z

As GitHub Pages builds from branches in the GitHub repository it is necessary to use a personal access token, no changes to Travis-CI will facilitate this.

Out of interest there is a script called travis-sphinx which some other projects are using. It does roughly the same thing as this PR but pushes in force mode.

Our Options:

Auto-deploy as in this PR.
Auto-deploy using this code but building the docs elsewhere (no real change):
- On a different CI platform;
- On Institutional or personal hardware.
Continue to build docs on institutional or personal hardware and deploy manually.
Move documentation elsewhere e.g. readthedocs.
- URLs wouls have to be re-directed.
- Would tie us in with the readthedocs theme.
- Would need to research if we could continue to use our slide builder there.

oliver-sanders · 2018-11-27T12:34:33Z

Rebased onto latest Rose.

matthewrmshin · 2018-11-27T14:44:13Z

etc/travis-scripts

+
+    # push gh-pages branch to GitHub where it will be auto-deployed to ghpages
+    set +o xtrace  # prevent leakage of the GH_TOKEN env var
+    git push "https://${GH_TOKEN}@github.com/${TRAVIS_REPO_SLUG}.git" \


This will still reveal the GH_TOKEN variable if we do ps -f listing while the git push command is running?

matthewrmshin · 2019-01-17T12:08:23Z

Apologies. Push back for now as we cannot see how the Personal Access Token can be deployed properly.

kinow · 2019-02-02T09:44:15Z

Hey 👋 ! Just passing by, noticed this part

Every now and again we will want to remove older versions from the documentation, this can be done manually:

Check out the gh-pages branch and delete the relevant commit.
If the relevant commit is the first one, delete the relevant directory and squash the commit with the next one.

FWIW, I use ghp-import for my site. As I am not using Jekyll, in order to deploy to GitHub pages, I have a small script that builds calling a static site generator for Python. This puts the contents in a folder like ./blog... then I just execute:

$ ghp-import -n -m "Publishing kinoshita.eti.br" -p -r origin -b gh-pages blog

The important flags are -p to push to the branch after committing, -r to specify which git remote repo. The gh-pages is the branch, and blog is the local folder to push.

Have no idea if that'd be helpful. Maybe there's something in ghp-import that's helpful 👋

oliver-sanders · 2019-02-04T10:07:36Z

There is a sphinx-build wrapper thinggy called travis-sphinx which sits atop ghp-import. I looked into it but it seemed that implementing the required logic was probably simpler. Unfortunately ghp-import has the same issue with personal access tokens, we are waiting on GitHub to come up with a cleaner solution.

matthewrmshin · 2019-03-14T20:34:46Z

I'll close this one down for now - until we find a way that does not require a personal access token.

oliver-sanders · 2020-03-17T03:44:46Z

I'll close this one down for now - until we find a way that does not require a personal access token.

Wahoo, there is now an alternative!

We can now use "deploy-keys", these are specific to a repository, to work for purpose they need to be given write permission.

Here's a GH action which does the job - https://github.com/peaceiris/actions-gh-pages#️-create-ssh-deploy-key

The permissions of the key are a little broader than I would desire (would be nice to be able to specify write for the gh-pages branch only, and also to ban force pushes), however, the tokens are repo specific and have a much less scary scope than personal access tokens.

I'm testing it on the cylc-sphinx-extensions repo

oliver-sanders added this to the next-release milestone Aug 16, 2018

oliver-sanders self-assigned this Aug 16, 2018

oliver-sanders requested review from matthewrmshin and sadielbartholomew August 16, 2018 12:11

oliver-sanders mentioned this pull request Aug 16, 2018

rose.config: python 3 compatibility #2221

Closed

matthewrmshin reviewed Aug 21, 2018

View reviewed changes

oliver-sanders force-pushed the docs-deploy-instructions branch 2 times, most recently from fbfbe2b to f4c2d02 Compare August 22, 2018 11:02

matthewrmshin approved these changes Sep 10, 2018

View reviewed changes

sadielbartholomew approved these changes Sep 10, 2018

View reviewed changes

oliver-sanders force-pushed the docs-deploy-instructions branch from 729437d to 0d159b2 Compare September 24, 2018 16:39

oliver-sanders force-pushed the docs-deploy-instructions branch from 0d159b2 to ed428ed Compare October 10, 2018 10:27

oliver-sanders added 2 commits November 27, 2018 12:34

deploy documentation from the Travis-CI integration

0a7da68

rose-make-docs: correct --default-version docs

cc28a50

oliver-sanders force-pushed the docs-deploy-instructions branch from ed428ed to cc28a50 Compare November 27, 2018 12:34

oliver-sanders mentioned this pull request Nov 27, 2018

Add coverage to project reports. #2230

Merged

matthewrmshin reviewed Nov 27, 2018

View reviewed changes

matthewrmshin modified the milestones: 2018.11.0, next-release Nov 28, 2018

matthewrmshin modified the milestones: next-release, soon Jan 17, 2019

matthewrmshin closed this Mar 14, 2019

matthewrmshin removed this from the soon milestone Mar 14, 2019

matthewrmshin added wontfix doc labels Mar 14, 2019

oliver-sanders mentioned this pull request Mar 22, 2019

Travis tidy #2318

Merged

oliver-sanders mentioned this pull request Mar 17, 2020

actions: fix deployment script cylc/cylc-sphinx-extensions#19

Merged

oliver-sanders mentioned this pull request Jan 27, 2021

docs: auto deploy docs to gh pages #2436

Open

MetRonnie added the infrastructure GH Actions, Codecov etc. label Mar 24, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: auto-deploy documentation from the Travis-CI build #2222

docs: auto-deploy documentation from the Travis-CI build #2222

oliver-sanders commented Aug 16, 2018

matthewrmshin Aug 21, 2018

oliver-sanders Aug 21, 2018

matthewrmshin Aug 21, 2018

oliver-sanders commented Aug 22, 2018

matthewrmshin commented Sep 10, 2018

sadielbartholomew left a comment

sadielbartholomew Sep 10, 2018

oliver-sanders Sep 10, 2018

sadielbartholomew Sep 10, 2018

oliver-sanders Sep 10, 2018

sadielbartholomew Sep 10, 2018

oliver-sanders Sep 10, 2018

sadielbartholomew Sep 10, 2018

oliver-sanders Sep 10, 2018

sadielbartholomew Sep 10, 2018

oliver-sanders commented Sep 24, 2018 •

edited

Loading

oliver-sanders commented Oct 31, 2018

oliver-sanders commented Nov 13, 2018

oliver-sanders commented Nov 27, 2018

matthewrmshin Nov 27, 2018

matthewrmshin commented Jan 17, 2019

kinow commented Feb 2, 2019

oliver-sanders commented Feb 4, 2019 •

edited

Loading

matthewrmshin commented Mar 14, 2019

oliver-sanders commented Mar 17, 2020

docs: auto-deploy documentation from the Travis-CI build #2222

docs: auto-deploy documentation from the Travis-CI build #2222

Conversation

oliver-sanders commented Aug 16, 2018

If building a git tag (as opposed to a branch) then append the built docs to the current gh-pages branch and push back the result, GitHub will take care of the rest.

Before this can work on metomi/rose

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oliver-sanders commented Aug 22, 2018

matthewrmshin commented Sep 10, 2018

sadielbartholomew left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oliver-sanders commented Sep 24, 2018 • edited Loading

oliver-sanders commented Oct 31, 2018

oliver-sanders commented Nov 13, 2018

oliver-sanders commented Nov 27, 2018

Choose a reason for hiding this comment

matthewrmshin commented Jan 17, 2019

kinow commented Feb 2, 2019

oliver-sanders commented Feb 4, 2019 • edited Loading

matthewrmshin commented Mar 14, 2019

oliver-sanders commented Mar 17, 2020

oliver-sanders commented Sep 24, 2018 •

edited

Loading

oliver-sanders commented Feb 4, 2019 •

edited

Loading