Support building arbitrary versions from PRs

[agjohnson]

We have a number of design decisions to make around how to support building an arbitrary version/repo from a PR status update from GitHub/Bitbucket/Gitlab. We will need to decide the scope of how these versions are hosted, how we track the status updates, and if we need to make temporary version/projects to support arbitrary versions/contributor repos. The integrations will also need to track the attached GitHub PR, as the status update will send us a pk to send updates against -- at least in the case of GitHub.

[cben]

A smaller(?) proposal that would still be helpful for reviewing doc changes:
Start without any GitLab etc webhook integration; just support ephemeral https://rawgit.com-style URL route that includes fork & branch name / commit hash, that when requested does the build on-demand.
(As with rawgit, this is also useful outside PRs — to preview your changes before sending a PR, to do "archeology", and to permalink specific versions)

Building on demand would obviously be slow & expensive if several people click it (or one clicks around pages...), so the build has to be cached for some time.
But you can treat it as just a cache — PR builds are of little interest in the long run, you don't have to guarantee retention; in some cases you'll waste cpu re-building it, but that's just a cpu/storage tradeoff...

At later stage, could integrate webhooks, which allows to:

• kick off a build (prewarming cache)
• have link to built docs in PR — more convenient and much more discovrable
• green/red build status if the changes actually break the build
• better heuristics for cache expiration

[stsewd]

+1 for this option

green/red build status if the changes actually break the build

Also exposing the build logs (no generate docs, just logs).

[stsewd]

I was thinking that this feature can be implemented like this:

• When a PR is updated a webhook is called
• RTD adds a new remote to the original repo and fetch the PR branch
• A build is triggered
• The resulting docs are served under other namespaces (/pr/{some_id}/ ?)

With that RTD will need to have a section for managing this (manual rebuild, deactivate and so on). When be the PRs deleted? Not sure about that, in case of GitHub we can configure a webhook. Also, when building a oficial version we can check if the repo contains a branch from a PR I think, and then delete those docs.

[humitos]

The resulting docs are served under other namespaces (/pr/{some_id}/ ?)

In any case, if we serve the PR's built documentation, we should think about how cleanup these ones (e.g. remove the docs after the PR is merged/after a couple of days/etc) otherwise we will end up with Gigabytes of useless documentation.

[stsewd]

We can use this option from git (or gitpython) to check if a branch was merged https://git-scm.com/docs/git-branch#git-branch---mergedltcommitgt git branch -r --merged master or git branch -r --no-merged master, on every build we can purge the merged branches. This would be like an additional protection since we can have this info from the GitHub webhook or from the API (pretty sure the others services have something similar).

I can see a problem when a PR is closed and we don't receive that info from the webhook (some connection error, for example). If we don't want to hit the api to know the PR status, we can have a max lifetime for each PR since the last build. Again, that can be checked on each build from an official version.

[jmbowman]

Would any of the code or ideas from Precog be useful in designing this? The hosted service is defunct, but the code is there under an open license. It's a webhook-driven Flask app which supports per-PR static sites by linking to the artifacts from a corresponding CircleCI build.

I'm currently looking into options for auto-building doc previews for the assorted pypa projects (setuptools, pip, etc.), in order to expedite reviews of doc changes. It would be great if RTD could handle this, since the repos are already integrated with it and publish the docs there.

It seems like there are a few different tickets now for largely the same thing, and each mentions some of the blockers. Should those blockers be prioritized and broken out into separate tickets so we can start making progress on them? Something like:

1. Retrieve remote repository references from webhook requests
2. Support ephemeral hosting for doc builds
3. Add preference for enabling/disabling per-PR doc builds (disabled by default at first)
4. Create an ephemeral doc build for each appropriate PR webhook request when the setting is enabled
5. Return a build outcome status to the PR, linking to the generated ephemeral build.

[humitos]

Another thing to consider is probably excluding the PRs from the search engines since we don't want people to arrive to these "in progress" docs.

Besides, I think we don't want to build each PR that's generated (otherwise we will end up with many stale docs versions that nobody will read or that will confuse users). We might want to build only those that touches some specific files (under docs/ for example).

Also, we don't want to show these versions in the flyout. Maybe a is_pr field?

[stsewd]

We have this feature in beta https://blog.readthedocs.com/building-docs-for-pull-requests/

[saadmk11]

Closing as this feature has been implemented. :)