Build documentation PR previews only if files under a path have changed

[choldgraf]

It would be helpful if we could optionally configure a path to our documentation, and have ReadTheDocs only build previews of the docs if files underneath that path were changed. I'd imagine a configuration box in the Admin panel that was something like

Build PR previews only for changes under folder: docs/

For documentation builds that take a while to build, this would reduce the time for PRs that only change code, but no docs, which is probably quite a lot of PRs! I think that it would also reduce the load on ReadTheDocs build servers, since a documentation change is probably more rare than a code change.

(note: I couldn't find a "feature request" issue template, which makes me wonder if this is not a good place to open suggestions like this, I am happy to direct this issue elsewhere if that's better)

[stsewd]

This is kind of related to #6046 and #871. This information is already present in the payload from github

https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads

[astrojuanlu]

Doesn't this open the door to tricky corner cases? I'm thinking:

1. I generate API docs using some solution (Sphinx own autodoc, autosummary, sphinx-autoapi)
2. I change some code, including function signatures or docstrings
3. The build doesn't trigger on RTD because nothing changed under docs/, but actually it should have changed

I admit that some (how many?) documentation projects don't document code though.

[humitos]

Thanks, @choldgraf for opening this issue.

We have discussed this already in different opportunities but we never implemented it. There are different reasons for that. Mainly, I think it was because it was too complex to it right and handle all the edge cases (as @astrojuanlu mentions) and users would be pretty confused.

I do see this as a benefit for Read the Docs itself, but I'm not seeing it clearly as a feature from a user perspective. I imagine that people would fight more with this feature (because of these edge cases) than the benefits the feature could provide to them.

I'd 👍🏼 on implementing something like this if we can communicate very clearly how it works, making it an "Advanced feature", reducing the edge cases as much as we can, and explaining the risks as much as we can.

That said, I think that considering that it would only work on PRs it won't be a huge problem if for some particular reason there is one case where the documentation doesn't get updated while working on the PR. The final release/build/version will always have the updated/correct version.

Besides, even when there are files modified under a different directory than the one specified by the user (docs/) we could still create the build in the database and show it in the Builds page with a message "Not executed due to no files modified under docs/" (it could include the list of modified files) and make the "Rebuild this build" link/button to force a full re-build of that PR in case the user suspects that something is not building properly. I think this will cover all the edge cases.

[astrojuanlu]

xref: https://twitter.com/drsarahlgibson/status/1451122665708720130

[sgibson91]

xref: twitter.com/drsarahlgibson/status/1451122665708720130

I actually think @choldgraf opened this issue after a discussion with me, I didn't realise he had! Super great to see this is on your radar though and I hope a solution you are comfortable releasing can be reached, but understandable if it's too complex :)

[astrojuanlu]

I think we could set reasonable expectations like

• Tracking the docs/ directory doesn't mean that the documentation can't change by other means. This is true for source code docstrings but also dependencies, external data, and possibly other factors.
• This would only apply for pull request builds, and there would always be a escape hatch to force a build
• Normal versions wouldn't have this feature and always build on commit

Does this sound like a useful thing?

[humitos]

Hi folks! We worked in a proposal for this at #9649. Basically, the UX for this feature is: "if you write a custom command that exists with code 439, your build will be skipped". I added an example on that PR using git to check for changes under the docs/ directory and skip the build if there weren't changes. Let me know you feedback on this 🙏🏼

[humitos]

There are different reasons for that. Mainly, I think it was because it was too complex to it right and handle all the edge cases (as @astrojuanlu mentions) and users would be pretty confused.

With the proposed implementation, we are delivering this responsibility to the user itself. The user is responsible to guarantee the check is correct for their own scenario. As it's using just an exit code from a custom command, it's pretty flexible to cover most of the use cases and people can write just a simple command or a complex Python script.