WIP: Re-enable Dependabot security updates

[ChrisBAshton]

GitHub automatically opts all qualifying repos in for security
updates, but this gets overwritten by the presence of a
dependabot.yml file, which we have in most repos. We’ll
therefore need to edit the Dependabot config to fix security
related PRs not being raised.

It’s not immediately clear from the docs exactly what’s needed
to opt in, but we believe our use of an allow-list may be the
issue.

We get some value from configuring Dependabot to raise fewer PRs
(i.e. only update the dependencies we care about), but we need
security updates for all dependencies, so this PR is an
attempt to arrive at a configuration that allows security PRs but that
doesn't break our existing behaviour.

See https://docs.github.com/en/code-security/dependabot/dependabot-security-updates/configuring-dependabot-security-updates#overriding-the-default-behavior-with-a-configuration-file

Trello: https://trello.com/c/nqliwwxV/2952-investigate-how-to-fix-dependabot-not-raising-security-prs

⚠️ This repo is Continuously Deployed: make sure you follow the guidance ⚠️

[ChrisBAshton]

Looks like there's no way to validate this config until it's on main 😢
dependabot/dependabot-core#4605

Unless I change the repo's default branch to reenable-security-updates? That way I can avoid tarnishing the commit history of main.

I'll do that now, and change it back after trying a Dependabot run.

[ChrisBAshton]

Yeah, that's not allowed...

Error message: The property '#/updates/1' is a duplicate. Update configs must have a unique combination of 'package-ecosystem', 'directory', and 'target-branch'

[ChrisBAshton]

Nice - so we do need to manually create the branch first.

[ChrisBAshton]

Triggered a Dependabot scan: https://github.com/alphagov/content-data-admin/network/updates/433990960
Nothing interesting. But I guess that makes sense - we've only configured Dependabot to look at X dependencies, not including Rails anymore, so not surprised it didn't try to raise a Rails PR.

The Dependabot security PRs presumably are raised through some other process, e.g. a nightly scan. I can't see anything to trigger a security scan in https://github.com/alphagov/content-data-admin/security - maybe if we leave it overnight it will do that? Or maybe security PRs are only raised for recently declared vulnerabilities?

[ChrisBAshton]

I'm afraid at this point I may have to call in our good ex colleague, @issyl0! 👋 Long time no speak! I believe you're on the Dependabot team these days...?

In RFC-126, we arrived at a Dependabot configuration that cut down the number of PRs raised, as hoped. An unexpected consequence of that is we've stopped receiving security updates for dependencies unless they're explicitly listed 😬

There was a recent CVE-2022-32224 in Active Record, which GitHub reviewed and categorised as High Severity, but Dependabot didn't open any PRs to bump it to the patched version. Running the Dependabot scanner on the email-alert-api repo, we can see:

updater | INFO <job_417588463> Starting update job for alphagov/email-alert-api
updater | INFO <job_417588463> Checking if activerecord 7.0.3 needs updating
  proxy | 2022/07/13 07:27:50 [016] GET https://rubygems.org:443/api/v1/versions/activerecord.json
  proxy | 2022/07/13 07:27:50 [016] 200 https://rubygems.org:443/api/v1/versions/activerecord.json
updater | INFO <job_417588463> Latest version is 7.0.3.1
updater | INFO <job_417588463> Dependabot cannot update to the required version as all versions were ignored for activerecord
updater | INFO <job_417588463> Finished job processing
updater | INFO Results:
updater | Dependabot encountered '1' error(s) during execution, please check the logs for more details.
updater | time="2022-07-13T07:27:50Z" level=info msg="task complete" container_id=job-417588463-updater exit_code=0 job_id=417588463 step=updater

Dependabot did eventually raise a PR bumping Rails, as that is allowed in the config. We ended up manually raising a number of PRs to bump ActiveRecord in all of our repositories.

Is there a way of enabling security-related updates for all dependencies & subdependencies, whilst still retaining the opinionated configs we have for general version updates? I was really hoping for a allow-all-security-updates: true option or something!

I've found https://docs.github.com/en/code-security/dependabot/dependabot-security-updates/configuring-dependabot-security-updates, but that's dedicated mostly to enabling security updates in the repo settings (which we've done). The open-pull-requests-limit 'hack' isn't suitable for us as it doesn't seem to allow any non-security related PRs. I did try (in this PR) seeing if I could mix and match multiple definitions for the same package manager, but that doesn't seem to be allowed.

Any suggestions would be gratefully received. 🙏 Thanks in advance!

[issyl0]

Hello!

I now work on the Code Scanning functionality, but luckily for you the wider Advanced Security group that I'm part of does include Dependabot and some very kind colleagues helped me out here. 🙇🏻

It would appear that your use of allow vs. ignore in dependabot.yml is causing you not to get security updates for dependencies outside of your allowlist. The docs on allow say that "allow" applies to both version and security updates, whereas the docs on ignore state that "ignore" only applies to version updates, as long as you don't additionally specify update-type for each ignored dependency because that will block security updates again. (The docs do read a bit confusingly, but I'm assured that is the behaviour and ignore does currently allow security updates through by default.)

TL;DR: You need to switch to using ignore lists of dependencies to ignore, because ignore only applies to version updates and should let all security updates through.

Thinking back to the size of the Gemfiles and the amount of lesser cared about gems and the maintenance of these lists across your repos vs. the smaller allowlist that you've got so far, I would probably suggest scripting the generation of the ignore lists, such that everything not in the list of n gems that you care about gets ignored. But you do you! ❤️

[ChrisBAshton]

Thanks @issyl0! 🙌