Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Schedule does not seems to be respected #3059

Closed
erwan-joly opened this issue Feb 1, 2021 · 47 comments
Closed

Schedule does not seems to be respected #3059

erwan-joly opened this issue Feb 1, 2021 · 47 comments
Labels
service 💁 Relates to Dependabot features GitHub provides T: bug 🐞 Something isn't working

Comments

@erwan-joly
Copy link

Hi, I'm trying to use the schedule parameter with time: "18:00" and timezone: "Pacific/Auckland" but it seems like dependabot doesn't respect the 18:00 for some reasons ? Am I doing something wrong ?

updates:
- directory: "/"
  package-ecosystem: nuget
  schedule:
    interval: daily
    time: "18:00"
    timezone: "Pacific/Auckland"
  open-pull-requests-limit: 100
@erwan-joly erwan-joly added the T: bug 🐞 Something isn't working label Feb 1, 2021
@jurre
Copy link
Member

jurre commented Feb 2, 2021

The schedule is ~best effort, and it's roughly when we start update jobs, it might take a while for PRs to actually be opened. There are also a few other reasons why a PR could be opened, for example a security advisory was published, and when a job has previously failed we also re-run it when a change to the manifest file has been pushed.

So what you're seeing could be explained by one of the above, could you share a bit more details about what you're seeing, and maybe share the repo name so I can check some logs etc?

@erwan-joly
Copy link
Author

Repo is a private (company one)
Those are definitely not security advisory.

The logs in the dependabot change didn't start too far from 6PM (7PM) yesterday 2021-02-02T06:59:06Z

Additionally All the PR got rebased at about 3PM (3hours before schedule) forcing our build pipeline to start building a big amount of open PR (those massive rebase were not in the dependabot changelog).

@jurre
Copy link
Member

jurre commented Feb 4, 2021

Those are definitely not security advisory.

👍

The logs in the dependabot change didn't start too far from 6PM (7PM) yesterday 2021-02-02T06:59:06Z

Without knowing the repo it's hard to check up on this, an hour later than scheduled is not great, but it can happen due to a lot of back pressure. Feel free to open a ticket via support so it's easier to share details if you'd like us to dig deeper!

Additionally All the PR got rebased at about 3PM

By default, PRs get rebased when dependabot detects conflicts, but this can be disabled: https://docs.github.com/en/github/administering-a-repository/configuration-options-for-dependency-updates#rebase-strategy

@MrChocolatine
Copy link

MrChocolatine commented Mar 29, 2021

Since you recommend to search for similar issues before posting one, with my company we have the same issue for one private repository.

We have something like ~30 repositories for Javascript projects, all of them sharing the same configuration (see below) which basically schedules 2 upgrades maximum every week.

But only for one repository Dependabot keeps creating upgrades as soon as there is one available. The initial limit has already been processed long time ago. No matter what we do (recreating the file for instance) Dependabot creates an upgrade, at any time.

The shared configuration
version: 2

updates:

  # Required options
  - package-ecosystem: npm
    directory: "/"
    schedule:
      interval: weekly
      day: saturday
      time: "00:00"
      timezone: Europe/Paris

    # Behaviour of pull requests
    open-pull-requests-limit: 2
    pull-request-branch-name:
      separator: "-"
    commit-message:
      prefix: build
      include: scope

    # Metadata of pull requests
    reviewers:
      - "xxx/yyy"

    # Control which dependencies are updated
    ignore:
      - dependency-name: "ember-cli"

@mueller-ma
Copy link

I have a PR that I can share here: openhab/openhab-android#2590
Opened at 21:38 MEST, but I configured 04:00:

version: 2
updates:
- package-ecosystem: github-actions
  directory: "/"
  schedule:
    interval: daily
    time: "04:00"
    timezone: Europe/Berlin
  open-pull-requests-limit: 99
- package-ecosystem: gradle
  directory: "/"
  schedule:
    interval: daily
    time: "04:00"
    timezone: Europe/Berlin
  open-pull-requests-limit: 99

Since there was another PR one minute earlier and none of them have something about security in their release notes, I don't think that's a security relase: openhab/openhab-android#2588

@guatedude2
Copy link

I'm having the same issue. I setup two repos with the following dependabot.yml configuration:

version: 2
updates:
  - package-ecosystem: "npm"
    directory: "/"
    schedule:
      interval: "weekly"
      day: "monday"
    ignore:
      - dependency-name: "*"
        update-types: ["version-update:semver-patch"]

For some reason, the PRs get opened right after I close any of the 5 open PRs. The schedule isn't respected at all and PRs are just opened whenever one is closed. Any ideas why that may be?

@mueller-ma
Copy link

Maybe because of the default pr limit? Can you try to increase it as in my config above?

@guatedude2
Copy link

Ok I applied the limit, i'll report back with status

@guatedude2
Copy link

Looks like that just kept on popping PRs up, still not respecting the schedule (weekly on Monday's) today is Sunday

@guatedude2
Copy link

I had to close them manually:
Screen Shot 2021-05-30 at 2 54 30 PM

@samsmithnz
Copy link

samsmithnz commented Jun 6, 2021

I have an example too. This should run daily at 6am, but for some reason - hasn't run for the last 2 days - I know there is a package to update too... It is a public repo: https://github.com/samsmithnz/AzurePipelinesToGitHubActionsConverterWeb/blob/main/.github/dependabot.yml

version: 2
updates:
- package-ecosystem: nuget
  directory: "/PipelinesToActions/PipelinesToActions"
  schedule:
    interval: daily
    time: "06:00"
    timezone: America/New_York
  open-pull-requests-limit: 10
  assignees:
  - "samsmithnz"

image

@thabemmz
Copy link

Having the exact same issue as @guatedude2.

Issue

As soon as we merge 1 PR, Dependabot creates a new one, effectively keeping us busy reviewing Dependabot updates all week.

What I'd expect

I'd expect Dependabot to respect the schedule and only offer updates on Sunday, 19:00 with a maximum of 10 Dependabot PR's open at the same time

As a work-around, we have disabled patch and minor updates for now, but this is of course FAR from ideal.

We have a private repository.

Our dependabot.yml:

version: 2
updates:
  - package-ecosystem: npm
    directory: "/"
    schedule:
      day: sunday
      interval: weekly
      time: "19:00"
      timezone: Europe/Amsterdam
    open-pull-requests-limit: 10

@MrChocolatine
Copy link

As soon as we merge 1 PR, Dependabot creates a new one, effectively keeping us busy reviewing Dependabot updates all week.

This ⬆️

@Narnach
Copy link

Narnach commented Jul 21, 2021

The behavior as described by others still persists.

It's almost as if Dependabot has its own internal queue of dependency updates (which is filled on the scheduled interval), but as soon as we merge or close a Dependabot PR, it will create the next one from its queue.

The desired behavior is that it only creates new (non-security update) PRs at the scheduled time, and not creates more PRs out-of-bound after closing previous PRs. The one exception to this would be when Dependabot is newly added to a project, or a new package ecosystem is added to the Dependabot config file.

The reason we use Dependabot is to allocate a roughly fixed amount of time each week to perform updates, to have an upper bound on the amount of time we spend on our maintenance and technical debt each week. Right now the out-of-bound Dependabot PRs clutter our PR list, which adds an annoyance factor to an otherwise awesome product.

@jurre
Copy link
Member

jurre commented Jul 21, 2021

What I think is happening here, is that dependabot is triggered when your manifest files (package.json for example) change, and kicks off a new job. Normally when this happens during the week for a manual change to those files, dependabot will not do anything because the open PR limit is reached, however when this happens when a PR is merged, once that job is finished, Dependabot will notice a "free" slot in that limit, and open a PR.

I get this is annoying and unexpected, I'm just not immediately sure of a good way to fix it. We might be able to disable the behavior of re-running dependabot on a manifest file change when we notice that the person who changed it was Dependabot, but that might be slightly tricky, as a merged PR will show the author as the person that merged 🤔

@Narnach
Copy link

Narnach commented Jul 21, 2021

Thank @jurre, that looks in line with what @asciimike describes in #3975 (comment)

We define the scheduled interval specifically to have one weekly moment where new Dependabot PRs are created. It's on Sunday night for us, so CI can run as long as it needs and we can review dependency PRs on Monday morning. We cherry pick the Dependabot PRs into one new branch/PR and perform some additional work (applying new linting rules, grabbing all @types/* updates if one specific type was suggested by Dependabot, etc). This is mostly due to having strict PR review procedures, so only having one PR review for all dependencies is the most efficient for us.

We thought that we could control the behavior via open-pull-requests-limit, combined with schedule: limit how often Dependabot checks dependencies, and limit how many PRs it creates when it checks.

What you describe re: creating new Dependabot PRs for existing package ecosystems in response to manifest file changes is explicitly an anti-goal for us. Looking at manifest changes to close or update existing PRs is expected behavior. Security updates already follow their own out-of-bounds schedule, so that's no reason for the behavior either.

What's the reasoning for looking at manifest file changes and opening new PRs, in context of there being a schedule and PR limits? If your intention behind the PR limits to have an unlimited "queue" of PRs Dependabot wants to create, but to simply limit the number of PRs which can exist at the same time? In this case maybe we need an additional max-internal-pr-queue-size: <integer> or a can-create-unscheduled-prs: <boolean>, or recheck-on-manifest-change: <boolean> setting to control this behavior so teams can configure this behavior in a way they prefer.

In the PR I mentioned I saw that the team raising the issue switched to Renovate over this. I can tell you that members of our team have mentioned them as an alternative to look at over this exact issue as well. Personally I like to see what we can do to help you solve the issue and stay on Dependabot, so I'm happy to help to try to think of solutions.

@jurre
Copy link
Member

jurre commented Jul 22, 2021

@Narnach could you share the repo this is happening on? I'd like to run some checks

@Narnach
Copy link

Narnach commented Jul 22, 2021

@jurre it's for reisbalans/reisbalans, a private repo

@jurre
Copy link
Member

jurre commented Jul 22, 2021

Thanks!

After digging deeper I think an update is only supposed to happen if the last update failed. The reasons there being that the failure is possibly resolved after a manifest file change.

So this shouldn't happen every time you merge a PR 🤔

I do see a few update jobs that have errors in the logs for that repository, could that explain what's happened? Or does it consistently happen every time even when there are no errors?

If you can share the pull request number for one of the PRs that was created after one of these merges that'd help a lot (so, github.com/org/repo/pulls/<number>)

@Narnach
Copy link

Narnach commented Jul 22, 2021

Dependabot consistently fails the final step of the linting check, because it tries to report annotated feedback to Github. I think Dependabot does not have sufficient rights to annotate our code, hence that job fails (Error trying to create GitHub check for Prettier: Received status code 403).

It works when our team members initiate the check. We figured it was due to Github tightening security measures over the last few months. Because we create a new PR for the actual merge, it's been a low priority to look into it.

The most recent dependency PR we merged was:

It created these PRs in response:

@jurre
Copy link
Member

jurre commented Jul 22, 2021

Dependabot consistently fails the final step of the linting check, because it tries to report annotated feedback to Github. I think Dependabot does not have sufficient rights to annotate our code, hence that job fails (Error trying to create GitHub check for Prettier: Received status code 403).

This is a CI check you're referring to? That should be fine and ignored, we only care about the dependabot itself failing (you should be able to see this on https://github.com/reisbalans/reisbalans/network/updates).

The most recent dependency PR we merged was:

https://github.com/reisbalans/reisbalans/pull/6041

I don't have access to this repo, but in the Dependabot DB I cannot find a reference to that PR, so am I right in assuming this was not created by us? Was the dependabot config file changed in that PR by any chance?

@Narnach
Copy link

Narnach commented Jul 22, 2021

Sorry, I thought you meant CI check logs 😅

The network updates check failures appear to be for hitting the limit on open PRs: package.json has a limit of 5 that's reached, and we've got a secondary Gemfile in a subdirectory for an embedded project which has been set to limit 0 to ignore it (seemed the best way to get Dependabot to ignore a manifest file 🤷 )

PR 6041 was indeed created by me, and is the aggregate dependency update PR for this week. I did update the Dependabot config there to make Dependabot ignore a gem which is pinned on a specific version (and will be phased out rather than updated soon), but the schedule segment did not get changed, so from our perspective a batch of new PRs was not intended or expected 😉

@jurre
Copy link
Member

jurre commented Jul 22, 2021

The network updates check failures appear to be for hitting the limit on open PRs: package.json has a limit of 5 that's reached, and we've got a secondary Gemfile in a subdirectory for an embedded project which has been set to limit 0 to ignore it

That should be fine and not count as an "error" state, that's just Dependabot working as intended and not opening more PRs than what's configured.

I did update the Dependabot config there to make Dependabot ignore a gem which is pinned on a specific version (and will be phased out rather than updated soon), but the schedule segment did not get changed, so from our perspective a batch of new PRs was not intended or expected

Ah, any change to the config file triggers updates immediately. Our assumption is that users will want these changes applied immediately in most cases.

I think that explains what may have happened here, or can you think of other cases where this definitely was not the culprit?

The only thing I can recommend right now is to separate changes to the config file out and apply them before merging/closing any of Dependabot's PRs.

This behavior is documented here fwiw, and I think it makes sense for most users. I suppose we could add an option to disable this, but to be honest I don't think our small team currently has the bandwidth to tackle it any time soon.

@Narnach
Copy link

Narnach commented Jul 22, 2021

Ah, any change to the config file triggers updates immediately. Our assumption is that users will want these changes applied immediately in most cases.

I think that explains what may have happened here, or can you think of other cases where this definitely was not the culprit?

Checking back I see evidence to support this theory (internal PRs):

The only thing I can recommend right now is to separate changes to the config file out and apply them before merging/closing any of Dependabot's PRs.

This behavior is documented here fwiw, and I think it makes sense for most users. I suppose we could add an option to disable this, but to be honest I don't think our small team currently has the bandwidth to tackle it any time soon.

Having an option to disable unscheduled non-security PR creation would probably be the most easily understandable option for users.

I did a quick search in the code, but scheduling does not appear to be part of the open source code, so I don't think I can help out with a PR for this.

I hope your team manages to pick it up at some point. I'll see if our increased understanding of Dependabot's behavior makes it less annoying for us.

Thanks for the informative conversation!

@jurre
Copy link
Member

jurre commented Jul 22, 2021

Thanks @Narnach

Having an option to disable unscheduled non-security PR creation would probably be the most easily understandable option for users.

cc @asciimike, for future reference

I did a quick search in the code, but scheduling does not appear to be part of the open source code, so I don't think I can help out with a PR for this.

Yeah that's right, this is all part of the services we use to run Dependabot at GitHub which are all private.

I hope your team manages to pick it up at some point. I'll see if our increased understanding of Dependabot's behavior makes it less annoying for us.

I hope it does! 💛

Thanks for the informative conversation!

Thanks for helping me figure out what happened here!

@Narnach
Copy link

Narnach commented Jul 26, 2021

@jurre I've just processed this week's dependency updates, but after merging my PR we got 5 new Dependabot PRs. My PR did not contain changes to .github/dependabot.yml. This seems to (at least partially) refute the hypothesis that changing the config file was the only cause of new PRs outside of scheduled times.

My dependencies PR:

After merging the PR, we got these additional PRs from Dependabot:

We did get a PR for an update to a Github Action, so that resulted in some of the workflow files in .github/workflows/*.yml getting updated. Thinking out loud: could it be that you have code somewhere checking for changes to .github/**/*.yml instead of only .github/dependabot.yml? 🤔

@jurre
Copy link
Member

jurre commented Jul 26, 2021

Thinking out loud: could it be that you have code somewhere checking for changes to .github/**/*.yml instead of only .github/dependabot.yml?

I don't think so, the code in question checks against this constant:

CONFIG_FILE_PATHS = %w(.github/dependabot.yml .github/dependabot.yaml).freeze

But thanks for these links, I'll try to find some time to figure out what happened here!

@rafalyesware
Copy link

rafalyesware commented Jul 30, 2021

[Update: at first I thought this was only for the npm ecosystem, but it appears to be unrelated to a specific ecosystem; I've now seen it for both npm and bundler updates]

One side-effect of this I've seen a number of times over the last 2-3 days, is that when dependabot creates a new PR to replace a closed one, sometimes those PRs appear to use stale state. Here's an example (unfortunately in a private repo, but at least illustrative):

Thu Jul 29 19:25:59 2021 +0000, I merge a dependabot PR to "Bump ngx-infinite-scroll from 9.1.0 to 10.0.1".
Thu Jul 29 19:29:55 2021 +0000, dependabot opens a PR to "Bump jasmine-core from 3.7.1 to 3.8.0". Inspecting the diffs (and the git log), this PR is based on the state of master previous to the PR above, and also includes the upgrade to "ngx-infinite-scroll" again.

While this is probably somewhat harmless in many cases, there are a couple of things that I find really annoying about this:

  • Everytime I've looked at one of these PRs that are based off stale state, I have to double-check myself and the state of the repo, wondering why this is also upgrading a package I already had upgraded previously, so those upgrades take up a lot more mental bandwidth.
  • Secondly, it breaks the rule of one-package-upgrade-per-PR, which makes diagnosing what broke our production app that much more difficult. Maybe this can only happen in the case when an upgrade to another package has just been merged, but it makes me lose faith in dependabot when the PR says "Bump X", but the contents are actually "Bump X & Y".
  • Occasionally this also causes conflicts, particularly in package-lock.json, which means the first thing I need to do is have dependabot rebase the PR before I can do anything else with it. This also adds work / decreases efficiency of being able to handle these.

@asciimike
Copy link
Contributor

I've got an ongoing conversation with the PR team about the "Dependabot PRs are in a stale state", though unfortunately it's not moving quickly. It's not specific to Dependabot, but Dependabot users happen to hit it a lot more frequently than the average user. Your comment about rebasing to resolve the "bump x and y" is the right course of action, though I agree it's a pain :/

@hisuwh
Copy link

hisuwh commented Aug 26, 2021

I'm seeing the same as other people and interestingly the same phenomenon as @rafalyesware - hadn't thought anything of this when I noticed it before.

I just closed a PR for ts-node and have had this one created straight away - including the update to ts-node again.

image

This issue makes trying to maintain a WIP limit impossible and turns managing dependabot prs into a game of whack-a-mole.

@MailboxArsonist
Copy link

I'm also having the same issue, has there been any update on a fix for this?

@usmonster
Copy link

All my team's Gradle projects see this same behavior because of consistent Dependabot update errors triggered by #3759. I would love to see the latter fixed to remove such errors, though I think the current immediate-retry-after-error-if-manifest-or-config-file-changed update logic is fine. Otherwise (or in addition), another way to make Dependabot update errors more visible would be welcome.

@jsyrjala
Copy link

I am seeing this same behavior with gradle projects. Schedule is not respected, and often new PRs get created soon after the old ones are merged.

@rubencodes
Copy link

rubencodes commented May 25, 2022

Same issue here for npm updates—new PRs get made right after the last one is closed, completely defeating the purpose of the "schedule" option, it would seem.

@martinvium
Copy link

Same problem, every time I close a PR a new one get's created.

Config looks something like this:

- package-ecosystem: bundler
  directory: "/"
  schedule:
    interval: weekly
    day: wednesday
    time: "06:00"
    timezone: Europe/Copenhagen
  open-pull-requests-limit: 5

@MrChocolatine
Copy link

Incredible how maintainers keep giving us feedback on this incorrect behaviour.

@nbcraft
Copy link

nbcraft commented Sep 13, 2022

Adding to this thread.
We have a monthly schedule to limit our time updating dependencies.
Pretty much experiencing what the other are saying as well:

  • PRs getting opened randomly
  • Merging some dependencies will create a whole bunch of new PRs
  • (Also had case of PR title saying it's bumping to next major version when in reality it's only doing a minor version bump, but that's probably a different topic)
version: 2

updates:
- package-ecosystem: bundler
  insecure-external-code-execution: allow
  directory: "/backend"
  schedule:
    interval: monthly
    time: "07:00"
    timezone: America/Vancouver
  open-pull-requests-limit: 99
  registries:
  - rubygems-server-gems-graphql-pro

@MrChocolatine
Copy link

MrChocolatine commented Jun 25, 2023

Folks, you can also use Renovate instead:
https://github.com/renovatebot/renovate

I feel like Dependabot is becoming an abandonware at this point.

@janhartigan
Copy link

@MrChocolatine thanks for sharing that, I've never seen that before so I'll evaluate it for our team.

Out of curiosity, why do you say Dependabot is becoming abandonware? Obviously this issue is concerning, but wondering if you have some more info.

@MrChocolatine
Copy link

MrChocolatine commented Jun 27, 2023

I may be completely wrong, I am just a simple user.

why do you say Dependabot is becoming abandonware

I am not claiming it is becoming an abandonware, but I feel like.
Given the lack of news, of feedback from the support team, how the product does not seem to evolve over the years...

Again, it's a personal feeling, perhaps I am expecting too much from this service.

@carogalvin
Copy link
Contributor

👋 Hey @MrChocolatine! I'm the PM for Dependabot. I'm sorry to hear that you feel like there's lack of news or feedback. This particular issue hasn't seen much traction because it's not one of our top priorities. But if you look at #1190 or #1736, we're working on some of the top requested features from the community, and actively seeking feedback.

GitHub's a pretty big company so sometimes news about our little bot can get lost in the noise. But if you keep an eye on our tag on the GitHub changelog, you can see all the stuff we're releasing.

I'd be happy to chat further and hear your feedback - feel free to select the time most convenient for you from my calendar: https://calendar.app.google/ZqyPQoHMuT6tsKVX9

@MrChocolatine
Copy link

I am glad to be wrong then, thank you for this message.

@raphael-theriault-swi
Copy link

Would it be possible to at least get a configuration option to ignore changes to manifest files and only trigger either on schedule or when the dependabot config is changed ? As it stands the schedule feature seems like it's pretty much useless

@abdulapopoola
Copy link
Member

@raphael-theriault-swi ; @deivid-rodriguez is working on this and we hope to have this resolved this week :). Thanks for being patient and raising this up.

@deivid-rodriguez
Copy link
Contributor

Hei! We deployed an internal fix for this. Please let us know if you receive any more PRs out of schedule from now on. (Note that when you edit your .dependabot.yml file, that will still retrigger a dependabot run, and that has not changed).

@kazazor
Copy link

kazazor commented Dec 6, 2023

@deivid-rodriguez I;'m still getting weird schedule problems with dependabot:

version: 2
updates:
  - package-ecosystem: "npm"
    directory: "/"
    schedule:
      interval: "monthly"

While configured to monthly, it runs several times a month.

Any input on that?

@deivid-rodriguez
Copy link
Contributor

The only situation when Dependabot runs out of schedule now is when the .dependabot.yml changes are pushed. If you're not experiencing this, I suggest you open a support ticket so that your particular situation can be investigated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
service 💁 Relates to Dependabot features GitHub provides T: bug 🐞 Something isn't working
Projects
None yet
Development

No branches or pull requests