Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dequeue upkeeps using the current dequeue iteration #12852

Closed
wants to merge 6 commits into from

Conversation

ferglor
Copy link
Collaborator

@ferglor ferglor commented Apr 16, 2024

https://smartcontract-it.atlassian.net/browse/AUTO-9357

This PR consists of the following key change:

  • Calculate the number of iterations needed to dequeue the minimum number of logs from the log buffer
    • Use a custom upkeep selector to determine which upkeeps we should dequeue logs for in the current dequeue round

The goal here is that, in cases where the number of upkeeps exceeds the maximum number of results for a single call to log dequeue, we want to dequeue logs for subsets of upkeep IDs, in successive calls to dequeue, until at least the minimum number of logs have been dequeued for all upkeep IDs.

For example, if maxResults is 100, but we store logs for 200 upkeeps; even if we choose to dequeue at most 1 log for each upkeep, we could still only dequeue logs for at most 100 upkeeps in a single dequeue call. This means that in order to dequeue 1 log for all 200 upkeeps; we would need to make 2 calls to dequeue, and for each call, ensure that we dequeue for two unique, distinct sets of upkeep IDs.

To do this determinstically, we can:

  • First, calculate the total number of dequeue calls we would need to make to dequeue for all upkeep IDs. We refer to this number as iterations
  • Add a state variable to the provider, that tracks the current iteration, in the range of (0...iterations), and in each dequeue round, define a custom upkeep selector such that upkeepID % iterations == currentIteration
  • At the end of each dequeue round, increment the currentIterations counter, so that on the next call to dequeue, the upkeep selector will cause a different subset of upkeep IDs to be selected

Testing

Node upgrade test is passing here: https://github.com/smartcontractkit/chainlink/actions/runs/8871312521
Load test run here: https://github.com/smartcontractkit/chainlink/actions/runs/8989012623/job/24691139888

Additionally, some unit tests have been added that demonstrate the behaviour when the buffer has logs for a small number of upkeeps, a large number of upkeeps, and a large number of upkeeps that increases midway through the dequeue rounds.

@ferglor ferglor requested a review from a team as a code owner April 16, 2024 22:06
@cl-sonarqube-production
Copy link

Comment on lines 299 to 304
p.calculateIterations = true
}

if p.calculateIterations {
p.calculateIterations = false
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit. can potentially be deleted

@cl-sonarqube-production
Copy link

@ferglor ferglor closed this Jul 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants