Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix] [broker] maintain last active info in memory only. #22794

Merged
merged 5 commits into from
Jun 3, 2024

Conversation

thetumbled
Copy link
Member

@thetumbled thetumbled commented May 29, 2024

Motivation

PR #17573 has fixed cases that consumer commit offset periodically. It update lastActive field of cursor when consumer commit the offset.
But there is still corner case that the consumer do not commit the offset for any reason. For example, there is no content in the topic could be read, so the consumer do not commit the offset reasonably. In such case we has no reason delete the subscription.
Anyway, we should be cautious with the delete operation, which could result into data duplication or loss.

This is the second solution trying to solve problem described in #21692.
PR #21692 try to update the last active info every time broker do expiration check.
But strictly speaking, we can't fix all the cases. If we fail to persist the cursor info, the incorrect sub deletion occurs too.
Incorrect deletion of subscription causes serious consequence, such as duplication or data lost.
In my opinion, missing deletion are better than incorrect deletion, so this pr try to avoid all the incorrect deletion even there is posibility of missing deletion.

Modifications

  • Do not recover last active info from cursor info in zookeeper.
  • Do not store the last active info into zk.

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (10MB)
  • Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository: thetumbled#54

@thetumbled
Copy link
Member Author

Copy link
Member

@nodece nodece left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good work!

BTW, in your test, after the consumer is closed, and then loading the topic, you only need to verify the sub1.getCursor().getLastActive() greater than the start time of topic loading.

Copy link
Member

@dao-jun dao-jun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just a minor comment

@thetumbled
Copy link
Member Author

/pulsarbot rerun-failure-checks

@poorbarcode poorbarcode merged commit 2c3909c into apache:master Jun 3, 2024
50 checks passed
lhotari pushed a commit that referenced this pull request Jun 4, 2024
nodece pushed a commit to ascentstream/pulsar that referenced this pull request Sep 4, 2024
(cherry picked from commit 2c3909c)
Signed-off-by: Zixuan Liu <nodeces@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants