Allow applying entries on durable remote quorum #37
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Currently, a node will only emit entries for application that are durably in the log locally. To provide high availability in the presence of grey failures such as disk stalls, it is desirable to be able to apply entries that are not durable locally but are nevertheless known to have been persisted on a quorum. This PR prototypes such a mechanism which is opt-in and only available in conjunction with with async storage writes. Applying entries that aren't in the local log requires some care: - if snapshots are sent based on the applied state, the leader may have to catch up the follower from its unstable storage. This is fine. If the leader crashes, the next leader will definitely have a durable log that will catch up the snapshot recipient thanks to the log completeness property. (i.e. the next leader is from the quorum). - orchestrating log truncation via a side effect of applying an entry is more complicated since the truncation may now be carried out on a portion of the log that will only be written in the future. I haven't thought much about other problems that may occur, since at present I'm sending this mostly for prototyping and discussion, not with an intention to merge.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, a node will only emit entries for application that are
durably in the log locally.
To provide high availability in the presence of grey failures such as
disk stalls, it is desirable to be able to apply entries that are not
durable locally but are nevertheless known to have been persisted on a
quorum.
This PR prototypes such a mechanism which is opt-in and only available
in conjunction with with async storage writes.
Applying entries that aren't in the local log requires some care:
to catch up the follower from its unstable storage. This is fine. If
the leader crashes, the next leader will definitely have a durable log
that will catch up the snapshot recipient thanks to the log completeness
property. (i.e. the next leader is from the quorum).
more complicated since the truncation may now be carried out on a
portion of the log that will only be written in the future.
I haven't thought much about other problems that may occur, since at
present I'm sending this mostly for prototyping and discussion, not
with an intention to merge.