[FLINK-35126] Rework default checkpoint progress check window #850
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mateczagany
reviewed
Jul 5, 2024
...ator/src/main/java/org/apache/flink/kubernetes/operator/observer/ClusterHealthEvaluator.java
Show resolved
Hide resolved
gaborgsomogyi
approved these changes
Jul 14, 2024
| <td>Duration</td> | ||
| <td>If no checkpoints are completed within the defined time window, the job is considered unhealthy. This must be bigger than checkpointing interval.</td> | ||
| <td>If no checkpoints are completed within the defined time window, the job is considered unhealthy. The minimum window size is `max(checkpointingInterval, checkpointTimeout) * (tolerableCheckpointFailures + 2)`, which also serves as the default value when checkpointing is enabled. For example with checkpoint interval 10 minutes and 0 tolerable failures, the default progress check window will be 20 minutes.</td> |
There was a problem hiding this comment.
Cool, that we added a description. This would be a hidden gem otherwise.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
Currently the checkpoint progress health check window is configurable by Duration. This makes it hard to enable by default as the sensible interval depends on the checkpoint interval.
At the same time the operator already contains logic for a minimum progress check interval computed from the checkpoint timeout , tolerable failures and checkpoint interval.
Furthermore for any job with checkpointing enabled this health check is very valuable to have enabled by default similar to the restart health check. This PR also proposes to enable this feature by default with the the minimum checkpoint check interval set.
Brief change log
Verifying this change
Unit tests + manual verification
Does this pull request potentially affect one of the following parts:
CustomResourceDescriptors: noDocumentation