Improve identity provider timeout polling #12120
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
One of our tests for OpenShift uses a manual remediation to install an identity provider before rescanning the environment. Something we've noticed is that the remediation will timeout in e2e runs because the authentication operator isn't ready yet after configuring the new identity provider. The default timeout is only 30 seconds, which likely isn't long enough to restart the authentication operator. We can make this remediation more robust by using `oc adm wait-for-stable-cluster`, which waits up to an hour for the authentication operator to come up. It also reduces the number of things we need to check for by encapsulating the checks into a single command.
|
/test |
|
@rhmdnd: The
Use
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/test e2e-aws-ocp4-pci-dss |
|
Start a new ephemeral environment with changes proposed in this pull request: Fedora Environment Oracle Linux 8 Environment |
|
🤖 A k8s content image for this PR is available at: Click here to see how to deploy itIf you alread have Compliance Operator deployed: Otherwise deploy the content and operator together by checking out ComplianceAsCode/compliance-operator and: |
|
Code Climate has analyzed commit 5936a3c and detected 0 issues on this pull request. The test coverage on the diff in this pull request is 100.0% (50% is the threshold). This pull request will bring the total coverage in the repository to 59.4% (0.0% change). View more on Code Climate. |
|
@rhmdnd: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
yuumasato
approved these changes
Jul 15, 2024
There was a problem hiding this comment.
/lgtm
The test timed out, but this time on ocp4e2e:
https://github.com/ComplianceAsCode/ocp4e2e/blob/main/helpers.go#L56
Maybe we should extend it to 1h to align with oc adm wait-for-stable-cluster.
But I wonder if it is reasonable that it takes more than 30 minutes to get a stable cluster.
I'd be surprised if it took more than an hour for the idp configs to apply consistently. Let's see if we keep experiencing a timeout at 30 minutes and reassess before we bump it up to an hour. Maybe we can find a more efficient/stable way to apply the remediation. |
rhmdnd
added a commit
to rhmdnd/content
that referenced
this pull request
Jul 26, 2024
Lately, we've been experiencing issues with manual remediations timing
out during functional testing. This manifests in the following error:
=== RUN TestE2e/Apply_manual_remediations
<snip>
helpers.go:1225: Running manual remediation '/tmp/content-3345141771/applications/openshift/networking/default_ingress_ca_replaced/tests/ocp4/e2e-remediation.sh'
helpers.go:1225: Running manual remediation '/tmp/content-3345141771/applications/openshift/general/file_integrity_notification_enabled/tests/ocp4/e2e-remediation.sh'
helpers.go:1231: Command '/tmp/content-3345141771/applications/openshift/authentication/idp_is_configured/tests/ocp4/e2e-remediation.sh' timed out
In this particular case, it looks like the remediation to add an
Identity Provider to the cluster failed, but this is actually an
unintended side-effect of another change that updated the
idp_is_configured remediation to use a more robust technique for
determining if the cluster applied the remediation successfully:
ComplianceAsCode#12120
ComplianceAsCode#12184
Because we updated the remediation to use `oc adm
wait-for-stable-cluster`, we're effectively checking all cluster
operators to ensure they're healthy.
This started causing timeouts because a separate, unrelated remediation
was also getting applied in our testing that updated the default CA, but
didn't include a ConfigMap that contained the CA bundle. As a result,
one of the operators didn't come up because it was looking for a
ConfigMap that didn't exist. The `oc adm wait-for-stable-cluster`
command was hanging on a legitimate issue in a separate remediation.
This commit attempts to fix that issue by updating the trusted CA
remediation by generating a certificate for testing purposes, then
creates a ConfigMap called `trusted-ca-bundle`, before updating the
trusted CA.
rhmdnd
added a commit
to rhmdnd/content
that referenced
this pull request
Jul 26, 2024
Lately, we've been experiencing issues with manual remediations timing
out during functional testing. This manifests in the following error:
=== RUN TestE2e/Apply_manual_remediations
<snip>
helpers.go:1225: Running manual remediation '/tmp/content-3345141771/applications/openshift/networking/default_ingress_ca_replaced/tests/ocp4/e2e-remediation.sh'
helpers.go:1225: Running manual remediation '/tmp/content-3345141771/applications/openshift/general/file_integrity_notification_enabled/tests/ocp4/e2e-remediation.sh'
helpers.go:1231: Command '/tmp/content-3345141771/applications/openshift/authentication/idp_is_configured/tests/ocp4/e2e-remediation.sh' timed out
In this particular case, it looks like the remediation to add an
Identity Provider to the cluster failed, but this is actually an
unintended side-effect of another change that updated the
idp_is_configured remediation to use a more robust technique for
determining if the cluster applied the remediation successfully:
ComplianceAsCode#12120
ComplianceAsCode#12184
Because we updated the remediation to use `oc adm
wait-for-stable-cluster`, we're effectively checking all cluster
operators to ensure they're healthy.
This started causing timeouts because a separate, unrelated remediation
was also getting applied in our testing that updated the default CA, but
didn't include a ConfigMap that contained the CA bundle. As a result,
one of the operators didn't come up because it was looking for a
ConfigMap that didn't exist. The `oc adm wait-for-stable-cluster`
command was hanging on a legitimate issue in a separate remediation.
This commit attempts to fix that issue by updating the trusted CA
remediation by generating a certificate for testing purposes.
rhmdnd
added a commit
to rhmdnd/content
that referenced
this pull request
Jul 30, 2024
Lately, we've been experiencing issues with manual remediations timing
out during functional testing. This manifests in the following error:
=== RUN TestE2e/Apply_manual_remediations
<snip>
helpers.go:1225: Running manual remediation '/tmp/content-3345141771/applications/openshift/networking/default_ingress_ca_replaced/tests/ocp4/e2e-remediation.sh'
helpers.go:1225: Running manual remediation '/tmp/content-3345141771/applications/openshift/general/file_integrity_notification_enabled/tests/ocp4/e2e-remediation.sh'
helpers.go:1231: Command '/tmp/content-3345141771/applications/openshift/authentication/idp_is_configured/tests/ocp4/e2e-remediation.sh' timed out
In this particular case, it looks like the remediation to add an
Identity Provider to the cluster failed, but this is actually an
unintended side-effect of another change that updated the
idp_is_configured remediation to use a more robust technique for
determining if the cluster applied the remediation successfully:
ComplianceAsCode#12120
ComplianceAsCode#12184
Because we updated the remediation to use `oc adm
wait-for-stable-cluster`, we're effectively checking all cluster
operators to ensure they're healthy.
This started causing timeouts because a separate, unrelated remediation
was also getting applied in our testing that updated the default CA, but
didn't include a ConfigMap that contained the CA bundle. As a result,
one of the operators didn't come up because it was looking for a
ConfigMap that didn't exist. The `oc adm wait-for-stable-cluster`
command was hanging on a legitimate issue in a separate remediation.
This commit attempts to fix that issue by updating the trusted CA
remediation by creating a configmap for the expected certificate bundle.
rhmdnd
added a commit
to rhmdnd/content
that referenced
this pull request
Jul 31, 2024
Lately, we've been experiencing issues with manual remediations timing
out during functional testing. This manifests in the following error:
=== RUN TestE2e/Apply_manual_remediations
<snip>
helpers.go:1225: Running manual remediation '/tmp/content-3345141771/applications/openshift/networking/default_ingress_ca_replaced/tests/ocp4/e2e-remediation.sh'
helpers.go:1225: Running manual remediation '/tmp/content-3345141771/applications/openshift/general/file_integrity_notification_enabled/tests/ocp4/e2e-remediation.sh'
helpers.go:1231: Command '/tmp/content-3345141771/applications/openshift/authentication/idp_is_configured/tests/ocp4/e2e-remediation.sh' timed out
In this particular case, it looks like the remediation to add an
Identity Provider to the cluster failed, but this is actually an
unintended side-effect of another change that updated the
idp_is_configured remediation to use a more robust technique for
determining if the cluster applied the remediation successfully:
ComplianceAsCode#12120
ComplianceAsCode#12184
Because we updated the remediation to use `oc adm
wait-for-stable-cluster`, we're effectively checking all cluster
operators to ensure they're healthy.
This started causing timeouts because a separate, unrelated remediation
was also getting applied in our testing that updated the default CA, but
didn't include a ConfigMap that contained the CA bundle. As a result,
one of the operators didn't come up because it was looking for a
ConfigMap that didn't exist. The `oc adm wait-for-stable-cluster`
command was hanging on a legitimate issue in a separate remediation.
This commit attempts to fix that issue by updating the trusted CA
remediation by creating a configmap for the expected certificate bundle.
vojtapolasek
pushed a commit
to vojtapolasek/content
that referenced
this pull request
Aug 1, 2024
Lately, we've been experiencing issues with manual remediations timing
out during functional testing. This manifests in the following error:
=== RUN TestE2e/Apply_manual_remediations
<snip>
helpers.go:1225: Running manual remediation '/tmp/content-3345141771/applications/openshift/networking/default_ingress_ca_replaced/tests/ocp4/e2e-remediation.sh'
helpers.go:1225: Running manual remediation '/tmp/content-3345141771/applications/openshift/general/file_integrity_notification_enabled/tests/ocp4/e2e-remediation.sh'
helpers.go:1231: Command '/tmp/content-3345141771/applications/openshift/authentication/idp_is_configured/tests/ocp4/e2e-remediation.sh' timed out
In this particular case, it looks like the remediation to add an
Identity Provider to the cluster failed, but this is actually an
unintended side-effect of another change that updated the
idp_is_configured remediation to use a more robust technique for
determining if the cluster applied the remediation successfully:
ComplianceAsCode#12120
ComplianceAsCode#12184
Because we updated the remediation to use `oc adm
wait-for-stable-cluster`, we're effectively checking all cluster
operators to ensure they're healthy.
This started causing timeouts because a separate, unrelated remediation
was also getting applied in our testing that updated the default CA, but
didn't include a ConfigMap that contained the CA bundle. As a result,
one of the operators didn't come up because it was looking for a
ConfigMap that didn't exist. The `oc adm wait-for-stable-cluster`
command was hanging on a legitimate issue in a separate remediation.
This commit attempts to fix that issue by updating the trusted CA
remediation by creating a configmap for the expected certificate bundle.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
One of our tests for OpenShift uses a manual remediation to install an
identity provider before rescanning the environment. Something we've
noticed is that the remediation will timeout in e2e runs because the
authentication operator isn't ready yet after configuring the new
identity provider.
The default timeout is only 30 seconds, which likely isn't long enough
to restart the authentication operator.
We can make this remediation more robust by using
oc adm wait-for-stable-cluster, which waits up to an hour for theauthentication operator to come up. It also reduces the number of things
we need to check for by encapsulating the checks into a single command.