add back liveness and readiness probes to operator container CSV v1.4.0

[shalberd]

Is your feature request related to a problem? Please describe.
In general, containers run more stable under Openshift when liveness and readiness probes are defined.
Nice to see that the container has a prometheus metrics port.

Operator container health checks were not added to the CSV after the move from bundle to deploy/olm-catalog/opendatahub See here for what was added back then when the csvs were still under bundles: https://github.com/opendatahub-io/opendatahub-operator/pull/144/files

The health checks (readiness, liveness) and ports reference is in the developer kustomize manifest https://github.com/opendatahub-io/opendatahub-operator/blob/master/deploy/operator.yaml#L20,

but not in the CSV starting from v1.1.1 https://github.com/opendatahub-io/opendatahub-operator/blob/master/deploy/olm-cata[…]anifests/opendatahub-operator.v1.3.0.clusterserviceversion.yaml

At least regarding the source code, the metrics port 8383 that is used here is still present: https://github.com/opendatahub-io/opendatahub-operator/blob/master/cmd/manager/main.go#L44 (

@VaishnaviHire said:

Hi @shalberd , I can not think of the particular issue, but we were seeing some failures with probes and they were removed for version v1.1.1 .
However I did a quick test of the latest operator image with OCP 4.11 and do not see any errors. Can you create an issue in the operator repo to add the health probes for 1.4.0 ?

Describe the solution you'd like
Add back liveness and readiness probes to the container.
Consider rolling update strategy type: recreate at deployments.spec.strategy

#146 (comment)

When there are just curly brackets, the default, rolling, kicks in, yes. Rolling update uses a readiness probe to check if a new pod is ready, before starting to scale down pods with the old version. -> maybe a good reason to keep the default strategy rolling and to add back the probes.

On the other hand, many examples with liveness and readiness probes in operator CSV context use type: Recreate.
It terminates all the pods and replaces them with the new version. It is often used in examples with operators and probes: https://docs.openshift.com/container-platform/4.9/operators/operator_sdk/osdk-generating-csvs.html#olm-enabling-operator-f[…]rk_osdk-generating-csvs

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[VaishnaviHire]

Probes added to v2.x version of operator