Allow users to disable version validation to downgrade clusters (#5272)

Adds support for a new annotation called k8s.eck.elastic.co/disable-downgrade-validation as a troubleshooting tool to allow users that accidentally upgraded to the incorrect version to correct this mistake and go back to the previous version as long as Elasticsearch has not booted up and converted the data directory. This can be the case if the user upgraded to a version of Elasticsearch that does not exist or that refuses to boot because it is not on the upgrade path from the current version.

docs/operating-eck/troubleshooting/common-problems.asciidoc

@@ -214,3 +214,10 @@ elastic-cloud-eck.v1.4.0       Elasticsearch (ECK) Operator   1.4.0       elasti
 If you are using one of the affected versions of OLM and upgrading OLM to a newer version is not possible then ECK
 can still be upgraded by uninstalling and reinstalling it. This can be done by removing the `Subscription` and both `ClusterServiceVersion` resources and adding them again.
 On OpenShift the same workaround can be performed in the UI by clicking on "Uninstall Operator" and then reinstalling it through OperatorHub.
+
+[id="{p}-{page_id}-version-downgrade"]
+== If you upgraded Elasticsearch to the wrong version
+If you accidentally upgrade one of your Elasticsearch clusters to a version that does not exist or a version to which a direct upgrade is not possible from your currently deployed version, a validation will prevent you from going back to the previous version.
+The reason for this validation is that ECK will not allow downgrades as this is not supported by Elasticsearch and once the data directory of Elasticsearch has been upgraded there is no way back to the old version without a link:https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html[snapshot restore].
+
+The two scenarios described above however are exceptions because Elasticsearch never started up successfully. If you annotate the Elasticsearch resource with `k8s.eck.elastic.co/disable-downgrade-validation=true` ECK will allow you to go back to the old version at your own risk. Please remove the annotation afterwards to prevent accidental downgrades and reduced availability.

pkg/apis/elasticsearch/v1/elasticsearch_types.go

@@ -25,6 +25,8 @@ const (
 	// SuspendAnnotation allows users to annotate the Elasticsearch resource with the names of Pods they want to suspend
 	// for debugging purposes.
 	SuspendAnnotation = "eck.k8s.elastic.co/suspend"
+	// DisableDowngradeValidationAnnotation allows circumventing downgrade/upgrade checks.
+	DisableDowngradeValidationAnnotation = "eck.k8s.elastic.co/disable-downgrade-validation"
 	// Kind is inferred from the struct name using reflection in SchemeBuilder.Register()
 	// we duplicate it as a constant here for practical purposes.
 	Kind = "Elasticsearch"
@@ -481,6 +483,12 @@ func (es Elasticsearch) IsMarkedForDeletion() bool {
 	return !es.DeletionTimestamp.IsZero()
 }
 
+// IsConfiguredToAllowDowngrades returns true if the DisableDowngradeValidation annotation is set to the value of true.
+func (es Elasticsearch) IsConfiguredToAllowDowngrades() bool {
+	val, exists := es.Annotations[DisableDowngradeValidationAnnotation]
+	return exists && val == "true"
+}
+
 func (es *Elasticsearch) ServiceAccountName() string {
 	return es.Spec.ServiceAccountName
 }

pkg/controller/elasticsearch/driver/driver.go

@@ -188,8 +188,12 @@ func (d *defaultDriver) Reconcile(ctx context.Context) *reconciler.Results {
 	// always update the elasticsearch state bits
 	d.ReconcileState.UpdateElasticsearchState(*resourcesState, observedState())
 
+	allowDownscales := d.ES.IsConfiguredToAllowDowngrades()
 	if err := d.verifySupportsExistingPods(resourcesState.CurrentPods); err != nil {
-		return results.WithError(err)
+		if !allowDownscales {
+			return results.WithError(err)
+		}
+		log.Info("Allowing downgrade on user request", "warning", err.Error())
 	}
 
 	// TODO: support user-supplied certificate (non-ca)

pkg/controller/elasticsearch/validation/validations.go

@@ -252,6 +252,12 @@ func checkNodeSetNameUniqueness(es esv1.Elasticsearch) field.ErrorList {
 
 func noDowngrades(current, proposed esv1.Elasticsearch) field.ErrorList {
 	var errs field.ErrorList
+
+	// allow disabling version validation
+	if proposed.IsConfiguredToAllowDowngrades() {
+		return errs
+	}
+
 	currentVer, err := version.Parse(current.Spec.Version)
 	if err != nil {
 		// this should not happen, since this is the already persisted version