[Merged by Bors] - Secret-aware pod scheduling

[nightkr]

Description

Fixes #57

This requires pod volume definitions to switch from using the csi volume type (which are fully contained within the Kubelet's Pod lifecycle) to the ephemeral volume type (which is reified into a full PersistentVolumeClaim by the scheduler). For example:

---
apiVersion: v1
kind: Pod
metadata:
  name: example-secret-consumer
spec:
  volumes:
    - name: tls
      csi:
        driver: secrets.stackable.tech
        volumeAttributes:
          secrets.stackable.tech/class: tls
          secrets.stackable.tech/scope: node,pod,service=secret-consumer-nginx
  containers:
    - name: ubuntu
      image: ubuntu
      stdin: true
      tty: true
      volumeMounts:
        - name: tls
          mountPath: /tls

should be changed into:

---
apiVersion: v1
kind: Pod
metadata:
  name: example-secret-consumer
spec:
  volumes:
    - name: tls
      ephemeral:
        volumeClaimTemplate:
          metadata:
            annotations:
              secrets.stackable.tech/class: secret
              secrets.stackable.tech/scope: node,pod,service=secret-consumer-nginx
          spec:
            storageClassName: secrets.stackable.tech
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: "1"
  containers:
    - name: ubuntu
      image: ubuntu
      stdin: true
      tty: true
      volumeMounts:
        - name: tls
          mountPath: /tls

csi volumes still work for now, but won't be able to take advantage of secret-aware scheduling. We will probably want to remove it after all operators and documentation have been migrated.

Review Checklist

• Code contains useful comments
• (Integration-)Test cases added (or not applicable)
• Documentation added (or not applicable)
• Changelog updated (or not applicable)
• Cargo.toml only contains references to git tags (not specific commits or branches)
• Helm chart can be installed and deployed operator works (or not applicable)

Once the review is done, comment bors r+ (or bors merge) to merge. Further information

[sbernauer]

LGTM overall % comments

[nightkr]

bors r+

[bors]

Pull request successfully merged into main.

Build succeeded:

• All tests passed