Kubeflow Tekton Pipelines: Sync manifests (kubeflow#1843)

Sync manifests for application "Kubeflow Tekton Pipelines".
Upstream manifests are copied from:
- Repo: https://github.com/kubeflow/kfp-tekton
- Path: manifests/kustomize
- Revision: v0.8.0-rc0

Signed-off-by: Yannis Zarkadas <yanniszark@arrikto.com>

apps/kfp-tekton/upstream/Makefile

@@ -0,0 +1,29 @@
+# This makefile is a quick test to verify all manifests can be hydrated.
+
+test: aws azure dev gcp platform-agnostic platform-agnostic-multi-user plain plain-multi-user
+
+aws: FORCE
+	kubectl kustomize env/aws
+
+azure: FORCE
+	kubectl kustomize env/azure
+
+dev: FORCE
+	kubectl kustomize env/dev
+
+gcp: FORCE
+	kubectl kustomize env/gcp
+
+platform-agnostic: FORCE
+	kubectl kustomize env/platform-agnostic
+
+platform-agnostic-multi-user: FORCE
+	kustomize build --load_restrictor none env/platform-agnostic-multi-user
+
+plain: FORCE
+	kubectl kustomize env/plain
+
+plain-multi-user: FORCE
+	kustomize build --load_restrictor none env/plain-multi-user
+
+FORCE: ;

apps/kfp-tekton/upstream/README.md

@@ -0,0 +1,158 @@
+# Kubeflow Pipelines Kustomize Manifest Folder
+
+## Install Kubeflow Pipelines
+
+This folder contains Kubeflow Pipelines Kustomize manifests for a light weight
+deployment. You can follow the instruction and deploy Kubeflow Pipelines in an
+existing cluster.
+
+To install Kubeflow Pipelines, you have several options.
+
+- Via an upcoming commandline tool.
+- Via Kubectl with Kustomize, it's detailed here.
+
+### Install via Kustomize
+
+Deploy latest version of Kubeflow Pipelines.
+
+It uses following default settings.
+
+- image: latest released images
+- namespace: kubeflow
+- application name: pipeline
+
+#### Option-1 Install it to any K8s cluster
+
+It's based on in-cluster PersistentVolumeClaim storage.
+
+```bash
+kubectl apply -k cluster-scoped-resources/
+kubectl wait crd/applications.app.k8s.io --for condition=established --timeout=60s
+kubectl apply -k env/platform-agnostic/
+kubectl wait pods -l application-crd-id=kubeflow-pipelines -n kubeflow --for condition=Ready --timeout=1800s
+kubectl port-forward -n kubeflow svc/ml-pipeline-ui 8080:80
+```
+
+Now you can access it via localhost:8080
+
+#### Option-2 Install it to GCP with in-cluster PersistentVolumeClaim storage
+
+It's based on in-cluster PersistentVolumeClaim storage.
+Additionally, it introduced a proxy in GCP to allow user easily access KFP safely.
+
+```bash
+kubectl apply -k cluster-scoped-resources/
+kubectl wait crd/applications.app.k8s.io --for condition=established --timeout=60s
+
+kubectl apply -k env/dev/
+kubectl wait applications/pipeline -n kubeflow --for condition=Ready --timeout=1800s
+
+# Or visit http://console.cloud.google.com/ai-platform/pipelines
+kubectl describe configmap inverse-proxy-config -n kubeflow | grep googleusercontent.com
+```
+
+#### Option-3 Install it to GCP with CloudSQL & GCS-Minio managed storage
+
+Its storage is based on CloudSQL & GCS. It's better than others for production usage.
+
+Please following [sample](sample/README.md) for a customized installation.
+
+#### Option-4 Install it to AWS with S3 and RDS MySQL
+
+Its storage is based on S3 & AWS RDS. It's more natural for AWS users to use this option.
+
+Please following [AWS Instructions](env/aws/README.md) for installation.
+
+Note: Community maintains a repo [e2fyi/kubeflow-aws](https://github.com/e2fyi/kubeflow-aws/tree/master/pipelines) for AWS.
+
+#### Option-5 Install it to IBM Cloud with in-cluster PersistentVolumeClaim storage
+
+It's based on in-cluster PersistentVolumeClaim storage.
+Additionally, it uses the ibm cloud NFS storage with UID support to make sure all pods can run as non-root users.
+
+Please follow the [IKS group ID storage setup](https://www.kubeflow.org/docs/ibm/deploy/install-kubeflow-on-iks/#ibm-cloud-group-id-storage-setup)
+before running the below commands.
+
+```bash
+kubectl apply -k cluster-scoped-resources/
+kubectl wait crd/applications.app.k8s.io --for condition=established --timeout=60s
+kubectl apply -k env/platform-agnostic/
+kubectl wait pods -l application-crd-id=kubeflow-pipelines -n kubeflow --for condition=Ready --timeout=1800s
+kubectl port-forward -n kubeflow svc/ml-pipeline-ui 8080:80
+```
+
+## Uninstall
+
+If the installation is based on CloudSQL/GCS, after the uninstall, the data is still there,
+reinstall a newer version can reuse the data.
+
+```bash
+### 1. namespace scoped
+# Depends on how you installed it:
+kubectl kustomize env/platform-agnostic/ | kubectl delete -f -
+# or
+kubectl kustomize env/dev | kubectl delete -f -
+# or
+kubectl kustomize env/gcp | kubectl delete -f -
+# or
+kubectl delete applications/pipeline -n kubeflow
+
+### 2. cluster scoped
+kubectl delete -k cluster-scoped-resources/
+```
+
+## Troubleshooting
+
+### Permission error installing Kubeflow Pipelines to a cluster
+
+Run
+
+```bash
+kubectl create clusterrolebinding your-binding --clusterrole=cluster-admin --user=[your-user-name]
+```
+
+### Samples requires "user-gcp-sa" secret
+
+If sample code requires a "user-gcp-sa" secret, you could create one by
+
+- First download the GCE VM service account token
+    [Document](https://cloud.google.com/iam/docs/creating-managing-service-account-keys#creating_service_account_keys)
+
+  ```bash
+  gcloud iam service-accounts keys create application_default_credentials.json \
+    --iam-account [SA-NAME]@[PROJECT-ID].iam.gserviceaccount.com
+  ```
+
+- Run
+
+  ```bash
+  kubectl create secret -n [your-namespace] generic user-gcp-sa --from-file=user-gcp-sa.json=application_default_credentials.json`
+  ```
+
+## Folder Structure
+
+### Overview
+
+- User facing manifest entrypoints are `cluster-scoped-resources` package and `env/<env-name>` package.
+  - `cluster-scoped-resources` should collect all cluster-scoped resources.
+  - `env/<env-name>` should collect env specific namespace-scoped resources.
+  - Note, for multi-user envs, they already included cluster-scoped resources.
+- KFP core components live in `base/<component-name>` folders.
+  - If a component requires cluster-scoped resources, it should have a folder inside named `cluster-scoped` with related resources, but note that `base/<component-name>/kustomization.yaml` shouldn't include the `cluster-scoped` folder. `cluster-scoped` folders should be collected by top level `cluster-scoped-resources` folder.
+- KFP core installations are in `base/installs/<install-type>`, they only include the core KFP components, not third party ones.
+- Third party components live in `third-party/<component-name>` folders.
+
+### For direct deployments
+
+Env specific overlays live in `env/<env-name>` folders, they compose above components to get ready for directly deploying.
+
+### For downstream consumers
+
+Please compose `base/installs/<install-type>` and third party dependencies based on your own requirements.
+
+### Rationale
+
+Constraints for namespaced installation we need to comply with (that drove above structure):
+
+- CRDs must be applied separately, because if we apply CRs in the same `kubectl apply` command, the CRD may not have been accepted by k8s api server (e.g. Application CRD).
+- [A Kubeflow 1.0 constraint](https://github.com/kubeflow/pipelines/issues/2884#issuecomment-577158715) is that we should separate cluster scoped resources from namespace scoped resources, because sometimes different roles are required to deploy them. Cluster scoped resources usually need a cluster admin role, while namespaced resources can be deployed by individual teams managing a namespace.

apps/kfp-tekton/upstream/base/application/application.yaml

@@ -0,0 +1,49 @@
+# Note, this application.yaml is not included by default for most environments.
+
+apiVersion: app.k8s.io/v1beta1
+kind: Application
+metadata:
+  name: $(kfp-app-name)
+  annotations:
+    kubernetes-engine.cloud.google.com/icon: >-
+      data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAADMAAAAyCAYAAADx/eOPAAALuUlEQVRogd2afWxd9XnHP99bK4pS3yhiGUIRiiJUVVVqbq8ppdR20ibqpuIMtDRkUYERp29J57gMZVuxsrZiK7oZXVv5re1AiOuoG+N1DMkuytprsGPEVMouxqQZJVHEWIdQlGVxZlmZdb/747zcc869jpMO+seOdPz7nd/L83tev89zzjX8Bq795Rq9o17zXp+Tey+Ijkyboela29DRWkhffyT733pH/Z3este9F2cC6N0kNjxtjD+FdRD8UF9X7u97y7UbQFPAivC0BdllS381slun3s3z3xVhhqeds90tqR/oMB7u68z19ZZra0E/l1if3WOziPx3skrDPTr+bvDxfxImEIJbgX6gGBJ7EfHJX/ySReDHwO9KYAenyWCMFKw21GSeslwa2Z17+TcuzPBRr7B8m6Df5oOJqdPAR/u6cm/2lmv3At+IT3GiXZqbcaxSLsfRoTsvn7XL2jE87ZXGnwf+VGiDY86ETM1wU1+XjvSW/RlgTJADQ2QaCZKWcX1/aDIcjE8i3SdzZLjn0lm8pJXD02417BM+gLmq2Rqjr/d16Vu95dp6wc8Ra5O8NrPIcoZCvIR1H+KZkd2qLcfnRYUZOuorJO+3uQt0RerolGYZR7r5+C9ZATwPviGyQprd6Liszy3bnwVKwGMjPbnFyxJmeNpX2T4gaR/QmmSpyYZTho/2depMb9k/kNh3KawuJ1bWauHzUcyXRpZAv5Zmg7aHBLcmNN9ECAFeAO3s69KZ3nLtDuF9dnBs0IT9JO24rbPb0JfP2syCZpFfE5q1mRWcvlgMNcwMTRq9z/+OWXdx4AGjvX1deqC37DbwPwOrMgsufol5mWMWs1ivEbjTrOCtLNNb+udygqsNbUBtopR/NkuuwTJ6Hxsw67KSuvH5MPDA/nJttfGTdUFCMUlp/ALwOtIs9muBxpnFnBzuSQf21oP/BbXclVvumWuTaDN8WNBm2GizJkxPM0CDMA2WGZ72bbb/Njue2TRj9Il/PcG87SeBz4ZTNaSTsmctHcO8SqDp14d7dCFLZ2v/3OpQ023Ah4n65kohvETUCdcsfmuilD+bpNdgGZvOODuHqYGIVGCec9g7+7o031v2jaBTiD0ysxbHRnZrPktzyz1zK7f0z10nh5pWwLRhvZro1KqznVJhNB8UyDeSsU4zAOiIXV1OuEqQ2AR79nflXgcY6dGLwIvR8q39cy1b+uc2Emo6dI824BpMSxz8iVhy4m/2WiYHdV5UmOHp2mpwm52ESCdwRn+9v0tPAWzpn9sAFAQbMdc60PaHsFZEWd9uxk4z8G3seykECfObTEd2KmuZG4CWyLXkYLMwtiYt+hMsTUdAEZQzjs9apv66SHJRk73ZjBQ+iRu29s+1VEr5OImmXs4MHUahVoLWgK23wbv6OrU4OulcuHYehWsVHhpXwpE2FNRayTszX2cwDpQEzTB+QvrJHCXUaigk+c++aXZiE98YmUVgV19X7u3ypH/fgfUA5h2usY2jNjmWoGVn50nvC9T2NviA5OPBGPW91OlG+0Xa1WJhhqadk3WjpKCilQIQFP19XZocnfIHgIeFWyNh6goXyX6gdNWfU8aJ5tNjEheAHZVS/ruGj0s8k6VPhh6ms6kwgoLl1aGuCEuSpwXfHZ2qrTJ+HHkNCpOjmbdFcEcGUIhUSj/H65rPO6j+766U8i/QXV0z8cqJc4btwF8AtWgtMb1wj+j41Df/s1EYQwdEDiqM3hDes9quGY3IKoYOvCrU7HlCoZtEWapPkzEpsU8uq8b36a6uBqaBv5l45URLpZT/pmGH8LnkvlAdAOt1oeXqRsuYTjlEMJiXvWN/Z+5szfqioKcOKo7qr/nAEesKiOyv2A/q88rOx8+8bPhK5dUTAA8jbUT6MuKnbKteNVHKP23xCeD1LC0F2TWOmzoAKEiWxmC+sr8rN1OerF2HGaqXFcZhDWaYj11S4ZxcXxVqyKqPZOeNTwM7Jkr5BeDPQJ8NFQaoC/gZ26rXT5TyxxAfRx6P94d0gU0pYYama+tsbwix/AHM4fKUrwAeB68kRJ5AZsWWieGTjLipsVCgrKCwKHF7pZQ/RXf104j76i4ZMmquxkzRXb2zUsqfxdxsfCiA70hRjZbpCDHmJcRdeZPDHkVck0Ul5PeHZ81DgHxKtglXaHCxVN9fr5TyR9hW3QA8Amqp5526SyKtBEbZVv1eZeZkbqKU7xfsFJwPqRW29s+11oUxnUhnkHf2dWoB+R5Jv5dNaGHh1wog8d/ZAI+0GgVpFPTp4AfJT2Hup7u6EvMk0tpkboutEz0HMPzHyD+mu3pFpZR/Aug0Pgm0RLkvFzLWYfjDvs7cqfKUt2LuXTLhue5mdWhVDJdEzxDDcRKawceN9lRePVkDfgBcR/LKVqNpz/s08DO6q4VKKT8j8zHgJ1HyzA1P11YZjfV1arw85auBR4RalDB5lEjDKi0CgPPphKZ0QiNRwUQeg88B2ydKreew9yH1NCxe/r4GaZpt1Vsrh/JnDDcBLwPkbLVgf6s86RXYj4KvtJKJM8KsGLkSlsmUL6mSg1RJY1xD7KmU8sfprnYgBqJsGVsiEfupsca7FfMo26p/OfHKiVqllB8HyPV16VxfV66G/G1QBwY5xvCgTT7X3/MTaBbFVr0fJvqw2ASZ+yul/FN0V68CHsesiDl3UopM3CwhDZDD/Dnwj3S/sjoYAMqTtc1YX02jVqYOiuuqsAKIkqZCfFIz/IrfFY8gDrKt2gI8irSuwQezyTeNaOl+6qYb+fpYGKEXJE9GSTObK5ItrheaLHE5/XRKcHul+kYN8x2kzWlLNNuVtUqibzKW5CBjxUoszO7NWrS1E/xWvMeJjck2WQHEKJeMD+qH4gWCSvg00m3AVxv5TMRKsp9Cs0Q/Ka/1BOZQNBSXMz2b9Q5oO9JCKgkqg2aKofl8uvTPeE1w3t5KKf8y26pFxINhLRa5R9JV6huT/aZuFu7Ds+A9jBdj+VIvZz2b9BL2Xi5yJQEgUFqinI9SZBDx358o5Q/HiRGtquOEmxJu6DcbC/afQWxnvHg+Odrwm2bP5txh5OEYjOM3vaiu8qqHJw1mPmK/Xs7HJf0LRncDMF5cAL6NWUxDrX/duwbczljxjSzvTX+gtXU3MBlrRCltrsxBTgorACKrRGf5bczOiVLrhUL74B2F9oHVjBd/iLwTWEhr+CIWaLYumDjIWLHha+aSwvRs1iJmJ9Kb9ZJRETS3ACsMC8i1ZNwgXZDYWTmU/1WhfeAW8Cjo+UL7wDrGik8jfid0kYz/Z2ODepv+GPIY+FAznpcUJhAo9w5mh81CFtEsWieCTzwXkogmfKBSyh8ttA98EDPqoPouYqYLxYEPMVY8itmEeTM+KEaqZhVAkiPPIL6QDPhLFiYQSC9J7M3mGlF/24zWSvwIM1xoH2gF/sFiTcSPxQakqUJxsIPx4jGCr0AzCUYTbROJ7DPAdsbSAX9ZwgDs3qTDiMGUOxF/1DgfekLVsPf0sw8DPARsDNwy8iYBXov4p0L7wC2MF99CfBJ4rqmbJbO/qYE+x1jx5HK8Xtp/aFgHDM/FX+RM9FFjHjjj4NV3HvlPsP4g+SqQgm6zCuvJQnHgi4wVz2JuAj8RnLGEVaCf8Y8cuRQ2L0mYEBB2Gb8ZHKD4NQBx+0Qpf7LQPrAVVGqiiWTpCcEn4QcLxcF7C7+aXMDahT1YX5IS5DHE/ZfC4yULEwr0DtIOWwuuvwZ8rVLKP1soDqzHPGJoyRao9b4SXiQQ30A8eO1/PJ8D7gK+BtQSJcQM8AXGlg747LUkmi91lad8J3CuZ5OeBii0D64ET2FdH1N0omWJvgLPkvwM8LmZf7lrnm3VO4CHsM4DH2P8I8vGSfK67P9q8v9wWPAcQLH4PbBHbK6Pq+3M9+Ml+6FL2dyC+WmhOLiWseKPMDeDd12uIPBrWCZ5Xds++AHsAwGlBKnoB5747c2J+aSJEuvRL8CDv/2Zz+cqh/LL/gPD//vrfwFjcI5oX6jDBwAAAABJRU5ErkJggg==
+    marketplace.cloud.google.com/deploy-info: '{"partner_id": "google-cloud-ai-platform", "product_id": "kubeflow-pipelines", "partner_name": "Google Cloud AI Platform"}'
+spec:
+  addOwnerRef: true
+  selector:
+    matchLabels:
+      application-crd-id: kubeflow-pipelines
+  descriptor:
+    version: $(kfp-app-version)
+    type: Kubeflow Pipelines
+    description: |-
+      Reusable end-to-end ML workflow
+    maintainers:
+      - name: Google Cloud AI Platform
+        url: https://cloud.google.com/ai-platform/
+      - name: Kubeflow Pipelines
+        url: https://github.com/kubeflow/pipelines
+    links:
+      - description: 'Kubeflow Pipelines Documentation'
+        url: https://www.kubeflow.org/docs/pipelines/
+    notes: |-
+      Please go to [Hosted Kubeflow Pipelines Console](https://console.cloud.google.com/ai-platform/pipelines/clusters).
+
+  info:
+    - name: Console
+      value: 'https://console.cloud.google.com/ai-platform/pipelines/clusters'
+  componentKinds:
+    - group: v1
+      kind: ServiceAccount
+    - group: rbac.authorization.k8s.io/v1
+      kind: Role
+    - group: rbac.authorization.k8s.io/v1
+      kind: RoleBinding
+    - group: v1
+      kind: Service
+    - group: v1
+      kind: ConfigMap
+    - group: v1
+      kind: Secret
+    - group: apps/v1
+      kind: Deployment

apps/kfp-tekton/upstream/base/application/kustomization.yaml

@@ -0,0 +1,5 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - application.yaml
+

apps/kfp-tekton/upstream/base/cache-deployer/cache-deployer-deployment.yaml

@@ -0,0 +1,29 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: cache-deployer-deployment
+  labels:
+    app: cache-deployer
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: cache-deployer
+  strategy:
+    type: Recreate
+  template:
+    metadata:
+      labels:
+        app: cache-deployer
+    spec:
+      containers:
+      - name: main
+        image: gcr.io/ml-pipeline/cache-deployer:dummy
+        imagePullPolicy: Always
+        env:
+        - name: NAMESPACE_TO_WATCH
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.namespace
+      serviceAccountName: kubeflow-pipelines-cache-deployer-sa
+      restartPolicy: Always

apps/kfp-tekton/upstream/base/cache-deployer/cache-deployer-role.yaml

@@ -0,0 +1,17 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  labels:
+    app: kubeflow-pipelines-cache-deployer-role
+  name: kubeflow-pipelines-cache-deployer-role
+rules:
+- apiGroups:
+  - ""
+  resources:
+  - secrets
+  verbs:
+  - create
+  - delete
+  - get
+  - patch
+  - list

apps/kfp-tekton/upstream/base/cache-deployer/cache-deployer-rolebinding.yaml

@@ -0,0 +1,11 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: kubeflow-pipelines-cache-deployer-rolebinding
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: kubeflow-pipelines-cache-deployer-role
+subjects:
+- kind: ServiceAccount
+  name: kubeflow-pipelines-cache-deployer-sa

apps/kfp-tekton/upstream/base/cache-deployer/cluster-scoped/cache-deployer-clusterrole.yaml

@@ -0,0 +1,35 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  labels:
+    app: kubeflow-pipelines-cache-deployer-clusterrole
+  name: kubeflow-pipelines-cache-deployer-clusterrole
+rules:
+- apiGroups:
+  - certificates.k8s.io
+  resources:
+  - certificatesigningrequests
+  - certificatesigningrequests/approval
+  verbs:
+  - create
+  - delete
+  - get
+  - update
+- apiGroups:
+  - admissionregistration.k8s.io
+  resources:
+  - mutatingwebhookconfigurations
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+- apiGroups:
+  - certificates.k8s.io
+  resources:
+  - signers
+  resourceNames:
+  - kubernetes.io/*
+  verbs:
+  - approve

apps/kfp-tekton/upstream/base/cache-deployer/cluster-scoped/cache-deployer-clusterrolebinding.yaml

@@ -0,0 +1,12 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: kubeflow-pipelines-cache-deployer-clusterrolebinding
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: kubeflow-pipelines-cache-deployer-clusterrole
+subjects:
+- kind: ServiceAccount
+  name: kubeflow-pipelines-cache-deployer-sa
+# namespace will be added by kustomize automatically according to the namespace field in kustomization.yaml

apps/kfp-tekton/upstream/base/cache-deployer/cluster-scoped/cache-deployer-sa.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: kubeflow-pipelines-cache-deployer-sa

apps/kfp-tekton/upstream/base/cache-deployer/cluster-scoped/kustomization.yaml

@@ -0,0 +1,11 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - cache-deployer-clusterrole.yaml
+  - cache-deployer-clusterrolebinding.yaml
+  # HACK: although a service account(SA) is not a cluster-scoped resource.
+  # Presence of a SA referred by a clusterrolebinding allows kustomize to auto-add
+  # namespace for the clusterrolebinding's SA ref.
+  - cache-deployer-sa.yaml
+

apps/kfp-tekton/upstream/base/cache-deployer/kustomization.yaml

@@ -0,0 +1,13 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+bases:
+  - cluster-scoped
+resources:
+  - cache-deployer-role.yaml
+  - cache-deployer-rolebinding.yaml
+  - cache-deployer-deployment.yaml
+commonLabels:
+  app: cache-deployer
+images:
+  - name: gcr.io/ml-pipeline/cache-deployer
+    newTag: 1.5.0-rc.2