RFE: Support for "SyncWaves" for ApplicationSets

[christianh814]

Support for "SyncWaves" for ApplicationSets

ApplicationSets currently does not have support for "syncwaves" like how "App of Apps" does. This is a RFE to provide this same concept with something that we're calling a "syncset". A "syncset" is conceptually the same as a "syncwave".

The current proposed specification is to match the "syncset" by the Application name that the ApplicationSet generates.

spec:
  generators: {}
  syncset:
    matches: foobar
    weight: 2

The matches will look for an Application name defined that the ApplicationSet generates. In the above example Application "foobar" will have the "syncset" set to "2".

The potential exists for regex as well

spec:
  generators: {}
  syncset:
    matches: ^cluster.*-foobar
    weight: 2

Above example would match cluster1-foobar, cluster2-foobar, cluster3-foobar but NOT cluster4-bazz

Things to note:

• The configuration targets only the Application name field
• If it cannot find a match, it will automatically assign 0 and log it "no match found"
• This is optional. The omission of .spec.sycnwave or an emtpy entry like syncset: {} will result in the ApplicationSet behaving like it does now. AKA "Application Factory"
• ApplicationSet itself, needs to be updated to also check the sync status of the Application. It's already checking if it exits, but now needs to check for sync status.

Flow:

• ApplicationSet will create the Application(s) as it normally does ("factory of Application manifest")
• Once all application manifests are created, it will order them based on the "weight" (lowest number first)
• It will then apply the first Application. It will wait for it to report "synced" before applying the next Application.
• If asyncset isn't defined,as stated above, no sync checks are performed.

Below are example manifests

List Generator

Since cluster1-bgd (which will be generated based on the Generator) isn't listed, it will be deployed with 0

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: bgd
  namespace: openshift-gitops
spec:
  generators:
  - list:
      elements:
      - cluster: cluster1
        url: https://api.cluster1.chx.osecloud.com:6443
      - cluster: cluster2
        url: https://api.cluster2.chx.osecloud.com:6443
      - cluster: cluster3
        url: https://api.cluster3.chx.osecloud.com:6443
  syncset:
  - matches: cluster2-bgd
    weight: 2
  - matches: cluster3-bgd
    weight: 3
  template:
    metadata:
      name: '{{cluster}}-bgd'
    spec:
      project: default
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
      source:
        repoURL: https://github.com/christianh814/gitops-examples
        targetRevision: master
        path: applicationsets/list-generator/overlays/{{cluster}}
      destination:
        server: '{{url}}'
        namespace: bgd

Cluster Generator

All clusters matching bgd=dev with the Application name matching will get 2

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: bgd
  namespace: openshift-gitops
spec:
  generators:
  - clusters:
      selector:
        matchLabels:
          bgd: dev
  syncset:
  - matches: cluster2-bgd
    weight: 2
  template:
    metadata:
      name: '{{name}}-bgd'
    spec:
      project: default
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
      source:
        repoURL: https://github.com/christianh814/gitops-examples
        targetRevision: master
        path: applicationsets/cluster-generator/overlays/dev/
      destination:
        server: '{{server}}'
        namespace: bgd

Git File Generator

Any name that ends with -bgd (from the config.json file) gets 2 (this is a regex example)

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: bgd
  namespace: openshift-gitops
spec:
  generators:
  - git:
      repoURL: https://github.com/christianh814/gitops-examples
      revision: master
      files:
      - path: "applicationsets/git-generator/cluster-config/**/config.json"
  syncset:
  - matches: .*-bgd
    weight: 2
  template:
    metadata:
      name: '{{cluster.name}}-bgd'
    spec:
      project: default
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
      source:
        repoURL: https://github.com/christianh814/gitops-examples
        targetRevision: master
        path: applicationsets/git-generator/overlays/{{cluster.overlay}}
      destination:
        server: '{{cluster.server}}'
        namespace: bgd

Git Directory Generator

Here each Application gets a "weight" assinged that matches the name and will be deployed in order.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: pricelist
  namespace: openshift-gitops
spec:
  generators:
  - git:
      repoURL: https://github.com/christianh814/gitops-examples
      revision: master
      directories:
      - path: applicationsets/git-dir-generator/apps/*
  syncset:
  - matches: pricelist-config
    weight: 1
  - matches: pricelist-db
    weight: 2
  - matches: pricelist-frondend
    weight: 3
  template:
    metadata:
      name: '{{path.basename}}'
    spec:
      project: default
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
      source:
        repoURL: https://github.com/christianh814/gitops-examples
        targetRevision: master
        path: '{{path}}'
      destination:
        server: https://api.cluster1.chx.osecloud.com:6443
        namespace: pricelist

Feedback

This is just a mockup of what it can look like. I'm open for suggestions. The "spirit" of the idea is assign the "syncset" number based on the Application name field.

[maruina]

Hey @christianh814 , we found the same issue and decided to build a controller around that: https://github.com/Skyscanner/argocd-progressive-rollout

You basically define in your CRD how you want to update your generated Applications, and the operator will do the sync for you

It's very alpha stage, we will have something for testing when we complete Milestone 2. Happy to hear any feedback you might have.

[rumstead]

#61 feels very similar to this.

[ghostsquad]

Has any progress been made on this front? I believe this to be a major blocker for adopting ApplicationSet.

[maruina]

Hey @ghostsquad, on our side (github.com/skyscanner/applicationset-progressive-sync) we've been doing some progress but for Q4 2021 we had to pause the project. We hope we will be able to resume it in Q1 2022.

It's blocking us as well but we other priorities got in the way.

[ghostsquad]

@maruina thank you for the update!

[jeffhubLR]

We are using app-of-apps and would like to adopt ApplicationSets but need functionality defined in this RFE. Thanks for documenting this, and it would meet our use-case, which is defined deployment ordering of Applications by leveraging sync-wave annotations on those Applications.

[ghostsquad]

Instead of using regex, could you use the semantics that many of us may already be aware of and used to, that being the K8s Preferred Affinity semantics?

preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 1
        preference:
          matchExpressions:
          - key: another-node-label-key
            operator: In
            values:
            - another-node-label-value

so basically, instead of requiring regex, which is not all that easy to use, instead leverage the labels of the applications, and the label selector functionality?

[christianh814]

Instead of using regex, could you use the semantics that many of us may already be aware of and used to, that being the K8s Preferred Affinity semantics?

preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 1
        preference:
          matchExpressions:
          - key: another-node-label-key
            operator: In
            values:
            - another-node-label-value

so basically, instead of requiring regex, which is not all that easy to use, instead leverage the labels of the applications, and the label selector functionality?

Yes.

Except this should be an "in addition to" and not "instead" 😁

[morningspace]

We have seen the similar issue when using ApplicationSet w/ app-of-apps pattern. Using a top level app to include all ApplicationSets, and all apps will be spawned simultaneously, although some apps need to be synced prior to others. Right now, as a workaround, I have to insert some "check" jobs into these apps to ensure the sync order, which makes it very ugly and invasive to existing apps.

[christianh814]

Related:

argoproj/argo-cd#7437
argoproj/argo-cd#3892

Looks like having the Application resource having a dependsOn feature would help here

[dmeytin]

Any news on this feature?

[christianh814]

@dmeytin It's in Alpha

https://argo-cd.readthedocs.io/en/latest/operator-manual/applicationset/Progressive-Rollouts/

[ron1]

Does this solution offer any sync-waves integration? I'm looking for a sync-wave compliant app-of-appsets capability.

[christianh814]

@ron1 that's what this does

[ron1]

Say I have an app with 10 appsets and those appsets each have 5 apps. Then say the 50 leaf apps are assigned sync-wave numbers 1 through 50. Will the 50 leaf apps then be deployed one-by-one in sync-wave order?

I didn't get the impression that the new progressive rollouts feature worked like I am describing above.

[dmeytin]

And another question is regarding rollbacks.
if applicationsetcontroller.default.application.progressing.timeout is exceeded, will the applications be rollbacked to the previous version using reverse ordering of progressing steps?

[dmeytin]

@christianh814