Automatically selecting pacts to verify when requesting pacts for verification

[bethesque]

AKA: Making consumer version selectors unnecessary

Proposal:

• Add "branch" to version resource that gets created when pacts are published (see Supporting explicit branches #382) [implemented in API, but not yet rolled out to the pact clients as of 15 Feb 2021]
• Add "mainDevelopmentBranches" property to pacticipant (eg. ["main"] - question, is there ever likely to be more than one?)
• Support recording of deployments as a replacement for "tag version as prod" (Supporting explicit deployments #384) so that we know exactly which version(s) are in an environment at a given time [in progress]
• When requesting the list of pacts to verify, the Pact Broker can then automatically return (when autoSelectPacts: true)
  • The pacts for the main branch(es) of development
  • The pacts for each of the currently deployed consumer versions (unlike the previous "tag as prod" approach, the "deployments" support will allow the broker to know if there are one/multiple versions of an application in an environment)
  • WIP pacts only if the provider is on one of the main development branches
    • QUESTION: Does anyone want to verify the WIP pacts on a feature branch of the provider (ie. one not is not listed as a main development branch)
  • Maybe automatically include pacts from any matching consumer branches if on a feature branch (ie. if not on a main development branch)
    • QUESTION: Should these be pending, not pending, or should this be configurable? Would this cause problems for release branches (eg. people using Git Flow who have a release or hotfix branches, would get pacts from the consumer's release or hotfix branches)
  • And automatically turn on pending pacts.

Basically, implementing the "best practice" recommended configuration without having the provider team having to understand it in the detail they do today.

The autoSelectPacts feature will make the configuration of the consumer version selectors unnecessary for most providers, as this is the list of pacts that we currently recommend to be verified at the moment, abstracted via the "tags". If a user has non-standard requirements, they can fall back to the current approach of specifying the consumer versions explicitly. This should reduce the cognitive load in implementing Pact in a CI/CD pipeline.

Thoughts welcome.

[bethesque]

Thinking about it more, I'm leaning towards the matching feature branch config being separate. I want the "autoSelect" feature to be what Pact beginners should just enable by default when they don't really understand what they're doing. The matching feature branch thing is a bit more of an advanced workflow, when everyone knows what they're doing.

[ertrzyiks]

I'm not quite sure what is the purpose of mainDevelopmentBranches setting.

In our setup mainDevelopmentBranches would be probably empty, because our main branch is nothing special in regards to contracts. The only special version we care about is production (currently a pact broker tag, not git branch). When we change the provider, we check it against production of the consumers. When a WIP contracts lands on production tag, it is not longer WIP to us.

Does anyone want WIP pacts on a feature branch

We treat feature branches of consumers as WIP because otherwise the contracts verifications are missing in the broker if the provider is released. It happens because the contract content doesn't change and we don't trigger the verification job for a single contract. The only thing that changed is a production version of the provider. Keeping such contracts WIP is our way to keep can-i-deploy script working (without missing verification errors) while the feature branch is open.

Should these be pending, not pending, or should this be configurable?

We are happy with the current workflow, so contract is pending for main branch of provider until it is verified on that main branch.

[bethesque]

It sounds like you have wip on for all branches of your provider then. Would you mind sharing your provider verification config? What is your development/release process in regards to branches? e.g features on feature branches, merge to master, release from master?

[ertrzyiks]

This is simplified version of our config:

# provider_revision: commit sha
# provider_branch: 'main' or a feature branch

Pact.service_provider(provider) do
    app_version(provider_revision)
    app_version_tags([provider_branch].compact)
    publish_verification_results(true)

    honours_pacts_from_pact_broker do
      pact_broker_base_url 'https://....',
                           username: ...,
                           password: ...

      consumer_version_selectors({
        {consumer: 'Consumer 1', tag: 'production', latest: true},
        {consumer: 'Consumer 2', tag: 'production', latest: true},
        {consumer: 'Consumer 3', tag: 'production', latest: true},
      })

      if should_check_wip?
        # WIP returns all consumers regardless of the selectors above,
        # see https://pact.canny.io/feature-requests/p/give-more-control-over-wip-contract-selection
        include_wip_pacts_since '2020-11-01' 
        enable_pending true
      end
    end
  end

Each provider and consumer has main branch. We create feature branches from the main branch and when they are ready we merge them into the main branch. The main branch is automatically released to staging and manually released to production. A staging release is not reflected in git/pact broker anyhow. A production release is reflected by a git tag with a version and a pact broker tag production applied to the deployed commit.

[ertrzyiks]

@bethesque from your other message #391 (comment) it seems that our flow would be covered 👍 Feature branches would stop be WIP as soon as they are merged, but we have a main branch specifically selected so it would be still verified. All good from our side I think 👌

[bethesque]

Just to be clear - the exisiting features and configuration would still be there. It's just about making the recommended defaults super easy to do.

[ertrzyiks]

Right! Also, I finally understood the purpose of mainDevelopmentBranches and it makes sense to me.

[bethesque]

As an example of how this would change the configuration, this is the existing recommended configuration:

{
consumerVersionSelectors: [
    {
      tag: "main",
      latest: true
    },
    {
      tag: process.env.GIT_BRANCH,
      latest: true
    },
    {
      tag: "test",
      latest: true
    },
    {
      tag: "production",
      latest: true
    }
  ],
  enablePending: true,
  includeWipPactsSince: process.env.GIT_BRANCH === "main" ? "2020-01-01" : undefined,
}

That would be replaced by:

{
  autoSelectPacts: true
}

(or whatever we decide to call the option). The rest of the current configuration would then go under an "advanced" section, for people who really knew what they were doing and wanted a customised selection of pacts.

[solarmosaic-kflorence]

This feature would work well for us. I also think it would be great for new users, as you said, to implement best practices currently there is quite a lot of cognitive overhead.

Does anyone want to verify the WIP pacts on a feature branch of the provider (ie. one not is not listed as a main development branch)

There is the potential benefit of getting faster feedback on WIP consumer changes to pacts when you enable WIP support for provider feature branches, however, in practice if you have WIP enabled on the provider feature branches along with pending pacts, it makes it difficult for the provider team to ensure they are implementing a new pact correctly (because even if they explicitly select the consumer pact tag they want to verify, it will be pending and so will not cause the build to fail if it fails -- we discovered this the hard way). It's also easy to just run the provider build against a development branch (e.g. master) and have WIP pacts enabled there instead. The feedback is still pretty fast, and it reduces some complexity in feature branches. So, this was a long-winded way of saying, I can't see why anyone would want that, even though it could seem desirable at first.

Maybe automatically include pacts from any matching consumer branches if on a feature branch (ie. if not on a main development branch)

This would allow for the "matching consumer/provider tag" verification pattern. Because you have to opt-in to that, it seems fine to include. Previously I may have been opposed to this because of #432 -- but now that it is fixed, I don't see any problem with having this behavior enabled by default as teams can still use explicit configuration if they want to.

Should these be pending, not pending, or should this be configurable? Would this cause problems for release branches (eg. people using Git Flow who have a release or hotfix branches, would get pacts from the consumer's release or hotfix branches)

Intuitively, I would expect those pacts to not be pending (e.g. if consumer and provider both use feat-x tag, I would expect the provider build to fail if verification fails). Thinking about release and hotfix branches, those are essentially a main development branch for a release... I can't think of a reason you would have a pact in a consumer that shouldn't be verifiable by a provider in one of those branches, but if someone faces that problem this could be configurable per-branch.

Some other thoughts:

• mainDevelopmentBranches could probably just be developmentBranches or maybe stableBranches (as an aside: do we want to tie this specifically to branches? or tags?)
• as an alternative to autoSelectPacts: true -- I wonder if you could introduce the idea of selecting pacts for a workflow, where this could be the default workflow. something like { "workflow": "default" }

[bethesque]

I'm interested in the workflow idea @solarmosaic-kflorence. What alternative workflows can you imagine?

mainDevelopmentBranches could probably just be developmentBranches or maybe stableBranches (as an aside: do we want to tie this specifically to branches? or tags?)

I've actually just refactored this to "mainBranch" (singular) as the more I thought of it, the more it only made sense to specify one main branch.

as an aside: do we want to tie this specifically to branches? or tags?

I started the Broker development 8 years ago with the idea that modelling branches and deployments was too workflow specific, and that the broker should be agnostic of that, and just let the "tags' tell us what we needed to know. The way tagging works is ok for advanced users who are able to understand the abstraction, but we've found it's actually one of the hardest things for people to pick up. Having recently added proper support for branches and recording deployments, I've found that it's simplified a HEAP of stuff, and has made many features possible that weren't before, because we couldn't infer any semantic meaning from the tags. It's also much easier to write descriptions and error messages using the proper terminology (eg. "the latest version with tag prod" vs "the version currently deployed to production").

Tags won't stop being supported, but once branches and environments are released, I can't see a good use for them any more.

[solarmosaic-kflorence]

Off the top of my head, I can't think of another workflow -- but our current use-case matches exactly your best practices. We don't use git flow, we deploy from a single main branch, we don't do mobile deployments, or anything else that would potentially make setup more complicated. I imagine as more features are added to Pact there may be more divergence in the workflow, or if Pact starts to become more integrated with other standards like OpenAPI or Blueprint.

The fact that you wish to tie this more explicitly to branches and deployments may mean this workflow should be more specific to that -- maybe something like a "branch-deployments" workflow 🤔