Signatures do not match across backing registries

[BenTheElder]

What happened:

See:
kubernetes/registry.k8s.io#187 and https://kubernetes.slack.com/archives/CJH2GBF7Y/p1679166550351119

Images should have identical digests no matter what region I pull from.

This does not appear to be the case for some of the sigstore images added by the image-promoter

What you expected to happen:

Images should be identical in all backing registries

How to reproduce it (as minimally and precisely as possible):

Check us-west1 vs us-west2 AR instances for provider-aws/aws-ebs-csi-driver:sha256-c75878156614efc7c501ea655cd9da1ede35e9aee252436a92ff01f67f1c53fa.sig

Anything else we need to know?:

Environment:

• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Others:

[BenTheElder]

See: kubernetes/registry.k8s.io#187 (comment)

I think this is relatively unlikely to break anyone as long as it remains scoped to sigstore signatures (which appears to be the case) given the usage patterns for this type of """image""". Still worth fixing.

[BenTheElder]

Looks like this is also "some signatures are missing in some regions" kubernetes/registry.k8s.io#187 (comment)

[BenTheElder]

xref: kubernetes/release#2962

[puerco]

The signature oci objects are diverging because the promoter died mid process. This PR will not fix the rate limit killing kpromo but at least will stop the propagation of new divergent .sig "images": #809

Remediation is being tracked in kubernetes/release#2962

[BenTheElder]

Thanks!

Re: killed by rate-limit – We should also consider retry with-backoff on error in case of e.g. network flakes. IIRC crane exposes an API for this and crane copy does this by default.