Publish build artifacts to correct locations from snapshot builds

[peternied]

Automation for 1.2.0 was building and uploading snapshots to maven at https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/common-utils/, there was another process that was building and uploading to maven which has deleted items under 1.2.0.0-SNAPSHOT/.

Disabled these jobs on private jenkins instance:

• opensearch_snapshots_creation_1_0_0
• opensearch_dashboards_snapshots_creation_1_0_0
  • These jobs have been updated so they are safe to run, but are only executed on demand, see Snapshots_CI/job/opensearch_snapshots_creation_1_0_0 on the fork jenkins fleet.

Acceptance Criteria:

• After snapshots are built in the distribution workflow they are uploaded onto a location like https://artifacts.opensearch.org/snapshots/core/opensearch-dashboards/2.0.0-SNAPSHOT/opensearch-dashboards-min-2.0.0-SNAPSHOT-linux-arm64-latest.tar.gz that can be consumed by distribution downloader
• Also include min build version to
  • https://artifacts.opensearch.org/release-candidates/core/opensearch/1.2.1/opensearch-min-1.2.1-linux-x64.tar.gz
  • https://artifacts.opensearch.org/release-candidates/core/opensearch/1.2.1/opensearch-min-1.2.1-linux-arm64.tar.gz

[peterzhuamazon]

We need to make sure the dev Jenkins are pushing min tarballs to artifacts.opensearch.org.
Once we can do that we can safely disable the old workflows, and thus allow distribution downloader to always have correct copy to download.

Thanks.

[peterzhuamazon]

Example link that is consumed by distribution downloader:

Snapshots Static Latest Link: https://artifacts.opensearch.org/snapshots/core/opensearch-dashboards/2.0.0-SNAPSHOT/opensearch-dashboards-min-2.0.0-SNAPSHOT-linux-arm64-latest.tar.gz

[peternied]

Thanks @peterzhuamazon, I've updated the description with these details so we can start working on updating the snapshot process on opensearch-build

[peternied]

Having trouble viewing the bucket contents for the moment, will circle back on this issue to check in.

[peternied]

@zylzulu Could you take a look at this today? In our build workflow it does build(snapshot=true) and then attempts to upload to maven, we should also have it push artifacts min artifacts to paths like https://artifacts.opensearch.org/snapshots/core/opensearch/1.2.2-SNAPSHOT/opensearch-min-1.2.2-SNAPSHOT-linux-x64-latest.tar.gz

See https://github.com/opensearch-project/opensearch-build/blob/main/jenkins/opensearch/distribution-build.jenkinsfile#L47 for a starting point

[peterzhuamazon]

Sync up with Yilin and she has the clear context now on how to proceed with this.
She will make changes to upload artifacts lib for Jenkins to add release-candidates promotion 1st after every bundle build.
We will talk about snapshots latest tar later

Thanks.

[dblock]

#1378

[gaiksaya]

#1472 @peternied Can you check with Acceptance Criteria and close this issue?
Thanks!

[peternied]

Did a quick check on 1.3.0 which is in development, the timestamp is from today, so the snapshot artifacts are being uploaded as expected. Thanks @gaiksaya

>curl https://artifacts.opensearch.org/snapshots/core/opensearch/1.3.0-SNAPSHOT/opensearch-min-1.3.0-SNAPSHOT-linux-arm64-latest.tar.gz -v -XHEAD
Warning: Setting custom HTTP method to HEAD with -X/--request may not work the
Warning: way you want. Consider using -I/--head instead.
*   Trying 143.204.154.26:443...
* Connected to artifacts.opensearch.org (143.204.154.26) port 443 (#0)
* schannel: disabled automatic use of client certificate
* schannel: ALPN, offering http/1.1
* schannel: ALPN, server accepted to use http/1.1
> HEAD /snapshots/core/opensearch/1.3.0-SNAPSHOT/opensearch-min-1.3.0-SNAPSHOT-linux-arm64-latest.tar.gz HTTP/1.1
> Host: artifacts.opensearch.org
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: application/x-gzip
< Content-Length: 220695130
< Connection: keep-alive
< Date: Tue, 18 Jan 2022 16:58:04 GMT
< Last-Modified: Tue, 18 Jan 2022 16:18:25 GMT
< ETag: "1e01ee8cb04fb29a6ff719980e177d0b-43"
< x-amz-server-side-encryption: AES256
< x-amz-version-id: Bx.4DsKAhBT9B6mJTPTuEvCB517hmmhC
< Accept-Ranges: bytes
< Server: AmazonS3
< X-Cache: Miss from cloudfront
< Via: 1.1 e67238cd96dfce710846014b9928af62.cloudfront.net (CloudFront)
< X-Amz-Cf-Pop: DFW3-C1
< X-Amz-Cf-Id: wnplrfnOchWXYmgmt9ADz-9udltlg1OxvIhW4bD0uXfBF0kwoquc5w==
<