Support snapshot uploading to s3 for every build #1472

gaiksaya · 2022-01-13T23:06:33Z

Description

The current jenkins workflow only updates the release candidates after merging #1378
This PR supports uploading snapshots to below location for every build too:

snapshots/core/opensearch/1.2.3-SNAPSHOT/opensearch-min-1.2.3-SNAPSHOT-linux-x64-latest.tar.gz
snapshots/core/opensearch/1.2.3-SNAPSHOT/opensearch-min-1.2.3-SNAPSHOT-linux-arm64-latest.tar.gz

Issues Resolved

#1442

Check List

Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

… into snapshots

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

codecov-commenter · 2022-01-13T23:12:45Z

Codecov Report

Merging #1472 (fb4187d) into main (fa1f6f2) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##               main    #1472   +/-   ##
=========================================
  Coverage     94.45%   94.45%           
  Complexity       12       12           
=========================================
  Files           141      142    +1     
  Lines          3084     3085    +1     
  Branches          8        8           
=========================================
+ Hits           2913     2914    +1     
  Misses          164      164           
  Partials          7        7

Impacted Files	Coverage Δ
tests/jenkins/jobs/uploadSnapshotsToS3_Jenkinsfile	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fa1f6f2...fb4187d. Read the comment docs.

peternied

Minor comments, thanks for working on this

jenkins/opensearch/distribution-build.jenkinsfile

vars/uploadMinSnapshotsToS3.groovy

jenkins/opensearch/distribution-build.jenkinsfile

dblock · 2022-01-13T23:17:39Z

tests/jenkins/UploadMinSnapshotsToS3.groovy

+        Path source = Path.of("tests/data/opensearch-build-1.1.0.yml");
+        target = Path.of("workspace/builds/opensearch/manifest.yml");
+        Files.createDirectories(target.getParent());
+        Files.copy(source, target, StandardCopyOption.REPLACE_EXISTING);


Can we have this setup pre-built since this is a test inside tests/data ahead of time? In which case cleanup isn't required either.

Not sure what you mean by pre-build. Can you elaborate? It is already a part of setUp()

I'll try to explain better. This code creates a folder and copies a file when setUp is run. I would just copy the file and commit it and remove this code.

Got it. Removed this code. And edited the workplace variable so that it fetched from write manifest.
Thanks!

@dblock Can you review it once before I merge?
Thanks!

vars/uploadMinSnapshotsToS3.groovy

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

tests/jenkins/jobs/uploadSnapshotsToS3_Jenkinsfile.txt

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

abhinavGupta16

Can we add a test for jenkins/opensearch/distribution-build.jenkinsfile

Thanks!

gaiksaya · 2022-01-14T01:01:55Z

Can we add a test for jenkins/opensearch/distribution-build.jenkinsfile

Thanks!

Out of the scope of this issue. We can create new issue and follow up on that.

Dismissing this review since this is an existing job and we'll be adding test cases for this job as a part of a separate issue.

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

tests/jenkins/jobs/uploadSnapshotsToS3_Jenkinsfile

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

tests/jenkins/TestUploadMinSnapshotsToS3.groovy

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

dblock

Looks good!

…1472) * Support snapshot builds Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

gaiksaya added 13 commits January 12, 2022 13:45

Support snapshot builds

541992a

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

Modify path for testing

d5e5f0c

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

Modify path for testing

f1f420b

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

Test

745d233

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

Test

fe681d6

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

Merge branch 'main' of github.com:opensearch-project/opensearch-build…

7c86d9b

… into snapshots

Use buildManifest

3754d5f

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

Use buildManifest

d911a90

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

Use buildManifest

2117e19

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

Use buildManifest

a23eb75

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

Rename

c515d05

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

Add tests

4f08288

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

Remove test env

335fd75

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

gaiksaya requested a review from a team as a code owner January 13, 2022 23:06

Modify error

e658c64

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

peternied approved these changes Jan 13, 2022

View reviewed changes

jenkins/opensearch/distribution-build.jenkinsfile Outdated Show resolved Hide resolved

vars/uploadMinSnapshotsToS3.groovy Show resolved Hide resolved

vars/uploadMinSnapshotsToS3.groovy Outdated Show resolved Hide resolved

gaiksaya marked this pull request as draft January 13, 2022 23:17

dblock requested changes Jan 13, 2022

View reviewed changes

gaiksaya added 2 commits January 13, 2022 16:18

format and log

b172d15

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

Remove condition

c4fb811

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

gaiksaya commented Jan 14, 2022

View reviewed changes

tests/jenkins/jobs/uploadSnapshotsToS3_Jenkinsfile.txt Outdated Show resolved Hide resolved

gaiksaya marked this pull request as ready for review January 14, 2022 00:25

Remove blank spaces

783c826

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

abhinavGupta16 previously requested changes Jan 14, 2022

View reviewed changes

Add snapshots manifest and update file path

1be274f

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

gaiksaya requested a review from dblock January 14, 2022 02:38

dblock approved these changes Jan 14, 2022

View reviewed changes

tests/jenkins/jobs/uploadSnapshotsToS3_Jenkinsfile Outdated Show resolved Hide resolved

Move test data and remove setup functions

fb4187d

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

dblock approved these changes Jan 14, 2022

View reviewed changes

tests/jenkins/TestUploadMinSnapshotsToS3.groovy Show resolved Hide resolved

gaiksaya added 2 commits January 14, 2022 11:00

Removed unsed libraries

7f8d7ac

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

Spaces

8a035aa

Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

dblock approved these changes Jan 14, 2022

View reviewed changes

gaiksaya merged commit 42140d1 into opensearch-project:main Jan 14, 2022

gaiksaya deleted the snapshots branch January 14, 2022 19:09

This was referenced Jan 14, 2022

Upload min targ.gz snapshots to publicly accessible bucket #1442

Closed

Publish build artifacts to correct locations from snapshot builds #1261

Closed

peterzhuamazon pushed a commit to peterzhuamazon/opensearch-build that referenced this pull request Feb 16, 2022

Support snapshot uploading to s3 for every build (opensearch-project#…

56d5cf0

…1472) * Support snapshot builds Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support snapshot uploading to s3 for every build #1472

Support snapshot uploading to s3 for every build #1472

gaiksaya commented Jan 13, 2022 •

edited

Loading

codecov-commenter commented Jan 13, 2022 •

edited

Loading

peternied left a comment

dblock Jan 13, 2022

gaiksaya Jan 14, 2022 •

edited

Loading

dblock Jan 14, 2022

gaiksaya Jan 14, 2022

gaiksaya Jan 14, 2022

abhinavGupta16 left a comment

gaiksaya commented Jan 14, 2022

dblock left a comment

Support snapshot uploading to s3 for every build #1472

Support snapshot uploading to s3 for every build #1472

Conversation

gaiksaya commented Jan 13, 2022 • edited Loading

Description

Issues Resolved

Check List

codecov-commenter commented Jan 13, 2022 • edited Loading

Codecov Report

peternied left a comment

Choose a reason for hiding this comment

dblock Jan 13, 2022

Choose a reason for hiding this comment

gaiksaya Jan 14, 2022 • edited Loading

Choose a reason for hiding this comment

dblock Jan 14, 2022

Choose a reason for hiding this comment

gaiksaya Jan 14, 2022

Choose a reason for hiding this comment

gaiksaya Jan 14, 2022

Choose a reason for hiding this comment

abhinavGupta16 left a comment

Choose a reason for hiding this comment

gaiksaya commented Jan 14, 2022

dblock left a comment

Choose a reason for hiding this comment

gaiksaya commented Jan 13, 2022 •

edited

Loading

codecov-commenter commented Jan 13, 2022 •

edited

Loading

gaiksaya Jan 14, 2022 •

edited

Loading