Skip to content

Commit

Permalink
Revert "CI: build build-tools image" (#6156)
Browse files Browse the repository at this point in the history 
It turns out the issue with skipped jobs is not so trivial (because
Github checks jobs transitively), a possible workaround with `if:
always() && contains(fromJSON('["success", "skipped"]'),
needs.build-buildtools-image.result)` will tangle the workflow really
bad. We'll need to come up with a better solution.

To unblock the main I'm going to revert
#6082.
  • Loading branch information
@bayandin
bayandin committed Dec 16, 2023
1 parent 4787347 commit 9bdc25f
Show file tree
Hide file tree
Showing 9 changed files with 21 additions and 434 deletions.
57 changes: 0 additions & 57 deletions .github/workflows/build_and_push_docker_image.yml
View file

This file was deleted.

82 changes: 18 additions & 64 deletions .github/workflows/build_and_test.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,13 +44,13 @@ jobs:
exit 1
tag:
needs: [ check-permissions ]
runs-on: [ self-hosted, gen3, small ]
container: 369495373322.dkr.ecr.eu-central-1.amazonaws.com/base:pinned
outputs:
build-tag: ${{steps.build-tag.outputs.tag}}
build-tools-tag: ${{steps.buildtools-tag.outputs.image_tag}}

steps:
- name: Checkout
Expand All @@ -74,50 +74,11 @@ jobs:
shell: bash
id: build-tag

- name: Get buildtools tag
run: |
if [[ "$GITHUB_EVENT_NAME" == "pull_request" ]]; then
IMAGE_TAG=$GITHUB_RUN_ID
else
IMAGE_TAG=pinned
fi
echo "image_tag=${IMAGE_TAG}" >> $GITHUB_OUTPUT
shell: bash
id: buildtools-tag

check-if-build-tools-dockerfile-changed:
needs: [ check-permissions ]
runs-on: ubuntu-latest
outputs:
docker_file_changed: ${{ steps.dockerfile.outputs.docker_file_changed }}
steps:
- name: Check if Dockerfile.buildtools has changed
id: dockerfile
run: |
updated_files=$(gh pr --repo neondatabase/neon diff ${{ github.event.pull_request.number }} --name-only)
if [[ $updated_files == *"Dockerfile.buildtools"* ]]; then
echo "docker_file_changed=true" >> $GITHUB_OUTPUT
fi
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

build-buildtools-image:
needs: [ check-permissions, check-if-build-tools-dockerfile-changed, tag]
uses: ./.github/workflows/build_and_push_docker_image.yml
if: needs.check-if-build-tools-dockerfile-changed.outputs.docker_file_changed == 'true'
with:
dockerfile-path: Dockerfile.buildtools
image-name: build-tools
tag: ${{ needs.tag.outputs.build-tools-tag}}
secrets: inherit

check-codestyle-python:
if: always() && contains(fromJSON('["success", "skipped"]'), needs.build-buildtools-image.result)
needs: [ check-permissions, build-buildtools-image , tag ]
needs: [ check-permissions ]
runs-on: [ self-hosted, gen3, small ]
container:
image: 369495373322.dkr.ecr.eu-central-1.amazonaws.com/build-tools:${{ needs.tag.outputs.build-tools-tag }}
image: 369495373322.dkr.ecr.eu-central-1.amazonaws.com/rust:pinned
options: --init

steps:
Expand Down Expand Up @@ -147,11 +108,10 @@ jobs:
run: poetry run mypy .

check-codestyle-rust:
if: always() && contains(fromJSON('["success", "skipped"]'), needs.build-buildtools-image.result)
needs: [ check-permissions, build-buildtools-image, tag ]
needs: [ check-permissions ]
runs-on: [ self-hosted, gen3, large ]
container:
image: 369495373322.dkr.ecr.eu-central-1.amazonaws.com/build-tools:${{ needs.tag.outputs.build-tools-tag }}
image: 369495373322.dkr.ecr.eu-central-1.amazonaws.com/rust:pinned
options: --init

steps:
Expand Down Expand Up @@ -215,11 +175,10 @@ jobs:
run: cargo deny check --hide-inclusion-graph

build-neon:
if: always() && contains(fromJSON('["success", "skipped"]'), needs.build-buildtools-image.result)
needs: [ check-permissions, tag, build-buildtools-image ]
needs: [ check-permissions, tag ]
runs-on: [ self-hosted, gen3, large ]
container:
image: 369495373322.dkr.ecr.eu-central-1.amazonaws.com/build-tools:${{ needs.tag.outputs.build-tools-tag }}
image: 369495373322.dkr.ecr.eu-central-1.amazonaws.com/rust:pinned
options: --init
strategy:
fail-fast: false
Expand Down Expand Up @@ -452,7 +411,7 @@ jobs:
needs: [ check-permissions, build-neon, tag ]
runs-on: [ self-hosted, gen3, large ]
container:
image: 369495373322.dkr.ecr.eu-central-1.amazonaws.com/build-tools:${{ needs.tag.outputs.build-tools-tag }}
image: 369495373322.dkr.ecr.eu-central-1.amazonaws.com/rust:pinned
# Default shared memory is 64mb
options: --init --shm-size=512mb
strategy:
Expand Down Expand Up @@ -488,10 +447,10 @@ jobs:
uses: ./.github/actions/save-coverage-data

benchmarks:
needs: [ check-permissions, build-neon, tag]
needs: [ check-permissions, build-neon ]
runs-on: [ self-hosted, gen3, small ]
container:
image: 369495373322.dkr.ecr.eu-central-1.amazonaws.com/build-tools:${{ needs.tag.outputs.build-tools-tag }}
image: 369495373322.dkr.ecr.eu-central-1.amazonaws.com/rust:pinned
# Default shared memory is 64mb
options: --init --shm-size=512mb
if: github.ref_name == 'main' || contains(github.event.pull_request.labels.*.name, 'run-benchmarks')
Expand Down Expand Up @@ -520,12 +479,12 @@ jobs:
# while coverage is currently collected for the debug ones

create-test-report:
needs: [ check-permissions, regress-tests, coverage-report, benchmarks, tag ]
needs: [ check-permissions, regress-tests, coverage-report, benchmarks ]
if: ${{ !cancelled() && contains(fromJSON('["skipped", "success"]'), needs.check-permissions.result) }}

runs-on: [ self-hosted, gen3, small ]
container:
image: 369495373322.dkr.ecr.eu-central-1.amazonaws.com/build-tools:${{ needs.tag.outputs.build-tools-tag }}
image: 369495373322.dkr.ecr.eu-central-1.amazonaws.com/rust:pinned
options: --init

steps:
Expand Down Expand Up @@ -567,10 +526,11 @@ jobs:
})
coverage-report:
needs: [ check-permissions, regress-tests, tag ]
needs: [ check-permissions, regress-tests ]

runs-on: [ self-hosted, gen3, small ]
container:
image: 369495373322.dkr.ecr.eu-central-1.amazonaws.com/build-tools:${{ needs.tag.outputs.build-tools-tag }}
image: 369495373322.dkr.ecr.eu-central-1.amazonaws.com/rust:pinned
options: --init
strategy:
fail-fast: false
Expand Down Expand Up @@ -734,8 +694,7 @@ jobs:
}"
neon-image:
if: always() && contains(fromJSON('["success", "skipped"]'), needs.build-buildtools-image.result)
needs: [ check-permissions, build-buildtools-image, tag ]
needs: [ check-permissions, tag ]
runs-on: [ self-hosted, gen3, large ]
container: gcr.io/kaniko-project/executor:v1.9.2-debug
defaults:
Expand Down Expand Up @@ -774,7 +733,6 @@ jobs:
--context .
--build-arg GIT_VERSION=${{ github.event.pull_request.head.sha || github.sha }}
--build-arg BUILD_TAG=${{ needs.tag.outputs.build-tag }}
--build-arg TAG=${{ needs.tag.outputs.build-tools-tag }}
--build-arg REPOSITORY=369495373322.dkr.ecr.eu-central-1.amazonaws.com
--destination 369495373322.dkr.ecr.eu-central-1.amazonaws.com/neon:${{needs.tag.outputs.build-tag}}
--destination neondatabase/neon:${{needs.tag.outputs.build-tag}}
Expand All @@ -784,9 +742,8 @@ jobs:
run: rm -rf ~/.ecr

compute-tools-image:
if: always() && contains(fromJSON('["success", "skipped"]'), needs.build-buildtools-image.result)
runs-on: [ self-hosted, gen3, large ]
needs: [ check-permissions, build-buildtools-image, tag ]
needs: [ check-permissions, tag ]
container: gcr.io/kaniko-project/executor:v1.9.2-debug
defaults:
run:
Expand Down Expand Up @@ -821,7 +778,6 @@ jobs:
--context .
--build-arg GIT_VERSION=${{ github.event.pull_request.head.sha || github.sha }}
--build-arg BUILD_TAG=${{needs.tag.outputs.build-tag}}
--build-arg TAG=${{needs.tag.outputs.build-tools-tag}}
--build-arg REPOSITORY=369495373322.dkr.ecr.eu-central-1.amazonaws.com
--dockerfile Dockerfile.compute-tools
--destination 369495373322.dkr.ecr.eu-central-1.amazonaws.com/compute-tools:${{needs.tag.outputs.build-tag}}
Expand All @@ -832,8 +788,7 @@ jobs:
run: rm -rf ~/.ecr

compute-node-image:
if: always() && contains(fromJSON('["success", "skipped"]'), needs.build-buildtools-image.result)
needs: [ check-permissions, build-buildtools-image, tag ]
needs: [ check-permissions, tag ]
runs-on: [ self-hosted, gen3, large ]
container:
image: gcr.io/kaniko-project/executor:v1.9.2-debug
Expand Down Expand Up @@ -881,7 +836,6 @@ jobs:
--build-arg GIT_VERSION=${{ github.event.pull_request.head.sha || github.sha }}
--build-arg PG_VERSION=${{ matrix.version }}
--build-arg BUILD_TAG=${{needs.tag.outputs.build-tag}}
--build-arg TAG=${{needs.tag.outputs.build-tools-tag}}
--build-arg REPOSITORY=369495373322.dkr.ecr.eu-central-1.amazonaws.com
--dockerfile Dockerfile.compute-node
--destination 369495373322.dkr.ecr.eu-central-1.amazonaws.com/compute-node-${{ matrix.version }}:${{needs.tag.outputs.build-tag}}
Expand Down
130 changes: 0 additions & 130 deletions .github/workflows/update_build_tools_image.yml
View file

This file was deleted.

1 change: 0 additions & 1 deletion .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@ __pycache__/
test_output/
.vscode
.idea
neon.iml
/.neon
/integration_tests/.neon

Expand Down
Loading

1 comment on commit 9bdc25f

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2258 tests run: 2167 passed, 0 failed, 91 skipped (full report)

Flaky tests (5)

Postgres 16

  • test_crafted_wal_end[last_wal_record_crossing_segment]: debug
  • test_statvfs_pressure_usage: debug
  • test_ondemand_download_timetravel: debug
  • test_pitr_gc: debug

Postgres 14

  • test_ondemand_download_timetravel: debug

Code coverage (full report)

  • functions: 55.1% (9683 of 17562 functions)
  • lines: 82.4% (55644 of 67492 lines)
The comment gets automatically updated with the latest test results
9bdc25f at 2023-12-16T13:12:43.215Z :recycle:

Please sign in to comment.