Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speedup CI with cache #3592

Merged
merged 137 commits into from
Aug 20, 2024
Merged

Speedup CI with cache #3592

merged 137 commits into from
Aug 20, 2024

Conversation

jochem-brouwer
Copy link
Member

@jochem-brouwer jochem-brouwer commented Aug 14, 2024

Fork of #2709

Dumps a CI cache after building the entire monorepo, so further CI jobs do not have to rebuild the repo

TODOs

  • Add new packages to use the cache also (wallet, verkle, maybe more?)
  • Verify that the CI does the expected job
  • Ensure that the cache is updated for each PR, and does not pull old versions (essentially running "old tests" on "old monorepo" thus giving the impression that all is fine, while it runs on old code / old monorepo which passed CI)

@acolytec3 acolytec3 added the type: test all hardforks This special label enables VM state and blockchain tests for all hardforks on the respective PR. label Aug 19, 2024
@holgerd77
Copy link
Member

Could this get a quick summary (but going beyond the initial description) what has been done and what has been achieved? Thanks! 🙂 🙏

@holgerd77
Copy link
Member

Also: still in Draft?

@holgerd77
Copy link
Member

Is coverage not running here anymore?

@acolytec3 acolytec3 marked this pull request as ready for review August 20, 2024 15:21
@jochem-brouwer
Copy link
Member Author

LGTM

@acolytec3 acolytec3 merged commit 0d576e2 into master Aug 20, 2024
66 of 70 checks passed
@acolytec3 acolytec3 deleted the speedup-ci branch August 20, 2024 15:26
holgerd77 pushed a commit that referenced this pull request Sep 9, 2024
* Call `dpt.destroy` inside rlpx

* Turn off github actions

* Add wtfnode

* shorten timeout on retry

* Ensure peer disconnects

* Slight disconnect tweak

* Ensure peer refreshIntervals are cleared

* clear interval in peer disconnect

* Add client wtfnode ci run

* run block and devp2p

* cache build outputs

* only run block coverage

* cache everything

* Adjust working directory

* fix working dir

* add submodules to block

* Add separate lint check

* add vm

* set correct job ordering

* Add restore step to vm:blockchain

* Add workflow dependency

* add jobs

* call block in devp2p

* Need build

* Try reusable workflows

* remove build dep

* update concurrency

* add ethash and common

* add trie and build

* Up setup-node to v3

* Update lockfile

* pass cache-key to block

* better hash key

* Cache key redux

* correct cache key usage

* remove package lock

* fix inputs

* fix syntax

* Remove echo

* add cachekey

* Add all package test suites to build script

* TYPO

* run ci if cache miss

* Add fake key to dispatch

* make cache key optional

* remove devp2p changes

* add additional packages

* add correct triggers

* update node to 20

* Add install if...

* just build block

* run build on node 20

* add default back

* remove conditional install step

* update action versions and package lock

* run install in correct directory

* update package-lock

* echo some stuff

* More tweaks

* try again

* reference correct output

* use common instead

* try skipping checkout

* fix util reference

* restore first

* restore first

* add checkout back

* add separate build step

* clarify naming on build vs dep cache

* update build cache key

* update build cache

* fix build key

* turn block on

* add inputs to block

* fix more things

* remove build cache step

* update package lock

* remove ignore scripts

* cache everything

* Always build and cache

* more experiments

* always run npm ci

* update to latest ver

* clean up client and vm jobs

* more cleanup

* remove multi-node strategy

* add correct concurrency groups

* fix concurrency deadlock

* remove recursive checkout from test jobs

* Fix devp2p

* remove retry on devp2p

* set working directory

* Move lint to separate job

* temp: test block test fail without changing pkg-lock

* update build.yml

* update build.yml

* test

* test

* restore correct code (this should pass block test now)

* use run_id

* test ethereum-tests with cache

* this should install the ethereum-tests in block since the cache will not be hit

* test 2

* test

* test ethereum-test failer

* restore

* test

* install ethereum-tests

* remove ethereum tetss from benchmark

* Add valid submodule cache key

* echo key

* add shell param

* echo that key

* add quotes

* add quotes

* fix the key

* make submodules recursive

* restore ethereum-tests

* fix cache key references

* make cache fail on miss

* update triggers and vm-build [no ci]

* try browser with alternatives for ethereum-tests

* check event_name

* Figure out how event trigger name

* Only clone if no inputs

* check for submodule cache key

* only restore if dep-cache key exists

* Fix restore logic

* fix everything

* turn the CI back on

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>
holgerd77 added a commit that referenced this pull request Sep 9, 2024
…ng Bug) (#3643)

* Ensure TransientStorage cleanups after Tx-level contract creation (#3625)

* evm: fix bug not clearing transient storage on tx-level create

* evm: add tests for tstore cleanup

* make cspell happy

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Add CHANGELOG entries, bump versions, update upstream dependency versions (EVM/VM)

* Rebuild package-lock.json

* Speedup CI with cache (#3592)

* Call `dpt.destroy` inside rlpx

* Turn off github actions

* Add wtfnode

* shorten timeout on retry

* Ensure peer disconnects

* Slight disconnect tweak

* Ensure peer refreshIntervals are cleared

* clear interval in peer disconnect

* Add client wtfnode ci run

* run block and devp2p

* cache build outputs

* only run block coverage

* cache everything

* Adjust working directory

* fix working dir

* add submodules to block

* Add separate lint check

* add vm

* set correct job ordering

* Add restore step to vm:blockchain

* Add workflow dependency

* add jobs

* call block in devp2p

* Need build

* Try reusable workflows

* remove build dep

* update concurrency

* add ethash and common

* add trie and build

* Up setup-node to v3

* Update lockfile

* pass cache-key to block

* better hash key

* Cache key redux

* correct cache key usage

* remove package lock

* fix inputs

* fix syntax

* Remove echo

* add cachekey

* Add all package test suites to build script

* TYPO

* run ci if cache miss

* Add fake key to dispatch

* make cache key optional

* remove devp2p changes

* add additional packages

* add correct triggers

* update node to 20

* Add install if...

* just build block

* run build on node 20

* add default back

* remove conditional install step

* update action versions and package lock

* run install in correct directory

* update package-lock

* echo some stuff

* More tweaks

* try again

* reference correct output

* use common instead

* try skipping checkout

* fix util reference

* restore first

* restore first

* add checkout back

* add separate build step

* clarify naming on build vs dep cache

* update build cache key

* update build cache

* fix build key

* turn block on

* add inputs to block

* fix more things

* remove build cache step

* update package lock

* remove ignore scripts

* cache everything

* Always build and cache

* more experiments

* always run npm ci

* update to latest ver

* clean up client and vm jobs

* more cleanup

* remove multi-node strategy

* add correct concurrency groups

* fix concurrency deadlock

* remove recursive checkout from test jobs

* Fix devp2p

* remove retry on devp2p

* set working directory

* Move lint to separate job

* temp: test block test fail without changing pkg-lock

* update build.yml

* update build.yml

* test

* test

* restore correct code (this should pass block test now)

* use run_id

* test ethereum-tests with cache

* this should install the ethereum-tests in block since the cache will not be hit

* test 2

* test

* test ethereum-test failer

* restore

* test

* install ethereum-tests

* remove ethereum tetss from benchmark

* Add valid submodule cache key

* echo key

* add shell param

* echo that key

* add quotes

* add quotes

* fix the key

* make submodules recursive

* restore ethereum-tests

* fix cache key references

* make cache fail on miss

* update triggers and vm-build [no ci]

* try browser with alternatives for ethereum-tests

* check event_name

* Figure out how event trigger name

* Only clone if no inputs

* check for submodule cache key

* only restore if dep-cache key exists

* Fix restore logic

* fix everything

* turn the CI back on

---------

Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>

* Dummy change to re-trigger CI

* Remove cspell CI job

* Attempt to fix CI (cannot find module @rollup/rollup-linux-x64-gnu)

* Rebuild package-lock.json

---------

Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
PR state: needs review type: CI type: test all hardforks This special label enables VM state and blockchain tests for all hardforks on the respective PR.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants