Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ethereumjs: add faster local test dockerfile #940

Merged
merged 2 commits into from
Nov 22, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions clients/ethereumjs/.dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
ethereumjs-monorepo/packages/ethereum-tests
ethereumjs-monorepo/packages/*/node_modules
ethereumjs-monorepo/packages/vm/benchmarks
ethereumjs-monorepo/node_modules
!ethereumjs-monorepo/node_modules/@ethereumjs
60 changes: 56 additions & 4 deletions clients/ethereumjs/Dockerfile.local
Original file line number Diff line number Diff line change
Expand Up @@ -6,23 +6,75 @@ FROM node:18-alpine
RUN apk update && apk add --no-cache bash git jq curl python3 gcc make g++ \
&& rm -rf /var/cache/apk/*

RUN npm i -g ts-node

# Default local client path: clients/ethereumjs/<ethereumjs-monorepo>
ARG local_path=ethereumjs-monorepo
COPY $local_path ethereumjs-monorepo

COPY ${local_path}/config ethereumjs-monorepo/config
COPY ${local_path}/package-lock.json ethereumjs-monorepo/package-lock.json
COPY ${local_path}/package.json ethereumjs-monorepo/package.json
COPY ${local_path}/scripts ethereumjs-monorepo/scripts

COPY ${local_path}/packages/block/package.json ethereumjs-monorepo/packages/block/package.json
COPY ${local_path}/packages/blockchain/package.json ethereumjs-monorepo/packages/blockchain/package.json
COPY ${local_path}/packages/client/package.json ethereumjs-monorepo/packages/client/package.json
COPY ${local_path}/packages/common/package.json ethereumjs-monorepo/packages/common/package.json
COPY ${local_path}/packages/devp2p/package.json ethereumjs-monorepo/packages/devp2p/package.json
COPY ${local_path}/packages/ethash/package.json ethereumjs-monorepo/packages/ethash/package.json
COPY ${local_path}/packages/evm/package.json ethereumjs-monorepo/packages/evm/package.json
COPY ${local_path}/packages/genesis/package.json ethereumjs-monorepo/packages/genesis/package.json
COPY ${local_path}/packages/rlp/package.json ethereumjs-monorepo/packages/rlp/package.json
COPY ${local_path}/packages/statemanager/package.json ethereumjs-monorepo/packages/statemanager/package.json
COPY ${local_path}/packages/trie/package.json ethereumjs-monorepo/packages/trie/package.json
COPY ${local_path}/packages/tx/package.json ethereumjs-monorepo/packages/tx/package.json
COPY ${local_path}/packages/util/package.json ethereumjs-monorepo/packages/util/package.json
COPY ${local_path}/packages/verkle/package.json ethereumjs-monorepo/packages/verkle/package.json
COPY ${local_path}/packages/vm/package.json ethereumjs-monorepo/packages/vm/package.json
COPY ${local_path}/packages/wallet/package.json ethereumjs-monorepo/packages/wallet/package.json
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not a fan of splitting out all these COPY lines. I guarantee they will get outdated soon.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can put all the files into a single COPY command, but some of these are directories. If I add those to the COPY command then the contents of that directory will get copied into the target folder, but it will not actually create the target directory. So if I COPY ${local_path}/config ethereumjs-monorepo/ then it will not create a config folder, it will instead dump all the files inside the ethereumjs-monorepo folder. I have tried many combinations but I cannot seem to make the COPY command to create the folder.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am assuming you mean that you want to reduce the amount of COPY commands, right? So put all these COPYd files/directories into a single COPY command? Or do you mean that the file structure here might change? (Deleted/renamed files/directories?)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I meant that the structure of the upstream sources can/will change. So the Dockerfile will require constant maintenance.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will cleanup these files since not all of these are necessary to run hive / the client. For our packages, it it not likely that we will add any packages in the near future, and even if we do so I would be very surprised if this would be more than 2 per year. However I can likely also prune that command.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have cleaned up the unnecessary files. However, for the other ones (our packages) I do not want to clean this up, since this means we have to re-install and re-build everything once any of our packages changes (and this is very slow). It is unlikely we add more than 2 packages in a year, and have not planned any new packages so far, so "constant updating" is (likely) not going to happen.


# for npm run prepare
RUN cd ethereumjs-monorepo && git init

RUN cd ethereumjs-monorepo && cp package.json package.json.bak && npm pkg set scripts.postinstall="echo no-postinstall"
RUN cd ethereumjs-monorepo && npm i
RUN cd ethereumjs-monorepo && cp package.json.bak package.json && rm package.json.bak

COPY ${local_path}/node_modules/@ethereumjs ethereumjs-monorepo/node_modules/@ethereumjs

COPY ${local_path}/packages/rlp ethereumjs-monorepo/packages/rlp
COPY ${local_path}/packages/util ethereumjs-monorepo/packages/util
COPY ${local_path}/packages/verkle ethereumjs-monorepo/packages/verkle
COPY ${local_path}/packages/wallet ethereumjs-monorepo/packages/wallet
COPY ${local_path}/packages/common ethereumjs-monorepo/packages/common
COPY ${local_path}/packages/devp2p ethereumjs-monorepo/packages/devp2p
COPY ${local_path}/packages/genesis ethereumjs-monorepo/packages/genesis
COPY ${local_path}/packages/trie ethereumjs-monorepo/packages/trie
COPY ${local_path}/packages/statemanager ethereumjs-monorepo/packages/statemanager
COPY ${local_path}/packages/tx ethereumjs-monorepo/packages/tx
COPY ${local_path}/packages/evm ethereumjs-monorepo/packages/evm
COPY ${local_path}/packages/block ethereumjs-monorepo/packages/block
COPY ${local_path}/packages/ethash ethereumjs-monorepo/packages/ethash
COPY ${local_path}/packages/blockchain ethereumjs-monorepo/packages/blockchain
COPY ${local_path}/packages/vm ethereumjs-monorepo/packages/vm
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here


RUN cd ethereumjs-monorepo/packages/client && cp package.json package.json.bak && npm pkg set scripts.build="echo no-build"
RUN cd ethereumjs-monorepo && npm run build --workspaces

COPY ${local_path}/packages/client ethereumjs-monorepo/packages/client

# Create version.txt
RUN cd ethereumjs-monorepo/packages/client && npm ethereumjs --version > /version.txt

# Add genesis mapper script, startup script, and enode URL retriever script
ADD genesis.json /genesis.json
ADD mapper.jq /mapper.jq
ADD ethereumjs.sh /ethereumjs.sh
ADD ethereumjs-local.sh /ethereumjs-local.sh
ADD enode.sh /hive-bin/enode.sh
ADD jwtsecret /jwtsecret

# Set execute permissions for scripts
RUN chmod +x /ethereumjs.sh /hive-bin/enode.sh
RUN chmod +x /ethereumjs-local.sh /hive-bin/enode.sh

# Expose networking ports
EXPOSE 8545 8546 8551 8547 30303 30303/udp
Expand All @@ -32,4 +84,4 @@ EXPOSE 8545 8546 8551 8547 30303 30303/udp
# since memory may spike during certain network conditions.
ENV NODE_OPTIONS=--max_old_space_size=6144

ENTRYPOINT ["/ethereumjs.sh"]
ENTRYPOINT ["/ethereumjs-local.sh"]
94 changes: 94 additions & 0 deletions clients/ethereumjs/ethereumjs-local.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
#!/bin/bash

# Startup script to initialize and boot a ethereum-js instance.
#
# This script assumes the following files:
# - `genesis.json` file is located in the filesystem root (mandatory)
# - `chain.rlp` file is located in the filesystem root (optional)
# - `blocks` folder is located in the filesystem root (optional)
# - `keys` folder is located in the filesystem root (optional)
#
# This script assumes the following environment variables:
#
# - HIVE_BOOTNODE enode URL of the remote bootstrap node
# - HIVE_NETWORK_ID network ID number to use for the eth protocol
# - HIVE_NODETYPE sync and pruning selector (archive, full, light)
#
# Forks:
#
# - HIVE_FORK_HOMESTEAD block number of the homestead hard-fork transition
# - HIVE_FORK_DAO_BLOCK block number of the DAO hard-fork transition
# - HIVE_FORK_DAO_VOTE whether the node support (or opposes) the DAO fork
# - HIVE_FORK_TANGERINE block number of Tangerine Whistle transition
# - HIVE_FORK_SPURIOUS block number of Spurious Dragon transition
# - HIVE_FORK_BYZANTIUM block number for Byzantium transition
# - HIVE_FORK_CONSTANTINOPLE block number for Constantinople transition
# - HIVE_FORK_PETERSBURG block number for ConstantinopleFix/PetersBurg transition
# - HIVE_FORK_ISTANBUL block number for Istanbul transition
# - HIVE_FORK_MUIRGLACIER block number for Muir Glacier transition
# - HIVE_FORK_BERLIN block number for Berlin transition
# - HIVE_FORK_LONDON block number for London
#
# Clique PoA:
#
# - HIVE_CLIQUE_PERIOD enables clique support. value is block time in seconds.
# - HIVE_CLIQUE_PRIVATEKEY private key for clique mining
#
# Other:
#
# - HIVE_MINER enable mining. value is coinbase address.
# - HIVE_MINER_EXTRA extra-data field to set for newly minted blocks
# - HIVE_LOGLEVEL client loglevel (0-5)
# - HIVE_GRAPHQL_ENABLED enables graphql on port 8545
# - HIVE_LES_SERVER set to '1' to enable LES server


# Immediately abort the script on any error encountered
set -e

cd /ethereumjs-monorepo/packages/client/

ethereumjs="ts-node /ethereumjs-monorepo/packages/client/bin/cli.ts"
FLAGS="--gethGenesis /genesis.json --rpc --rpcEngine --saveReceipts --rpcAddr 0.0.0.0 --rpcEngineAddr 0.0.0.0 --rpcEnginePort 8551 --ws false --logLevel debug --rpcDebug all --rpcDebugVerbose all --isSingleNode"

# Configure the chain.
mv /genesis.json /genesis-input.json
jq -f /mapper.jq /genesis-input.json > /genesis.json

# Dump genesis.
if [ "$HIVE_LOGLEVEL" -lt 4 ]; then
echo "Supplied genesis state (trimmed, use --sim.loglevel 4 or 5 for full output):"
jq 'del(.alloc[] | select(.balance == "0x123450000000000000000"))' /genesis.json
else
echo "Supplied genesis state:"
cat /genesis.json
fi

# Import clique signing key.
if [ "$HIVE_CLIQUE_PRIVATEKEY" != "" ]; then
# Create password file.
echo "Importing clique key..."
echo -n "$HIVE_CLIQUE_PRIVATEKEY" > /private_key.txt
# Ensure password file is used when running ethereumjs in mining mode.
if [ "$HIVE_MINER" != "" ]; then
FLAGS="$FLAGS --mine --unlock /private_key.txt --minerCoinbase $HIVE_MINER"
fi
fi

if [ "$HIVE_TERMINAL_TOTAL_DIFFICULTY" != "" ]; then
FLAGS="$FLAGS --jwt-secret /jwtsecret"
fi

# Load the test chain if present
echo "Loading initial blockchain..."
if [ -f /chain.rlp ]; then
FLAGS="$FLAGS --loadBlocksFromRlp=/chain.rlp"
else
echo "Warning: chain.rlp not found."
fi

if [ "$HIVE_BOOTNODE" != "" ]; then
FLAGS="$FLAGS --bootnodes=$HIVE_BOOTNODE"
fi
echo "Running ethereumjs with flags $FLAGS"
$ethereumjs $FLAGS