Skip to content

Commit

Permalink
Simulator improvements (#619)
Browse files Browse the repository at this point in the history
* Refactor load simulator

* Cleanup

* simulator: support http as well as websocket endpoints

* simulator: move help err handling

* simulator: Fix distribute funds bug

* simulator: add metrics at the end of simulator run

* Support no timeout

* Address review comments

* Remove unused struct

* Update cmd/simulator/config/flags.go

* Update cmd/simulator/load/funder.go

* Remove . from simulator flags

* Add comment to scripts/run_simulator.sh

* Address PR comments
  • Loading branch information
aaronbuchwald authored Apr 22, 2023
1 parent e6bf2a0 commit 9895149
Show file tree
Hide file tree
Showing 19 changed files with 704 additions and 958 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ bin/
build/

cmd/simulator/.simulator/*
cmd/simulator/simulator

# goreleaser
dist/
34 changes: 0 additions & 34 deletions cmd/simulator/.gitignore

This file was deleted.

47 changes: 10 additions & 37 deletions cmd/simulator/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Load Simulator

When building developing your own blockchain using `subnet-evm`, you may want to analyze how your fee parameterization behaves and/or how many resources your VM uses under different load patterns. For this reason, we developed `cmd/simulator`. `cmd/simulator` lets you drive arbitrary load across any number of [endpoints] with a user-specified `keys` directory (insecure) `timeout`, `concurrency`, `base-fee`, and `priority-fee`.
When building developing your own blockchain using `subnet-evm`, you may want to analyze how your fee parameterization behaves and/or how many resources your VM uses under different load patterns. For this reason, we developed `cmd/simulator`. `cmd/simulator` lets you drive arbitrary load across any number of [endpoints] with a user-specified `keys` directory (insecure) `timeout`, `workers`, `max-fee-cap`, and `max-tip-cap`.

## Building the Load Simulator

Expand All @@ -13,19 +13,19 @@ cd $GOPATH/src/github.com/ava-labs/subnet-evm/cmd/simulator
Build the simulator:

```bash
go build -o ./simulator *.go
go build -o ./simulator main/*.go
```

To confirm that you built successfully, run the simulator and print the version:

```bash
./simulator -v
./simulator --version
```

This should give the following output:

```
v0.0.1
v0.1.0
```

To run the load simulator, you must first start an EVM based network. The load simulator works on both the C-Chain and Subnet-EVM, so we will start a single node network and run the load simulator on the C-Chain.
Expand All @@ -45,45 +45,18 @@ The staking-enabled flag is only for local testing. Disabling staking serves two
2. Automatically opts in to validate every Subnet
:::

Once you have AvalancheGo running locally, it will be running an HTTP Server on the default port `9650`. This means that the RPC Endpoint for the C-Chain will be http://127.0.0.1:9650/ext/bc/C/rpc.
Once you have AvalancheGo running locally, it will be running an HTTP Server on the default port `9650`. This means that the RPC Endpoint for the C-Chain will be http://127.0.0.1:9650/ext/bc/C/rpc and ws://127.0.0.1:9650/ext/bc/C/ws for WebSocket connections.

Now, we can run the simulator command to simulate some load on the local C-Chain for 30s:

```bash
RPC_ENDPOINTS=http://127.0.0.1:9650/ext/bc/C/rpc
./simulator --rpc-endpoints=$RPC_ENDPOINTS --keys=./.simulator/keys --timeout=30s --concurrency=10 --base-fee=300 --priority-fee=100
./simulator --timeout=1m --concurrency=1 --max-fee-cap=300 --max-tip-cap=10 --txs-per-worker=50
```

## Command Line Flags

### `rpc-endpoints` (string)
To see all of the command line flag options, run

`rpc-endpoints` is a comma separated list of RPC endpoints to hit during the load test.

### `keys` (string)

`keys` specifies the directory to find the private keys to use throughout the test. The directory should contain files with the hex address as the name and the corresponding private key as the only content.

If the test needs to generate more keys (to meet the number of workers specified by `concurrency`), it will save them in this directory to ensure that the private keys holding funds at the end of the test are preserved.

:::warning
The `keys` directory is not a secure form of storage for private keys. This should only be used in local network and short lived network testing where losing or compromising the keys is not an issue.
:::

Note: if none of the keys in this directory have any funds, then the simulator will log an address that it expects to receive funds in order to fund the load test and wait for those funds to arrive.

### `timeout` (duration)

`timeout` specifies the duration to simulate load on the network for this test.

### `concurrency` (int)

`concurrency` specifies the number of concurrent workers that should send transactions to the network throughout the test. Each worker in the load test is a pairing of a private key and an RPC Endpoint. The private key is used to generate a stream of transactions, which are issued to the RPC Endpoint.

### `base-fee` (int)

`base-fee` specifies the base fee (denominated in GWei) to use for every transaction generated during the load test (generates Dynamic Fee Transactions).

### `priority-fee` (int)

`priority-fee` specifies the priority fee (denominated in GWei) to use for every transaction generated during the load test (generates Dynamic Fee Transactions).
```bash
./simulator --help
```
117 changes: 117 additions & 0 deletions cmd/simulator/config/flags.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
// (c) 2023, Ava Labs, Inc. All rights reserved.
// See the file LICENSE for licensing terms.

package config

import (
"errors"
"fmt"
"strings"
"time"

"github.com/spf13/pflag"
"github.com/spf13/viper"
)

const Version = "v0.1.0"

const (
ConfigFilePathKey = "config-file"
LogLevelKey = "log-level"
EndpointsKey = "endpoints"
MaxFeeCapKey = "max-fee-cap"
MaxTipCapKey = "max-tip-cap"
WorkersKey = "workers"
TxsPerWorkerKey = "txs-per-worker"
KeyDirKey = "key-dir"
VersionKey = "version"
TimeoutKey = "timeout"
)

var (
ErrNoEndpoints = errors.New("must specify at least one endpoint")
ErrNoWorkers = errors.New("must specify non-zero number of workers")
ErrNoTxs = errors.New("must specify non-zero number of txs-per-worker")
)

type Config struct {
Endpoints []string `json:"endpoints"`
MaxFeeCap int64 `json:"max-fee-cap"`
MaxTipCap int64 `json:"max-tip-cap"`
Workers int `json:"workers"`
TxsPerWorker uint64 `json:"txs-per-worker"`
KeyDir string `json:"key-dir"`
Timeout time.Duration `json:"timeout"`
}

func BuildConfig(v *viper.Viper) (Config, error) {
c := Config{
Endpoints: v.GetStringSlice(EndpointsKey),
MaxFeeCap: v.GetInt64(MaxFeeCapKey),
MaxTipCap: v.GetInt64(MaxTipCapKey),
Workers: v.GetInt(WorkersKey),
TxsPerWorker: v.GetUint64(TxsPerWorkerKey),
KeyDir: v.GetString(KeyDirKey),
Timeout: v.GetDuration(TimeoutKey),
}
if len(c.Endpoints) == 0 {
return c, ErrNoEndpoints
}
if c.Workers == 0 {
return c, ErrNoWorkers
}
if c.TxsPerWorker == 0 {
return c, ErrNoTxs
}
// Note: it's technically valid for the fee/tip cap to be 0, but cannot
// be less than 0.
if c.MaxFeeCap < 0 {
return c, fmt.Errorf("invalid max fee cap %d < 0", c.MaxFeeCap)
}
if c.MaxTipCap < 0 {
return c, fmt.Errorf("invalid max tip cap %d <= 0", c.MaxTipCap)
}
return c, nil
}

func BuildViper(fs *pflag.FlagSet, args []string) (*viper.Viper, error) {
if err := fs.Parse(args); err != nil {
return nil, err
}

v := viper.New()
v.AutomaticEnv()
v.SetEnvKeyReplacer(strings.NewReplacer("-", "_"))
v.SetEnvPrefix("evm_simulator")
if err := v.BindPFlags(fs); err != nil {
return nil, err
}

if v.IsSet(ConfigFilePathKey) {
v.SetConfigFile(v.GetString(ConfigFilePathKey))
if err := v.ReadInConfig(); err != nil {
return nil, err
}
}
return v, nil
}

// BuildFlagSet returns a complete set of flags for simulator
func BuildFlagSet() *pflag.FlagSet {
fs := pflag.NewFlagSet("simulator", pflag.ContinueOnError)
addSimulatorFlags(fs)
return fs
}

func addSimulatorFlags(fs *pflag.FlagSet) {
fs.Bool(VersionKey, false, "Print the version and exit")
fs.String(ConfigFilePathKey, "", "Specify the config path to use to load a YAML config for the simulator")
fs.StringSlice(EndpointsKey, []string{"ws://127.0.0.1:9650/ext/bc/C/ws"}, "Specify a comma separated list of RPC Websocket Endpoints (minimum of 1 endpoint)")
fs.Int64(MaxFeeCapKey, 50, "Specify the maximum fee cap to use for transactions denominated in GWei (must be > 0)")
fs.Int64(MaxTipCapKey, 1, "Specify the max tip cap for transactions denominated in GWei (must be >= 0)")
fs.Uint64(TxsPerWorkerKey, 100, "Specify the number of transactions to create per worker (must be > 0)")
fs.Int(WorkersKey, 1, "Specify the number of workers to create for the simulator (must be > 0)")
fs.String(KeyDirKey, ".simulator/keys", "Specify the directory to save private keys in (INSECURE: only use for testing)")
fs.Duration(TimeoutKey, 5*time.Minute, "Specify the timeout for the simulator to complete (0 indicates no timeout)")
fs.String(LogLevelKey, "info", "Specify the log level to use in the simulator")
}
45 changes: 0 additions & 45 deletions cmd/simulator/go.mod

This file was deleted.

Loading

0 comments on commit 9895149

Please sign in to comment.