Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sync primary network checkpoints during bootstrapping #2752

Merged
merged 97 commits into from
Apr 17, 2024
Merged
Show file tree
Hide file tree
Changes from 93 commits
Commits
Show all changes
97 commits
Select commit Hold shift + click to select a range
d016bae
Add checkpoints and validators
StephenButtolph Feb 19, 2024
27206ba
Fetch checkpoints during bootstrapping
StephenButtolph Feb 19, 2024
31ca68f
Fix comment
StephenButtolph Feb 19, 2024
a1a9f18
Improve comment
StephenButtolph Feb 19, 2024
b61359a
Merge branch 'master' into add-checkpoints
StephenButtolph Feb 20, 2024
c5b7aa9
Implement interval tree for syncing
StephenButtolph Feb 20, 2024
63efd3b
Persist intervals
StephenButtolph Feb 20, 2024
2454c46
rename
StephenButtolph Feb 20, 2024
0efa5eb
Add block execution
StephenButtolph Feb 21, 2024
6a65a2a
Add length + TODOs
StephenButtolph Feb 21, 2024
f2234bd
nit
StephenButtolph Feb 21, 2024
1f63238
error rather than panic
StephenButtolph Feb 21, 2024
6947876
Refactor database passing
StephenButtolph Feb 21, 2024
28207f4
Merge branch 'interval-tree-syncing' into interval-tree-syncing-integ…
StephenButtolph Mar 19, 2024
be64296
Add interval tests
StephenButtolph Mar 19, 2024
884f39c
Fix too frequent iterator releases and batch writes
StephenButtolph Mar 19, 2024
e4e92cc
nit + support cancellation
StephenButtolph Mar 19, 2024
72314d1
Add invariant comment + tests
StephenButtolph Mar 19, 2024
e494033
Merge branch 'interval-tree-syncing' into interval-tree-syncing-integ…
StephenButtolph Mar 19, 2024
a64b863
wip
StephenButtolph Mar 19, 2024
782a02c
wip
StephenButtolph Mar 20, 2024
8851470
Merge branch 'master' into interval-tree-syncing
StephenButtolph Mar 20, 2024
2b07179
Merge branch 'interval-tree-syncing' into interval-tree-syncing-integ…
StephenButtolph Mar 20, 2024
5358587
support cancellation
StephenButtolph Mar 20, 2024
9d24b20
save
StephenButtolph Mar 20, 2024
0de6142
remove test code
StephenButtolph Mar 20, 2024
e658404
keep acceptor callback
StephenButtolph Mar 20, 2024
03f60c6
lint
StephenButtolph Mar 20, 2024
30ac2fb
Merge branch 'interval-tree-syncing' into interval-tree-syncing-integ…
StephenButtolph Mar 20, 2024
f81281a
fix test
StephenButtolph Mar 20, 2024
410db8c
Merge branch 'interval-tree-syncing-integration' of github.com:ava-la…
StephenButtolph Mar 20, 2024
984849c
remove incorrect db cleanup
StephenButtolph Mar 20, 2024
f4a724a
nits
StephenButtolph Mar 20, 2024
f01b6a3
allow specifying the log level
StephenButtolph Mar 20, 2024
9d7383f
merged
StephenButtolph Mar 20, 2024
0021f3e
cleanup metrics
StephenButtolph Mar 20, 2024
f1933d6
lint
StephenButtolph Mar 20, 2024
dd4a57f
add error information
StephenButtolph Mar 20, 2024
d5da1e9
upstream
StephenButtolph Mar 20, 2024
7373a89
Merge branch 'interval-tree-syncing' into interval-tree-syncing-integ…
StephenButtolph Mar 20, 2024
25e2801
cleanup
StephenButtolph Mar 20, 2024
5ca206b
reduce copied code
StephenButtolph Mar 20, 2024
cd90ef1
Merge branch 'interval-tree-syncing-integration' of github.com:ava-la…
StephenButtolph Mar 20, 2024
d9a7f8a
nit cleanup
StephenButtolph Mar 20, 2024
bd2b2d2
nit cleanup
StephenButtolph Mar 20, 2024
f0c5c10
Merge branch 'master' into interval-tree-syncing
StephenButtolph Mar 20, 2024
68ea448
Merge branch 'interval-tree-syncing' into interval-tree-syncing-integ…
StephenButtolph Mar 20, 2024
bef4ed5
merged
StephenButtolph Mar 20, 2024
f05e324
nit
StephenButtolph Mar 20, 2024
b6d6ba9
Merge branch 'master' into interval-tree-syncing
StephenButtolph Mar 21, 2024
281dc8d
merged
StephenButtolph Mar 21, 2024
06fd656
Merge branch 'master' into interval-tree-syncing
StephenButtolph Mar 21, 2024
672c23f
merged
StephenButtolph Mar 21, 2024
175ab8e
nit
StephenButtolph Mar 21, 2024
c89d24d
Remove redundent interface
StephenButtolph Mar 21, 2024
249819c
Merge branch 'interval-tree-syncing' into interval-tree-syncing-integ…
StephenButtolph Mar 21, 2024
a736e6c
nit
StephenButtolph Mar 21, 2024
faa5329
nit
StephenButtolph Mar 21, 2024
bc3fedf
nit
StephenButtolph Mar 25, 2024
8134e16
nit
StephenButtolph Mar 25, 2024
9991e39
Move GetMissingBlockIDs and Execute out of the interval package
StephenButtolph Mar 25, 2024
2877502
Merge branch 'master' into interval-tree-syncing
StephenButtolph Mar 25, 2024
50754ad
Merge branch 'interval-tree-syncing' into interval-tree-syncing-integ…
StephenButtolph Mar 25, 2024
a7e2458
wip
StephenButtolph Mar 25, 2024
f1dd0a6
nit
StephenButtolph Mar 26, 2024
bf29d86
remove storage
StephenButtolph Mar 26, 2024
0626129
merged
StephenButtolph Mar 26, 2024
5a8fadc
nit
StephenButtolph Mar 26, 2024
69ed897
nit
StephenButtolph Mar 26, 2024
9815047
nit
StephenButtolph Mar 26, 2024
5bf4899
nit
StephenButtolph Mar 26, 2024
9da3bc1
nit
StephenButtolph Mar 26, 2024
93aac21
Merge branch 'interval-tree-syncing' into interval-tree-syncing-integ…
StephenButtolph Mar 26, 2024
bd5687a
nit
StephenButtolph Mar 26, 2024
b6e3ee1
merged
StephenButtolph Mar 26, 2024
c376df2
add process test
StephenButtolph Mar 26, 2024
53bdb78
nit
StephenButtolph Mar 26, 2024
7310832
nit
StephenButtolph Mar 26, 2024
7de5499
wip
StephenButtolph Mar 26, 2024
4f88827
wip
StephenButtolph Mar 26, 2024
21d1fd1
remove duplicate code
StephenButtolph Mar 26, 2024
dd98c10
nit
StephenButtolph Mar 26, 2024
81f4b67
nit fix log check
StephenButtolph Mar 26, 2024
03a22b8
nit
StephenButtolph Mar 26, 2024
3499480
merged
StephenButtolph Mar 26, 2024
f19715d
merged
StephenButtolph Apr 16, 2024
ab8688c
add comment
StephenButtolph Apr 16, 2024
29913c9
update checkpoints
StephenButtolph Apr 16, 2024
baec0cc
fix merge
StephenButtolph Apr 16, 2024
5ab0a6e
update validators
StephenButtolph Apr 16, 2024
348cb0f
Manually track recent validators
StephenButtolph Apr 16, 2024
667372c
increase number of checkpoints
StephenButtolph Apr 16, 2024
db7c0c8
update validators
StephenButtolph Apr 16, 2024
ed549e3
Add regression test
StephenButtolph Apr 16, 2024
ad60d8a
Apply suggestions from code review
StephenButtolph Apr 16, 2024
bcf1c26
nits
StephenButtolph Apr 16, 2024
754fd1c
regen
StephenButtolph Apr 16, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions genesis/checkpoints.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
// Copyright (C) 2019-2024, Ava Labs, Inc. All rights reserved.
// See the file LICENSE for licensing terms.

package genesis

import (
"encoding/json"
"fmt"

_ "embed"

"github.com/ava-labs/avalanchego/ids"
"github.com/ava-labs/avalanchego/utils/constants"
"github.com/ava-labs/avalanchego/utils/set"
)

var (
//go:embed checkpoints.json
checkpointsPerNetworkJSON []byte

checkpointsPerNetwork map[string]map[ids.ID]set.Set[ids.ID]
)

func init() {
if err := json.Unmarshal(checkpointsPerNetworkJSON, &checkpointsPerNetwork); err != nil {
panic(fmt.Sprintf("failed to decode checkpoints.json %v", err))
StephenButtolph marked this conversation as resolved.
Show resolved Hide resolved
}
}

// GetCheckpoints returns all known checkpoints for the chain on the requested
// network.
func GetCheckpoints(networkID uint32, chainID ids.ID) set.Set[ids.ID] {
networkName := constants.NetworkIDToNetworkName[networkID]
return checkpointsPerNetwork[networkName][chainID]
}
612 changes: 612 additions & 0 deletions genesis/checkpoints.json

Large diffs are not rendered by default.

122 changes: 122 additions & 0 deletions genesis/generate/checkpoints/main.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
// Copyright (C) 2019-2024, Ava Labs, Inc. All rights reserved.
// See the file LICENSE for licensing terms.

package main

import (
"context"
"encoding/json"
"fmt"
"log"

"github.com/ava-labs/avalanchego/ids"
"github.com/ava-labs/avalanchego/indexer"
"github.com/ava-labs/avalanchego/utils/constants"
"github.com/ava-labs/avalanchego/utils/perms"
"github.com/ava-labs/avalanchego/utils/set"
)

const (
fujiURI = "http://localhost:9650"
mainnetURI = "http://localhost:9660"

maxNumCheckpoints = 100
)

var (
fujiXChainID = ids.FromStringOrPanic("2JVSBoinj9C2J33VntvzYtVJNZdN2NKiwwKjcumHUWEb5DbBrm")
fujiCChainID = ids.FromStringOrPanic("yH8D7ThNJkxmtkuv2jgBa4P1Rn3Qpr4pPr7QYNfcdoS6k6HWp")
mainnetXChainID = ids.FromStringOrPanic("2oYMBNV4eNHyqk2fjjV5nVQLDbtmNJzq5s3qs3Lo6ftnC6FByM")
mainnetCChainID = ids.FromStringOrPanic("2q9e4r6Mu3U68nU1fYjgbR6JvwrRx36CohpAX5UQxse55x1Q5")
)

// This fetches IDs of blocks periodically accepted on the P-chain, X-chain, and
// C-chain on both Fuji and Mainnet.
//
// This expects to be able to communicate with a Fuji node at [fujiURI] and a
// Mainnet node at [mainnetURI]. Both nodes must have the index API enabled.
func main() {
ctx := context.Background()

fujiPChainCheckpoints, err := getCheckpoints(ctx, fujiURI, "P")
if err != nil {
log.Fatalf("failed to fetch Fuji P-chain checkpoints: %v", err)
}
fujiXChainCheckpoints, err := getCheckpoints(ctx, fujiURI, "X")
if err != nil {
log.Fatalf("failed to fetch Fuji X-chain checkpoints: %v", err)
}
fujiCChainCheckpoints, err := getCheckpoints(ctx, fujiURI, "C")
if err != nil {
log.Fatalf("failed to fetch Fuji C-chain checkpoints: %v", err)
}

mainnetPChainCheckpoints, err := getCheckpoints(ctx, mainnetURI, "P")
if err != nil {
log.Fatalf("failed to fetch Mainnet P-chain checkpoints: %v", err)
}
mainnetXChainCheckpoints, err := getCheckpoints(ctx, mainnetURI, "X")
if err != nil {
log.Fatalf("failed to fetch Mainnet X-chain checkpoints: %v", err)
}
mainnetCChainCheckpoints, err := getCheckpoints(ctx, mainnetURI, "C")
if err != nil {
log.Fatalf("failed to fetch Mainnet C-chain checkpoints: %v", err)
}

checkpoints := map[string]map[ids.ID]set.Set[ids.ID]{
constants.FujiName: {
constants.PlatformChainID: fujiPChainCheckpoints,
fujiXChainID: fujiXChainCheckpoints,
fujiCChainID: fujiCChainCheckpoints,
},
constants.MainnetName: {
constants.PlatformChainID: mainnetPChainCheckpoints,
mainnetXChainID: mainnetXChainCheckpoints,
mainnetCChainID: mainnetCChainCheckpoints,
},
}
checkpointsJSON, err := json.MarshalIndent(checkpoints, "", " ")
StephenButtolph marked this conversation as resolved.
Show resolved Hide resolved
if err != nil {
log.Fatalf("failed to marshal checkpoints: %v", err)
}

err = perms.WriteFile("checkpoints.json", checkpointsJSON, perms.ReadWrite)
if err != nil {
log.Fatalf("failed to write checkpoints: %v", err)
}
}
StephenButtolph marked this conversation as resolved.
Show resolved Hide resolved

func getCheckpoints(
ctx context.Context,
uri string,
chainAlias string,
) (set.Set[ids.ID], error) {
var (
chainURI = fmt.Sprintf("%s/ext/index/%s/block", uri, chainAlias)
client = indexer.NewClient(chainURI)
)

// If there haven't been any blocks accepted, this will return an error.
_, lastIndex, err := client.GetLastAccepted(ctx)
if err != nil {
return nil, err
}

var (
numAccepted = lastIndex + 1
// interval is rounded up to ensure that the number of checkpoints
// fetched is at most maxNumCheckpoints.
interval = (numAccepted + maxNumCheckpoints - 1) / maxNumCheckpoints
checkpoints set.Set[ids.ID]
)
for index := interval - 1; index <= lastIndex; index += interval {
container, err := client.GetContainerByIndex(ctx, index)
if err != nil {
return nil, err
}

checkpoints.Add(container.ID)
}
return checkpoints, nil
}
64 changes: 64 additions & 0 deletions genesis/generate/validators/main.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
// Copyright (C) 2019-2024, Ava Labs, Inc. All rights reserved.
// See the file LICENSE for licensing terms.

package main

import (
"context"
"encoding/json"
"log"

"github.com/ava-labs/avalanchego/ids"
"github.com/ava-labs/avalanchego/utils/constants"
"github.com/ava-labs/avalanchego/utils/perms"
"github.com/ava-labs/avalanchego/utils/set"
"github.com/ava-labs/avalanchego/vms/platformvm"
"github.com/ava-labs/avalanchego/wallet/subnet/primary"
)

// This fetches the current validator set of both Fuji and Mainnet.
func main() {
ctx := context.Background()

fujiValidators, err := getCurrentValidators(ctx, primary.FujiAPIURI)
if err != nil {
log.Fatalf("failed to fetch Fuji validators: %v", err)
}

mainnetValidators, err := getCurrentValidators(ctx, primary.MainnetAPIURI)
if err != nil {
log.Fatalf("failed to fetch Mainnet validators: %v", err)
}

validators := map[string]set.Set[ids.NodeID]{
constants.FujiName: fujiValidators,
constants.MainnetName: mainnetValidators,
}
validatorsJSON, err := json.MarshalIndent(validators, "", " ")
if err != nil {
log.Fatalf("failed to marshal validators: %v", err)
}

err = perms.WriteFile("validators.json", validatorsJSON, perms.ReadWrite)
if err != nil {
log.Fatalf("failed to write validators: %v", err)
}
StephenButtolph marked this conversation as resolved.
Show resolved Hide resolved
}

func getCurrentValidators(ctx context.Context, uri string) (set.Set[ids.NodeID], error) {
client := platformvm.NewClient(uri)
currentValidators, err := client.GetCurrentValidators(
ctx,
constants.PrimaryNetworkID,
nil, // fetch all validators
)
if err != nil {
return nil, err
}

var nodeIDs set.Set[ids.NodeID]
for _, validator := range currentValidators {
nodeIDs.Add(validator.NodeID)
}
return nodeIDs, nil
}
34 changes: 34 additions & 0 deletions genesis/validators.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
// Copyright (C) 2019-2024, Ava Labs, Inc. All rights reserved.
// See the file LICENSE for licensing terms.

package genesis

import (
"encoding/json"
"fmt"

_ "embed"

"github.com/ava-labs/avalanchego/ids"
"github.com/ava-labs/avalanchego/utils/constants"
"github.com/ava-labs/avalanchego/utils/set"
)

var (
//go:embed validators.json
validatorsPerNetworkJSON []byte

validatorsPerNetwork map[string]set.Set[ids.NodeID]
)

func init() {
if err := json.Unmarshal(validatorsPerNetworkJSON, &validatorsPerNetwork); err != nil {
panic(fmt.Sprintf("failed to decode validators.json %v", err))
StephenButtolph marked this conversation as resolved.
Show resolved Hide resolved
}
}

// GetValidators returns recent validators for the requested network.
func GetValidators(networkID uint32) set.Set[ids.NodeID] {
networkName := constants.NetworkIDToNetworkName[networkID]
return validatorsPerNetwork[networkName]
}
Loading
Loading