Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

E2 snapshot uploading #9056

Merged
merged 78 commits into from
Dec 27, 2023
Merged

E2 snapshot uploading #9056

merged 78 commits into from
Dec 27, 2023

Conversation

mh0lt
Copy link
Contributor

@mh0lt mh0lt commented Dec 22, 2023

This change introduces additional processes to manage snapshot uploading for E2 snapshots:

erigon snapshots upload

The snapshots uploader command starts a version of erigon customized for uploading snapshot files to
a remote location.

It breaks the stage execution process after the senders stage and then uses the snapshot stage to send
uploaded headers, bodies and (in the case of polygon) bor spans and events to snapshot files. Because
this process avoids execution in run signifigantly faster than a standard erigon configuration.

The uploader uses rclone to send seedable (100K or 500K blocks) to a remote storage location specified
in the rclone config file.

The uploader is configured to minimize disk usage by doing the following:

  • It removes snapshots once they are loaded
  • It aggressively prunes the database once entities are transferred to snapshots

in addition to this it has the following performance related features:

  • maximizes the workers allocated to snapshot processing to improve throughput
  • Can be started from scratch by downloading the latest snapshots from the remote location to seed processing

snapshots command

Is a stand alone command for managing remote snapshots it has the following sub commands

  • cmp - compare snapshots
  • copy - copy snapshots
  • verify - verify snapshots
  • manifest - manage the manifest file in the root of remote snapshot locations
  • torrent - manage snapshot torrent files

@mh0lt mh0lt merged commit 79ed8ca into devel Dec 27, 2023
7 checks passed
@mh0lt mh0lt deleted the e2_snapshot_uploading branch December 27, 2023 22:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants