Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

trie: implement lazy loading of trie nodes from disk #2838

Closed
qdm12 opened this issue Sep 15, 2022 · 5 comments
Closed

trie: implement lazy loading of trie nodes from disk #2838

qdm12 opened this issue Sep 15, 2022 · 5 comments
Assignees
Labels
A-design issue requires design work to think about how it would best be accomplished. C-complex Complex changes across multiple modules. Possibly will require additional research. S-trie issues related to the pkg/trie package.
Milestone

Comments

@qdm12
Copy link
Contributor

qdm12 commented Sep 15, 2022

Our memory usage is high since we keep all trie nodes in memory.

Instead we should (like Substrate) store all node hash -> node encoding on disk, and lazy load them as we need to access the trie (both read and write). Only the root hash should be kept in memory, and every node should be retrieved from disk as we go deeper in the trie (to insert or retrieve).

We should also implement pruning for nodes inserted/modified in uncle blocks (and make sure the finalized block doesn't contain them).

We can also implement a cache to keep the most frequently used nodes in memory and prevent decoding them again

Related with #2835 and #2836

@dimartiro
Copy link
Contributor

I found that Ethereum is also using a lazy loading strategy with the node keys -> encoded nodes

https://github.com/ethereum/go-ethereum/blob/d76efbb9be403689bc8d677a16af453728a7a513/trie/triedb/hashdb/database.go#L186-L224

@dimartiro dimartiro changed the title lib/trie: implement lazy loading of nodes from disk lib/trie: implement lazy loading of trie nodes from disk Nov 23, 2023
@dimartiro
Copy link
Contributor

dimartiro commented Nov 23, 2023

In substrate, they are using this trie crate for all trie operations, and they create a TrieDB structure backed by a DB with a cache, and they are implementing the lazy loading there

@dimartiro
Copy link
Contributor

This change will help us to improve our memory usage and our node startup time when we are using an snapshot, also improving our debugging time when we are trying to sync to the tip of the chain and we need to use snapshots for that.

This was referenced Nov 29, 2023
@P1sar P1sar removed trie labels Jan 15, 2024
@dimartiro dimartiro added the C-complex Complex changes across multiple modules. Possibly will require additional research. label Jan 18, 2024
@dimartiro dimartiro changed the title lib/trie: implement lazy loading of trie nodes from disk trie: implement lazy loading of trie nodes from disk Jan 18, 2024
@P1sar P1sar added the S-trie issues related to the pkg/trie package. label Jan 18, 2024
@P1sar P1sar added this to the Full Node MVP milestone Jan 18, 2024
@dimartiro
Copy link
Contributor

This issue will be Done when #3805 is completed

@P1sar P1sar added the A-design issue requires design work to think about how it would best be accomplished. label Mar 21, 2024
@P1sar
Copy link
Member

P1sar commented Mar 21, 2024

Design Design considered complete

@P1sar P1sar closed this as completed Mar 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-design issue requires design work to think about how it would best be accomplished. C-complex Complex changes across multiple modules. Possibly will require additional research. S-trie issues related to the pkg/trie package.
Projects
None yet
Development

No branches or pull requests

4 participants