-
Notifications
You must be signed in to change notification settings - Fork 66
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
enhancement: store BlockData in the DB and ensure the indexers don't miss blocks #1297
Conversation
28d50ff
to
edafc3a
Compare
@@ -681,7 +711,7 @@ pub async fn last_block_height_for_indexer( | |||
Ok(row | |||
.try_get::<i32, usize>(0) | |||
.map(|id| id.to_u32().expect("Bad block height.")) | |||
.unwrap_or_else(|_e| 1)) | |||
.unwrap_or(0)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we need to return zero here, for consistency.
If an indexer processed blocks 1, 2, and 3, the result would be 3
.
If an indexer processed block 1, the result would be 1
.
If an indexer processed no blocks, we shouldn't return 1
(as we currently do) but 0
.
I made changes to get_start_block
to account for this.
@@ -387,7 +387,8 @@ pub async fn get_start_block( | |||
.await?; | |||
let start = manifest.start_block().unwrap_or(last); | |||
let block = if *resumable { | |||
std::cmp::max(start, last) | |||
// if the last processed block is N, we want to resume from N+1 | |||
std::cmp::max(start, last + 1) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the indexer processed no blocks and last_block_height_for_indexer
returned 0, we bump it to 1—the first new block it needs to process.
Similarly, if the indexer processed blocks 1, 2, 3, and last_block_height_for_indexer
returned 3, we bump it to 4—the first new block.
806525c
to
912d9e2
Compare
2a0c8ee
to
52cfe6c
Compare
14096e0
to
badf8c3
Compare
ca1c432
to
bd0c68a
Compare
@lostman Is this PR good to close? |
Deprecated in favor of #1369 |
Description
This PR adds
index_block_data
table in which we store serializedBlockData
structs. We can use these to get a new indexer up to speed without fetching blocks from the client again.The table contains a trigger that ensures we have a full range of blocks—no
block_height
may be missing.Additionally, this PR adds the same trigger to each indexer's
indexmetadataentity
table. Again, this ensures that an indexer must process each block in sequence. If the indexer fails to process a block or processes blocks out of order, the trigger will raise an exception (and the indexer's transaction will fail).Note
The trigger ensures the indexer can't process blocks out of order or miss any blocks. However, it doesn't stop the indexer from running—it simply doesn't progress. To stop the indexer, we need the WASM error codes #1337 and return an error (and trigger an early exit!) when
put_object
fails.Testing steps
Please provide the exact testing steps for the reviewer(s) if this PR requires testing.
Changelog
Please add neat Changelog info here, according to our Contributor's Guide.