feat: migration("re-indexing"), backfilling and diasgnostics tooling for the `ChainIndexer` #12450

aarshkshah1992 · 2024-09-11T15:23:06Z

ChainIndexer Migration and Diagnostics Tooling

This PR implements the "migration" (really re-indexing / backfilling), and diagnostics tooling for the ChainIndexer implemented in PR #12450, and is part of the work for #12453. This tooling takes the form of both RPC APIs on the daemon and lotus-shed CLI commands.

Re-indexing Process

The re-indexing tool enables clients to index their entire existing ChainState in the ChainIndexer. This process is necessary due to the removal of the existing MsgIndex, EthTxIndex, and EventIndex from Lotus.

Why Re-index Instead of Migrate?

We've chosen to re-index rather than migrate data from existing indices for two primary reasons:

Known issues: The existing indices have multiple known problems, and migrating could perpetuate incorrect index entries.
Lack of garbage collection: Existing indices contain many entries for which the corresponding tipset messages/events no longer exist in the ChainStore due to splitstore GC.

Instead, we're re-indexing the Chainstore/Chainstate on the node into the ChainIndexer. This ensures that all re-indexed entries have gone through the indexing logic of the new ChainIndexer and that the Index is in sync/reflects the actual contents of the Chainstore/Chainstate post re-indexing.

Diagnostics Tooling

This PR introduces diagnostic tools for detecting corrupt Index entries at specific epochs or epoch ranges.

While this PR implements functionality for optionally backfilling missing Index entries, it does not yet include the capability to "repair" corrupted Indexed entries. The repair functionality will be introduced in a subsequent PR. This approach allows us to first gather and analyze user reports, helping us understand the types and causes of corrupted Indexed entries(and if all they exist in the new ChainIndexer) before implementing repair mechanisms.

Core API

The fundamental building block for this tooling is the following RPC API:

type IndexValidation struct {
	TipsetKey string
	Height    uint64

	TotalMessages  uint64
	TotalEvents    uint64
	EventsReverted bool

	Backfilled bool
}

func (si *SqliteIndexer) ChainValidateIndex(ctx context.Context, epoch abi.ChainEpoch, backfill bool) (*types.IndexValidation, error)

This API has the following features:

Optionally backfills the Index with a tipset on the canonical chain for the given epoch if it is absent in the Index
Returns some aggregated stats for an indexed entry for diagnostics/inspection
Reports errors/corrupted indexed entries at the given epoch. Forms of Index corruption that can be diagnosed includes:
- Presence of multiple non-reverted tipsets at the given epoch
- Complete absence of a non-reverted tipset at the given epoch that does contain reverted tipsets
- Mismatch between the Chainstore state and the Indexed entries (tipset messages/events)
- Incorrect Indexing of null rounds at the given epoch

`lotus-shed` CLI tooling

The lotus-shed CLI tooling for both re-indexing/backfilling and diagnostics can then invoke this RPC API over epoch ranges. The corresponding lotus-shed backfill index [from, to] and lotus-shed inspect index [from, to] can then backfill/inspect/diagnose the Index for the given epoch ranges.

TODO

lotus-shed commands
RPC API should compare the Indexed entries with the chain store to detect a mismatch between indexed content and chain store content
Fill out all the diagnostics information in the RPC response by queying the messages and events tables in the Index.
automated tests

github-actions

Please update the PR title to match https://github.com/filecoin-project/lotus/blob/master/CONTRIBUTING.md#pr-title-conventions

chainindex/api.go

aarshkshah1992 · 2024-09-11T15:32:03Z

chainindex/api.go

+	return &types.IndexValidation{
+		TipsetKey:  ts.Key().String(),
+		Height:     uint64(ts.Height()),
+		Backfilled: true,


Fill out the other fields here using SQL queries.

chain/types/tipset.go

chainindex/indexer.go

chainindex/api.go

chainindex/ddls.go

aarshkshah1992 · 2024-09-12T06:55:37Z

@rvagg Would be great to have a first round of review here when you get the time.

github-actions

Please update the PR title to match https://github.com/filecoin-project/lotus/blob/master/CONTRIBUTING.md#pr-title-conventions

github-actions

Please update the PR title to match https://github.com/filecoin-project/lotus/blob/master/CONTRIBUTING.md#pr-title-conventions

node/builder_chain.go

itests/eth_filter_test.go

chain/types/index.go

rvagg · 2024-09-17T06:34:46Z

chain/index/indexer.go

@@ -220,6 +233,9 @@ func (si *SqliteIndexer) Apply(ctx context.Context, from, to *types.TipSet) erro
 	}
 	si.closeLk.RUnlock()

+	si.writerLk.Lock()
+	defer si.writerLk.Unlock()


maybe unlock before notifyUpdateSubs?

rvagg · 2024-09-17T06:35:39Z

chain/index/indexer.go

@@ -337,6 +353,9 @@ func (si *SqliteIndexer) Revert(ctx context.Context, from, to *types.TipSet) err
 	}
 	si.closeLk.RUnlock()

+	si.writerLk.Lock()
+	defer si.writerLk.Unlock()


ditto on unlocking before notifying subs?

rvagg · 2024-09-17T06:39:35Z

chain/index/ddls.go

 		&ps.getMinNonRevertedHeightStmt:           "SELECT MIN(height) FROM tipset_message WHERE reverted = 0",
 		&ps.hasNonRevertedTipsetStmt:              "SELECT EXISTS(SELECT 1 FROM tipset_message WHERE tipset_key_cid = ? AND reverted = 0)",
 		&ps.updateEventsToRevertedStmt:            "UPDATE event SET reverted = 1 WHERE message_id IN (SELECT message_id FROM tipset_message WHERE tipset_key_cid = ?)",
 		&ps.updateEventsToNonRevertedStmt:         "UPDATE event SET reverted = 0 WHERE message_id IN (SELECT message_id FROM tipset_message WHERE tipset_key_cid = ?)",
 		&ps.getMsgIdForMsgCidAndTipsetStmt:        "SELECT message_id FROM tipset_message WHERE tipset_key_cid = ? AND message_cid = ? AND reverted = 0",
-		&ps.insertEventStmt:                       "INSERT INTO event (message_id, event_index, emitter_addr, reverted) VALUES (?, ?, ?, ?)",
+		&ps.insertEventStmt:                       "INSERT INTO event (message_id, event_index, emitter_addr, reverted) VALUES (?, ?, ?, ?) ON CONFLICT (message_id, event_index) DO UPDATE SET reverted = 0",


some kind of test that exercises this and demonstrates it working well would be nice, also does the query plan show it behaving nicely? I can imagine this getting a bit weird matching for conflicts

@rvagg

Will be writing a test for this.

While sqliite does not generate query plans for INSERT STATEMENTS, the query plan for an equivalent query says this will be performant as we have a covering index for (message_id, event_index)

/.lotus-calibnet/sqlite% sqlite3 chainindex.db "EXPLAIN QUERY PLAN INSERT INTO event (message_id, event_index, emitter_addr, reverted) SELECT 1, 2, 'a', 0 WHERE NOT EXISTS ( SELECT 1 FROM event WHERE message_id = 1 AND event_index = 2 );" QUERY PLAN |--SCAN CONSTANT ROW `--SCALAR SUBQUERY 1 `--SEARCH event USING COVERING INDEX sqlite_autoindex_event_1 (message_id=? AND event_index=?)

This is because we have an UNIQUE constraint on (message_id, event_index)

chain/index/api.go

rvagg · 2024-09-17T06:54:25Z

chain/index/api.go

+	return revertedCount, nonRevertedCount, nil
+}
+
+func (si *SqliteIndexer) ChainValidateIndex(ctx context.Context, epoch abi.ChainEpoch, backfill bool) (*types.IndexValidation, error) {


if we did this all in a single transaction, would that remove the need for a write lock?

IIUC, transaction provides atomicity, not exclusivity. Imagine this scenario:

Backfill is called for epoch e -> it reads tipset ts at that epoch

Revertexecutes for ts and marks the tipset as reverted in TX1

Backfill executes for ts and marks it as applied in TX2

TX1 is committed first and then TX2 is comitted.

Basically, it's hard to reason about state here without this lock.

@rvagg

One work-around here it to only allow backfilling for "safe" epochs i.e. <=head - 30.

Then, the rewrite can only ever happen in an edge case.

chain/index/api.go

chain/index/ddls.go

rvagg · 2024-09-17T07:04:05Z

chain/index/api.go

+
+	// fetch the non-reverted tipset at this epoch
+	var indexedTsKeyCidBytes []byte
+	err = si.stmts.getNonRevertedTipsetAtHeightStmt.QueryRowContext(ctx, epoch).Scan(&indexedTsKeyCidBytes)


see the LIMIT 1 below if you're only expecting one result

rvagg · 2024-09-17T07:05:51Z

chain/index/api.go

+	}
+
+	return &types.IndexValidation{
+		TipsetKey: expectedTs.Key().String(),


see below, I think just the key would be good here, also note capitalisation of TipSetKey

rvagg · 2024-09-17T07:06:45Z

chain/index/api.go

+	return &types.IndexValidation{
+		TipsetKey:  ts.Key().String(),
+		Height:     uint64(ts.Height()),
+		Backfilled: true,


todo: other fields

* feat: finish todos of validation api * feat: add indexed data verification with chain store * feat: address comments and finish TODO * fix: build issue * address comments * fix: ci issue

aarshkshah1992 · 2024-09-17T07:48:50Z

Thanks for all your work here @akaladarshi 👍

Co-authored-by: Rod Vagg <rod@vagg.org>

akaladarshi · 2024-09-17T11:31:53Z

chain/types/index.go

-	NonRevertedEventsCount  uint64
-	Backfilled              bool
+	IndexedMessagesCount uint64
+	IndexedEventsCount   uint64


Should we rename this to IndexedNonRevertedMsgCount and IndexedNonRevertedEventsCount to be precise?

@akaladarshi That is an implementation detail that will confuse users. They're only asking for the indexed entries at a specific epoch here and so ofcourse expect only non-reverted tipsets back.

* feat: add lotus-shed command for backfilling chain indexer * feat: add lotus-shed command for inspecting the chain indexer * feat: use single lotus-shed command to inspect and backfill * fix: remove the unused queries * small changes * add change log

aarshkshah1992 · 2024-09-19T09:10:29Z

@BigLep This is now ready for review. Working on a user guide for this tooling as discussed offline.

aarshkshah1992 · 2024-09-19T09:10:56Z

@rvagg This is now in a good shape and has been tested on a calibnet node. Please take a look when you get the time.

…tion-api

github-actions bot previously requested changes Sep 11, 2024

View reviewed changes