Don't allow the service to miss blocks

[ra0x3]

• Using the explorer dashboard I can see we're missing a few block when indexing for the explorer
  • Grafana link
• Obviously the explorer should not be missing any blocks
• However, it's also safe to assume we're going to have some 0.01% margin of error
• We need to define some way to index the blocks we missed

Additional context

• The quick win here is probably to just create a solution for the explorer only
  • The better/longer solution would be to define a solution that will work for any indexer

[ra0x3]

Comment to keep in mind

[ra0x3]

• Indexing blocks should always be sequential - Block(1), Block(2), Block(3), ... Block(n)
• If the indexer is ever at a state where $LAST_INDEXED_BLOCK_HEIGHT - $CURRENT_BLOCK_HEIGHT != 1 the the service should pause/wait until the issue is remediated
• The service should not continue to index blocks if things are out of order
• I.e., we should detect whether we are $IN_SYNC or $OUT_OF_SYNC before even calling handle_event in the executor
• How TheGraph allows indexers to opt-in to this (somewhat similar)
  • syncrhonize: true/false (or something similar) might be a good config option to add to the manifest

[ra0x3]

• Just an FYI here that we are most likely going to have to devise a mechanism to allow the fuel-indexer (the service) to store blocks, outside of the context of any individual indexer
• We will use this cache of blocks in order to determine whether any given indexer is in or out of sync
• We have to store this cache of blocks on the service level, because Blocks and Transactions are not required for indexers
• This ☝🏽 is slightly related to feature: internal block store for backfilling newly-deployed indexers #990, just without the concept of migrating/backfilling (although those would side benefits we'd get for free)