improve visibility of LID within Boost UI

[nonsense]

Information that should be surfaced

1. How much TiB/PiB of deal data an SP has
2. How many unsealed sectors we have
3. How many sealed sectors, without unsealed copies we have
4. Piece directory stats per sector? — how much was indexed… what sectors/deals are missing indices?

---

5. Split unsealed errors / sectors vs other issues
6. Display stats around sectors with no unsealed copies vs sectors with unsealed copies (% of number of sectors)
7. Flagged pieces — number of flagged pieces ; % of data that is flagged, vs total data held by miner
8. Fix lifecycle of deals — i.e. false positives of flagged pieces
9. Logs / progress around re-indexing
10. Block count & Average block size (Inspect Piece Page)

UX Changes

The Inspect page today

Proposal

Change the name of the "Inspect" menu item to "Local Index Directory"

This menu item links to a LID landing page.
The landing page links to a page with a list of sectors (like `lotus-miner sectors list) and a page with a list of flagged pieces (similar to what we have on the Inspect page today):

[LexLuthr]

Can we please also add lotus-miner info output to this list? We should probably have basic info like miner ID etc on our landing page and move deal list to another page. Landing page can also have all the above things as a nice summary.
Maybe move settings from their own page to here as well?
One stop shop for overview and config.

[nonsense]

@LexLuthr yes, I think this is a good idea. We should bring in a bit more details from lotus-miner into Boost as you suggest.

[nonsense]

In terms of implementation, I think we don't have to store any of this data in Boost, as it can change from underneath us (i.e. if a SP removes unsealed copies, or they get corrupted, or smth like that). We could have a periodic background worker that populates in-memory information about:

• number of sectors that contain deals, how many deals, and how much storage in total
• how many unsealed sectors with deals, vs sectors with deals that lack unsealed copies (try to get a CAR reader on each of them to make sure they aren't corrupted)
• number of index files, cross-referenced with the pieces in the sectors

That would already give intuition on the health of the unsealed copies + indices on a given SPs system, prior to initialising LID.

[brendalee]

A few additional considerations (copying over from slack):

To make it more clear to SPs where to get more info on if they need to unseal things, or index things, we can consider adding labels - such as data index state (for SPs to see the health of their data indexes), and sector state (to see the status of how many sealed/unsealed sectors they have, etc.).

Adding this could be helpful, when I went back to review this page, I got a bit confused on what Sector Unsealed Copies and Sector Proving State were for in regards to LID.

[nonsense]

The LID landing page frontend part is done. The backend part is hard to get right, considering that we need to collect stats from many components of Boost and Lotus. At the moment we have:

• Sectors Copies panel ready - data seems to be correct, should be confirmed further on filcollins and sofiaminer
• Sectors Proving State panel ready - data is incorrect, needs investigation on why we are getting Inactive sectors for filcollins.
• Deal Data not ready
• Pieces not ready
• Flagged Pieces - page needs to be refactored, and data needs to be reviewed
• Block stats not ready

Note that data in DAGstore and LID is currently incorrect - we have expired deals and slashed deals, i.e. sector and deal lifecycle is not handled well within Boost - this is something we should fix somehow as a stopgap, so that we don't display a lot of incorrect data to SPs when we release LID.

Furthermore there are piececids for deals that were submitted with a PSD, but never sealed - again, incorrect metadata within Boost, that should be considered when flagging pieces, as this could be very confusing to SPs.

As an SP when I open the LID landing page, I want 95% (or more) of data on it to be correct and 95% (or more) of flagged pieces to really be repairable or at least to be clear why they are flagged and not unsealed and not retrieveable.