feat!: per-miner & per-client daily deal stats

[bajtos]

Rework the daily_deals table into the following schema:

For each (day, miner_id, client_id), we want to know the following numbers (counts):

• tested: (NEW) total deals tested
• index_majority_found: (NEW) deals where we found a majority agreeing on the same result of the IPNI query
• indexed: deals announcing retrievals to IPNI (HTTP or Graphsync retrievals)
• indexed_http: (NEW) deals announcing HTTP retrievals to IPNI
• retrieval_majority_found: (NEW) deals where we found a majority agreeing on the same result of the retrieval request
• retrievable: deals where the majority agrees the content can be retrieved

BREAKING CHANGE: spark-stats endpoints consuming this table need to change queries to use tested instead of total.

Links:

• Find majorities in Spark measurements space-meridian/roadmap#59
• Deal Retrievability Score space-meridian/roadmap#156
• https://www.notion.so/spacemeridian/Move-Spark-score-from-individual-retrievals-to-retrievable-deals-09dc84797aea48ee84dd7a78f09a8b49?d=356ac20afa604231be63e2f30364dd3a#e81f9849d9a34006a0013830e409843d

Remaining work:

• add tests for the new columns
• add index to speed up date-based lookups (INDEX ON daily_deals (day))

[juliangruber]

BREAKING CHANGE: spark-stats endpoints consuming this table need to change queries to use tested instead of total.

Can we use this as an opportunity to fix this coupling, by adding an HTTP endpoint to spark-evaluate and letting spark-stats consume that, instead of querying its db?

[juliangruber]

Please see #336 (comment)

[bajtos]

BREAKING CHANGE: spark-stats endpoints consuming this table need to change queries to use tested instead of total.

Can we use this as an opportunity to fix this coupling, by adding an HTTP endpoint to spark-evaluate and letting spark-stats consume that, instead of querying its db?

spark-evaluate does not expose any HTTP interface right now. Changing that is way beyond the scope of this work, IMO.

IIRC, the direction we wanted to take is to improve spark-evaluate to publish the evaluation results to IPFS & commit the CID to the smart contract and then move all code building publicly-retrievable stats from spark-evaluate to spark-stats repository.

[bajtos]

@juliangruber I realised I need two values for majority_found - one regarding the indexer result and another regarding the retrieval result. It's possible for a deal to be correctly indexed but to not have majority agreement on whether it can be retrieved.

What would be good column names?

I feel that retrievable_majority_found and indexed_majority_found could be confusing, I read it as "somebody found an indexed majority" or "a majority that's retrievable".

How about has_result_indexed and has_result_retrievable?

The scores will be then calculated as follows:

• deals indexed: indexed / has_result_indexed
• deals offering HTTP retrievals: indexed_http / has_result_indexed
• retrievable deals: retrievable / has_result_retrievable

[juliangruber]

What about these:

• index_majority_found
• retrieval_majority_found

I think these make sense because they stand for "majority found in the index process" and "majority found in the retrieval process". Strictly speaking it's "index resolution", so should be index_resolution_majority_found, but I don't think this level of precision is necessary

[bajtos]

The PR is ready for final review & landing. @juliangruber PTAL 🙏🏻

[bajtos]

@juliangruber LGTY now?

[juliangruber]

Great work, Miro!!

[sentry-io]

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

• ‼️ Error: contract runner does not support sending transactions (operation="sendTransaction", code=UNSUPPOR... ?(platform-stats) View Issue

_{Did you find this useful? React with a 👍 or 👎}