Deterministic task<->node assignment

[bajtos]

ETA: 2024-07-31

At the moment, SPARK nodes pick tasks from the list of per-round tasks at random and the evaluation service rewards all measurements completed.

Let's rework this part to follow what we designed in SPARK Tasking v2.

At a high level:

• Within each subnet and for each job, assign the reward to the node inside that subnet that is “closest” to the job, using the XOR distance between the node’s public key and the hash of the job definition. (See below for details.)
• The checker node should ask the tasker for the DRAND randomness, committee index and the list of tasks for the current SPARK round. Then, it should pick TN closes tasks from the list, using XOR against the checker's public key as the distance metric. (See the Notion doc for more details.)
• The evaluation service should repeat the same process to decide which measurements will be further used for verifications of proofs and eligible for rewards.

Prerequisites:

• Deterministic randomness for each SPARK round #27

Tasks

• Implement the new tasking algorithm in the Spark checker.
• Wait until the network picks up the change.
• Observe the new committee sizes and compare the real data against my modelling exercise. Check that all works as expected.
• Rework the evaluation service to apply the new tasking algorithm when evaluating the measurements

[bajtos]

Design proposal:
https://www.notion.so/spacemeridian/Spark-Tasking-v3-745b0e1020bb4000ac77acafee09e683

[bajtos]

Proof of concept:
filecoin-station/spark-evaluate#287