Proof of concept: deterministic tasking using SHA256(StationID) ^ SHA256(Task)

[bajtos]

See https://www.notion.so/spacemeridian/Spark-Tasking-v3-745b0e1020bb4000ac77acafee09e683

The step of building the list of per-station tasks takes 3124ms to complete, increasing the duration of the evaluation by 50% to 9641ms.

Links:

• Deterministic task<->node assignment spark#38
• Deterministic randomness for each SPARK round spark#27

[bajtos]

Analysis of committee sizes before and after this change:

https://www.notion.so/spacemeridian/Spark-Tasking-v3-745b0e1020bb4000ac77acafee09e683?pvs=4#900b01ef2cb148cd9d12a9a82ce3b21a

Conclusion

The proposed design has acceptable performance and significantly improves the quality of committees.

To have enough confidence that a majority of each committee is honest, we need each committee to have at least 40-50 participants. The current algorithm does not meet that requirement for at least 10% of committees. The proposed algorithm seems to meet this requirement in all committees.

[juliangruber]

conceptually, what's the difference between invalid_task and task_not_allowed? The first includes the second I think, but maybe we can rename it to be more precise?

[bajtos]

Great question! In the current code:

• INVALID_TASK means the task does not match any of the 1000 tasks defined for the round.
• TASK_NOT_ALLOWED means the task is valid for the round, but this node is not allowed to perform it.

I'd like to keep the distinction. INVALID_TASK is typically used for measurements that are committed to the wrong round. Typically, all measurements in the first batch published after a new round starts are measurements for tasks from the previous round because spark-api & checker nodes haven't noticed the new round yet.

I agree to find more descriptive names 👍🏻

[juliangruber]

Got it! What about

• TASK_NOT_FOUND
• TASK_WRONG_NODE

[bajtos]

What do you think about this?

• TASK_NOT_IN_ROUND
• TASK_NOT_FOR_NODE

On the second thought, TASK_WRONG_NODE works too 👍🏻

[juliangruber]

Suggested change

	const taskWithKeys = await Promise.all(sparkRoundDetails.retrievalTasks.map(async (task) => {
	const tasksWithKeys = await Promise.all(sparkRoundDetails.retrievalTasks.map(async (task) => {

[bajtos]

How about keyedTasks?

[juliangruber]

Suggested change

	const seeker = new closest.Seeker([...taskWithKeys], (targetKey, t) => t.key ^ targetKey)
	const seeker = new closest.Seeker([...tasksWithKeys], (targetKey, t) => t.key ^ targetKey)

[juliangruber]

Suggested change

	/* eslint-disable-next-line camelcase */
	for (const { stationId, participantAddress, inet_group } of measurements) {
	if (stations.has(stationId)) continue
	/* eslint-disable-next-line camelcase */
	stations.set(stationId, { participantAddress, inet_group })
	for (const { stationId, participantAddress, inet_group: inetGroup } of measurements) {
	if (stations.has(stationId)) continue
	/* eslint-disable-next-line camelcase */
	stations.set(stationId, { participantAddress, inetGroup })

I think it's better to rename props as soon as you use them

[bajtos]

Closing in favour of #296