feat: improve compaction job state management

[aleks-p]

The main change here is how we update the state of compaction jobs, in particular when workers are polling with state updates and ask for jobs.

The current implementation intermixes the persistence (boltdb) and in-memory state updates. This causes a few cases where an error could leave the 2 storage layers in an inconsistent state.

The new implementation does everything in memory first and constructs a list of items that need to be durably stored. If we fail to durably store something or otherwise end up in an unexpected state while persisting, the application will panic.

Bonus:

• handle failed compaction jobs (with max retries)
• prioritize jobs on compaction level before lease expiry
• unit tests for compaction job creation and state management
• convert some flags to config variables
• rename "job pre queue" to "job block queue"

[aleks-p]

This is still missing a failure reason from the worker, I am considering adding it so that we can persist it and show it in tooling. However, after reaching max failures we currently delete the job to avoid accumulating jobs so the failure reason would only be observed temporarily.

We could implement a job retention policy for this (we might need it anyway) but I would do that separately.

[kolesnikovae]

We need to ensure atomicity of operations within a shard. Consider a case:

• We remove source blocks.
• We handle query that targets the blocks we just have removed.
• We add compacted blocks.

For me it's not very apparent how we're handling this. I suspect that the query will return incomplete results

[aleks-p]

Good point!

Currently from what I can see in the read path we read data directly from memory. 9442699 locks the shards mutex while replacing source blocks with compacted ones so things should be a bit better. Ideally we should lock a single shard but this makes the code more complex and swapping of blocks should anyway be quite fast.