Task queue for Aleph and ingest-file backed by RabbitMQ

[sunu]

Part of fixing alephdata/aleph#2127

• Implements a worker that listens on one or more RabbitMQ queues.
• Queue state per collection is stored in Redis and it tracks finished task count as an integer and pending tasks and running tasks as a set
• Late acknowledgement is enabled in RabbitMQ. So tasks are retried if a worker disconnects without sending an acknowledgement

[sunu]

@Rosencrantz I want to merge this and make a minor release. Since we have added the new worker implementation in a different module, merging this branch won't break anything that use the old worker implementation. What do you think?

[Rosencrantz]

@Rosencrantz I want to merge this and make a minor release. Since we have added the new worker implementation in a different module, merging this branch won't break anything that use the old worker implementation. What do you think?

To be clear, you're looking to do some sort of beta or RC type release as discussed right?

[Rosencrantz]

This looks fine, the work here (in the most part) seems additive rather than changing existing code so I'm not too concerned about it having a detrimental impact. Also this is to be used as a 'beta' for further testing.