Implement task queuing / scheduling system in Django

[jtwillis92]

Description:
In order to support ingesting uploaded data into Elasticsearch we will need to implement a queuing / task scheduling system with our Django backend. By parsing the data first and then placing it in a queue to be ingested into Elasticsearch we would be able to asynchronously process the data ingestion and let the user know beforehand if the data could not be parsed without making them wait for the ingestion operation to complete. This would also increase system resiliency, as failed jobs could be retried until successful. Otherwise, if an error were to happen while parsed data was being ingested into Elasticsearch the data would end up getting dropped with no way to recover.

By using a queueing system to buffer the messages between Django and ES we would be able to implement things like retry loops and give ourselves more troubleshooting data in the event of problems. An excellent option for doing this in Django is django-rq which utilizes Redis to queue messages and allows creation of jobs with arbitrary python code. This would also be useful for many other future efforts and Redis is supported by Cloud.gov

The ideal solution would integrate with Django admin to show all of the queued jobs and allow filtering for failed or invalid jobs. (Django RQ does this)

This could also be used for other scheduled tasks such as Nightly SFTP Transfers and the nightly OWASP ZAP scans currently performed by Circle CI.

Acceptance Criteria:
Create a list of functional outcomes that must be achieved to complete this issue

• Asynchronous tasks can be queued or scheduled from Django
• Queued tasks are tracked in Django Admin
• Failed jobs have a retry loop or other mechanism to alert system admins.
• Testing Checklist has been run and all tests pass
• README is updated, if necessary

Tasks:
Create a list of granular, specific work items that must be completed to deliver the desired outcomes of this issue

• Integrate a queuing system with Django
• Update any relevant Terraform configurations to deploy any required services (ie. Redis).
• Add unit tests that assert job queuing functionality
• Run Testing Checklist and confirm all tests pass

Notes:

Supporting Documentation:

Open Questions:

[jtwillis92]

@jorgegonzalez requested to work on this once it has been refined and scheduled for a sprint.

[andrew-jameson]

If #1784 and/or it's follow-up is complete, this would serve as a good test tasks for this sytem.

[stevenino]

Completed by #1133