You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Description:
In order to support ingesting uploaded data into Elasticsearch we will need to implement a queuing / task scheduling system with our Django backend. By parsing the data first and then placing it in a queue to be ingested into Elasticsearch we would be able to asynchronously process the data ingestion and let the user know beforehand if the data could not be parsed without making them wait for the ingestion operation to complete. This would also increase system resiliency, as failed jobs could be retried until successful. Otherwise, if an error were to happen while parsed data was being ingested into Elasticsearch the data would end up getting dropped with no way to recover.
By using a queueing system to buffer the messages between Django and ES we would be able to implement things like retry loops and give ourselves more troubleshooting data in the event of problems. An excellent option for doing this in Django is django-rq which utilizes Redis to queue messages and allows creation of jobs with arbitrary python code. This would also be useful for many other future efforts and Redis is supported by Cloud.gov
The ideal solution would integrate with Django admin to show all of the queued jobs and allow filtering for failed or invalid jobs. (Django RQ does this)
This could also be used for other scheduled tasks such as Nightly SFTP Transfers and the nightly OWASP ZAP scans currently performed by Circle CI.
Acceptance Criteria: Create a list of functional outcomes that must be achieved to complete this issue
Asynchronous tasks can be queued or scheduled from Django
Queued tasks are tracked in Django Admin
Failed jobs have a retry loop or other mechanism to alert system admins.
Testing Checklist has been run and all tests pass
README is updated, if necessary
Tasks: Create a list of granular, specific work items that must be completed to deliver the desired outcomes of this issue
Integrate a queuing system with Django
Update any relevant Terraform configurations to deploy any required services (ie. Redis).
Add unit tests that assert job queuing functionality
Run Testing Checklist and confirm all tests pass
Notes:
Supporting Documentation:
Open Questions:
The text was updated successfully, but these errors were encountered:
Description:
In order to support ingesting uploaded data into Elasticsearch we will need to implement a queuing / task scheduling system with our Django backend. By parsing the data first and then placing it in a queue to be ingested into Elasticsearch we would be able to asynchronously process the data ingestion and let the user know beforehand if the data could not be parsed without making them wait for the ingestion operation to complete. This would also increase system resiliency, as failed jobs could be retried until successful. Otherwise, if an error were to happen while parsed data was being ingested into Elasticsearch the data would end up getting dropped with no way to recover.
By using a queueing system to buffer the messages between Django and ES we would be able to implement things like retry loops and give ourselves more troubleshooting data in the event of problems. An excellent option for doing this in Django is django-rq which utilizes Redis to queue messages and allows creation of jobs with arbitrary python code. This would also be useful for many other future efforts and Redis is supported by Cloud.gov
The ideal solution would integrate with Django admin to show all of the queued jobs and allow filtering for failed or invalid jobs. (Django RQ does this)
This could also be used for other scheduled tasks such as Nightly SFTP Transfers and the nightly OWASP ZAP scans currently performed by Circle CI.
Acceptance Criteria:
Create a list of functional outcomes that must be achieved to complete this issue
Tasks:
Create a list of granular, specific work items that must be completed to deliver the desired outcomes of this issue
Notes:
Supporting Documentation:
Open Questions:
The text was updated successfully, but these errors were encountered: