Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat 1133 files transfer to ACF Titan #1918

Merged
merged 121 commits into from
Sep 16, 2022
Merged
Show file tree
Hide file tree
Changes from 139 commits
Commits
Show all changes
121 commits
Select commit Hold shift + click to select a range
529380c
added ToDo comments where the logic has to be added
raftmsohani Jun 28, 2022
7252738
1133: redis works, celery doesn't connect
raftmsohani Jul 1, 2022
6917bbb
1133: redis works
raftmsohani Jul 1, 2022
d880c6f
1133: redis works with celery installed in scheduling app
raftmsohani Jul 1, 2022
c006861
corrections
msohanifr Jul 5, 2022
c595ffb
changes to use django-configurations
msohanifr Jul 6, 2022
22a20e2
remove white space
msohanifr Jul 6, 2022
6cf5549
delete __init__ file
msohanifr Jul 6, 2022
51c818c
added __init__
msohanifr Jul 6, 2022
c9c7b7b
added a new app, and sub task
msohanifr Jul 11, 2022
81c8ea1
added django_celery_beat
msohanifr Jul 11, 2022
06df917
added the task for upload
msohanifr Jul 14, 2022
06d2f0d
added paramiko
msohanifr Jul 14, 2022
15daafb
added upload task
raftmsohani Jul 19, 2022
661c411
added shceduled tasks
raftmsohani Jul 19, 2022
c0a2652
Working scheduler
raftmsohani Jul 21, 2022
2f30262
FileUpload
raftmsohani Jul 25, 2022
c587b59
FileUpload - Refactored scheduled task to let delay task to be scheduled
raftmsohani Jul 25, 2022
5a2b3b5
FileUpload - added error handling
raftmsohani Jul 25, 2022
628bc2a
1133 - Added env vars, creating directories, putting the file, error …
raftmsohani Jul 28, 2022
384830f
1133 - Removed the scheduled tasks
raftmsohani Jul 29, 2022
7a92e00
1133 - Removed the tasks app
raftmsohani Jul 29, 2022
7f190c2
1133 - Linting
raftmsohani Jul 29, 2022
33276b0
1133 - Linting again
raftmsohani Jul 29, 2022
2f0bc79
1133 - Linting
raftmsohani Jul 29, 2022
4bba520
1133 - Linting
raftmsohani Jul 29, 2022
eaa8e32
1133 - Linting
raftmsohani Jul 29, 2022
01a04a1
Merge branch 'develop' into feat/1133-files-transferred-to-ACF-Titan
raftmsohani Aug 1, 2022
25adae8
added redis-cli apt
raftmsohani Aug 1, 2022
319b46d
added dependency packages
raftmsohani Aug 1, 2022
91384b7
added local redis
raftmsohani Aug 1, 2022
17cad9c
1133 - Added redis-server to gunicorn_start.sh and changed the Celery…
raftmsohani Aug 1, 2022
5956650
1133 - Removed the test files
raftmsohani Aug 1, 2022
ac202a5
1133 - Refactored redis URI to env vars
raftmsohani Aug 1, 2022
e4fb775
1133: absolute path instead of home path
raftmsohani Aug 2, 2022
56559e6
1133: absolute path instead of home path
raftmsohani Aug 2, 2022
d31af21
1133: corrected flower uri
raftmsohani Aug 2, 2022
b518124
1133: corrected gunicorn_start.sh
raftmsohani Aug 2, 2022
dd2d8f2
1133: corrected gunicorn_start.sh
raftmsohani Aug 2, 2022
d1b3130
1133: corrected gunicorn_start.sh
raftmsohani Aug 2, 2022
51c91de
1133: corrected gunicorn_start.sh
raftmsohani Aug 2, 2022
4a83bba
1133: wrapped the task in try/exception
raftmsohani Aug 2, 2022
30d5f7f
1133: additional log
raftmsohani Aug 2, 2022
5597c5b
1133: additional log
raftmsohani Aug 2, 2022
c3b2e65
1133: additional log
raftmsohani Aug 2, 2022
5ca7b04
1133: additional log
raftmsohani Aug 2, 2022
3cb3055
1133: Added error handling, starting redis on cloud.gov script
raftmsohani Aug 4, 2022
5eca1db
1133: Linting
raftmsohani Aug 4, 2022
63ad4c1
1133: Upload Pytest, and linting
raftmsohani Aug 15, 2022
fc7f9ba
1133: linting
raftmsohani Aug 15, 2022
20bb9a3
1133: linting
raftmsohani Aug 15, 2022
8e2fe88
1133: changes for passing the test
raftmsohani Aug 17, 2022
a759ed9
1133: added import
raftmsohani Aug 17, 2022
0a00a4f
1133: add logging
raftmsohani Aug 17, 2022
0584942
1133: add logging
raftmsohani Aug 17, 2022
f7eaeb5
1133: add logging
raftmsohani Aug 17, 2022
e8e1e74
1133: add logging
raftmsohani Aug 17, 2022
f93005e
1133: add logging
raftmsohani Aug 17, 2022
f29e7a9
1133: add logging
raftmsohani Aug 17, 2022
c737ed7
1133: add logging
raftmsohani Aug 17, 2022
98ba904
1133: add logging to celery.py
raftmsohani Aug 17, 2022
c4a899c
1133: change celery start step
raftmsohani Aug 17, 2022
2f19dc5
1133: change celery.py for linting
raftmsohani Aug 17, 2022
21a1418
1133: linting
raftmsohani Aug 17, 2022
878f635
1133: update config
raftmsohani Aug 17, 2022
966b2c6
1133: linting
raftmsohani Aug 17, 2022
899b903
1133: removed scheduling
raftmsohani Aug 17, 2022
a8a8525
1133: permissions
raftmsohani Aug 17, 2022
5913030
1133: permissions
raftmsohani Aug 17, 2022
15343de
1133: changed redis and celery memory
raftmsohani Aug 17, 2022
6b9d748
1133: changed redis and celery memory
raftmsohani Aug 17, 2022
ebd5049
1133: changed redis and celery memory and added commenting
raftmsohani Aug 18, 2022
b1b190d
1133: changed redis and celery memory by adding redis.conf, removed l…
raftmsohani Aug 18, 2022
30b5162
1133: removed cat ACF key file
raftmsohani Aug 18, 2022
a2196a4
1133: settings file: added if statement
raftmsohani Aug 18, 2022
ea861d2
1133: settings file linting
raftmsohani Aug 18, 2022
6cd1f1f
1133: settings file linting
raftmsohani Aug 18, 2022
0753710
1133: changed the pytest_key env var pull up to common.py
raftmsohani Aug 18, 2022
440b32b
1133: common settings file for env vars
raftmsohani Aug 18, 2022
36ea557
1133: common settings file for env vars
raftmsohani Aug 18, 2022
9cccf75
1133: Added an env var
raftmsohani Aug 18, 2022
5ab9c96
1133: Added an env var
raftmsohani Aug 18, 2022
4e52f2b
1133: Removed default for testing
raftmsohani Aug 18, 2022
e82c259
1133: Added end of line replacement for ssh key
raftmsohani Aug 18, 2022
73c5f51
1133: linting
raftmsohani Aug 18, 2022
38d4015
1133: changing private key and replace '_' with \n
raftmsohani Aug 18, 2022
1c6650e
1133: changed to pkey to resolve primary key error
raftmsohani Aug 18, 2022
741dd80
1133: look for key to false
raftmsohani Aug 18, 2022
448c342
1133: disabled private key algorithms
raftmsohani Aug 18, 2022
cebfd50
1133: Added comments to settings files
raftmsohani Aug 19, 2022
6d667aa
1133: linting
raftmsohani Aug 19, 2022
a1d8cb0
python script to change EOL with _
raftmsohani Aug 23, 2022
e8cc8ea
ENV variable for specific ACF TITAN directory
raftmsohani Aug 23, 2022
9117057
linting, chaging upload filename, etc
raftmsohani Aug 24, 2022
49292ba
test file reference change
raftmsohani Aug 24, 2022
ed4c9fc
correct typo
raftmsohani Aug 25, 2022
f1680f0
increase timeout for OWASP
raftmsohani Aug 25, 2022
4dac74d
increase timeout for OWASP
raftmsohani Aug 26, 2022
4681a0e
Added env var for local Redis
raftmsohani Aug 26, 2022
f88eeee
Added env var for owasp scan to use redis
raftmsohani Aug 26, 2022
887a210
Changed redis_server_local to TRUE
raftmsohani Aug 26, 2022
271dc53
corrected the directory config
raftmsohani Aug 26, 2022
c1ea902
Update tdrs-backend/tdpservice/scheduling/test/test_file_upload.py
raftmsohani Aug 26, 2022
37c5a26
Merge branch 'develop' into release/v2.0.0
andrew-jameson Aug 30, 2022
0af229d
Merge pull request #2098 from raft-tech/release/v2.0.0
andrew-jameson Aug 30, 2022
f9c23d2
Merge branch 'develop' into release/v2.0.1
andrew-jameson Aug 30, 2022
2b87189
Merge pull request #2099 from raft-tech/release/v2.0.1
andrew-jameson Aug 30, 2022
82b508e
Update sftp_upload task to use filename
raftmsohani Sep 6, 2022
728a71d
changed stt.filename to return Tribal filename and added comment on T…
raftmsohani Sep 6, 2022
2925995
changed method to use property
raftmsohani Sep 7, 2022
d251370
linting
raftmsohani Sep 7, 2022
642d573
linting + changed test file
raftmsohani Sep 8, 2022
bc08c38
Merge branch 'feat/2090-Transferred-files-must-renamed-ACF-Titan' int…
raftmsohani Sep 9, 2022
6c8f2e6
Update models.py
raftmsohani Sep 12, 2022
6cc737c
corrected Tribe logic
raftmsohani Sep 13, 2022
51e676f
corrected Tribe logic
raftmsohani Sep 13, 2022
ce849d7
force deployment when label is set to QASP REVIEW
n0remac Sep 13, 2022
3bb6517
changed the logic
raftmsohani Sep 13, 2022
1e0e4fb
removed label force
n0remac Sep 13, 2022
c534e51
merged 2090
raftmsohani Sep 13, 2022
af5f23b
Updated the comment
raftmsohani Sep 14, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 51 additions & 37 deletions docs/Product-Strategy/Roadmap-and-Backlog.md
Original file line number Diff line number Diff line change
@@ -1,50 +1,64 @@
_updated January 2022_
_updated July 2022_
# Product Roadmap
Our [roadmap](https://app.mural.co/invitation/mural/raft2792/1629476801275?sender=laurenfrohlich3146&key=5328c2c6-a097-4b3d-bcf7-f2e551a01a72) :lock: represents our latest thinking about the order in which we’ll tackle the various pieces of the overarching problem.
Our [roadmap](https://app.mural.co/t/raft2792/m/raft2792/1649193957647/f8656ffaae4f5dfd47017eb981b04ff2ab7e792d?fromVisitorModal=true&sender=uc514273731b3f70763f30539) : represents our latest thinking about the order in which we’ll tackle the various pieces of the overarching problem.

This roadmap provides a high level plan through Release 4. There might be a few shifts in approach, timing or scope, but in general, these outcomes will be worked on by the team. Metrics for success will be added as they are defined for releases.

Beyond that, we're still discovering and planning on what best serves our users. This doc will continue to be updated as we make decisions and scope releases.
Beyond that, our iterative agile framework will lend itself to feature sets that best serve our users. This doc will continue to be updated as we make decisions and scope releases.

## ATO
Value Delivered: Get approval for the authority to operate and create a production environment.

| Outcome | Status |
| -------- | ------- |
| User can log in using login.gov | Complete
| Users with appropriate privileges can manage users | Complete
| Users can upload data files by section and quarter | Complete
| Users with appropriate privileges can download files that were previously uploaded | Complete
|Create production environment | In Progress

## Release 1: Secure access and upload to TDP
Our first release to production will include the functionality built for ATO (above) and also include secure ways for users to access the system via ACF AMS for ACF users and NextGen XMS/Login.gov for non-ACF users. It is important that these measures are put into place before sensitive production data is uploaded to the system.

| Outcome | Status |
| -------- | ------- |
| TDP is secure and compliant as a system and for all users. | Nearly Complete
| TDP Users have a smooth onboarding and login experience that is secure. | Nearly Complete
| TDP platform is hardened and stable and robust for deployment and live in a production environment. | In Progress


## Release 2: Early Secure Release
This release will allow users to securely upload data into our system in production, replacing a less secure way of doing so, while increasing communication with the users and not increasing burden for OFA Admin staff. This release will allow approximately 8 tribes to pilot the use of login, upload, and download of files, while maintaining TDRS on the backend. While this type of workflow isn't our long term workflow, including it at this point delivers value to the users more quickly.

_Risks:_ The users who would be onboarded to this process would potentially have to learn and adjust with future releases and changes in functionality and workflow. Increased communication with them should help here.
| Outcome | Status |
| -------- | ------- |
| Tribal users are engaged with OFA and communication channels are clear. | In Progress|
| Users know how to access and onboard TDP easily. | In Progress|
| Users can upload and transmit files securely to TDRS database. |Not Started|
| Early Tribe and State users can securely upload reports with TDP instead of using unsecured email. |Not Started|
| Users can resubmit files with error fixes |Not Started|
Approval for the authority to operate.

| Outcome |
| -------- |
| User can log in using login.gov |
| Users with appropriate privileges can manage users |
| Users can upload data files by section and quarter |
| Users with appropriate privileges can download files that were previously uploaded |
| Approval to create a production environment |

## Release 1
This is the first release to the production environment including the ATO and reliable application access to internal and external users in a secure production environment. Additionally it includes user management interface for OFA and System Admin. It is important that these measures are put into place before sensitive production data is uploaded to the system.

| Outcome |
| -------- |
| TDP is secure and compliant as a system and for all users. |
| TDP platform is hardened and stable and robust for deployment and live in a production environment. |
| TDP Users have a secure login experience. |
| Users with appropriate privileges can download files that were previously uploaded |


## Release 2: Pilot Release
This release will allow users to securely upload data into our system in production, replacing a less secure way of doing so, while increasing communication with the users and not increasing burden for OFA Admin staff. This release will allow approximately 15 STTs to pilot the use of login, upload, and download of files, while maintaining TDRS on the backend. While this type of workflow isn't our long term workflow, including it at this point delivers value to the users more quickly and provides the team valuable user feedback to validate and improve the application features.


| Outcome |
| -------- |
| Tribal users are engaged with OFA and communication channels are clear. |
| Users know how to access and onboard TDP easily. |
| Users can upload and transmit files securely to TDRS database. |
| Users can resubmit files with error fixes. |
| Users will receive email notifications on the status of their account and data. |
| Users can differentiate TANF and SSP files upon submission. |
| A research oriented TDP environment is created. |



## Release 3: Parity, Data Parsing, Automated Status, Notifications
This release will parse and validate submitted data and store accepted data in a database. This workflow will include automated email notifications and user-friendly in-app error messages to help users better understand their data errors and submission history. This release will also include onboarding more users and will eventually deprecate the legacy TANF Data Reporting System.
This release will parse and validate submitted data and store accepted data in a database. This workflow will include additional automated email notifications and user-friendly in-app error messages to help users better understand their data errors and submission history. This release will also include Django admin enhancements and onboard all users as it will meet parity with the legacy TANF Data Reporting System.

| Outcome |
| -------- |
| TDP can parse all data sections and types through to the Elasticsearch database. |
| TDP runs validation checks on all submitted files. |
| Users receive easy to understand error messages. |
| Users can view their submission history. |
| All users will submit all of their data through TDP . |
| TDP enhancements for System administrator users. |


## Release 4 and Beyond
In Release 4, we will begin to deliver features beyond parity with TDRS to address user needs, including (but not limited to) pre-submission data cleansing and validation, additional user access management tools, user interface enhancements based on usability testing, and reports and analytics.
In Release 4, we will begin to deliver features beyond parity with TDRS to address user needs, including (but not limited to) additional user access management tools, user interface enhancements based on usability testing, and reports and analytics.

## Backlog
The backlog can be found in the [raft-tech fork of the TANF-app GitHub public repository](https://github.com/raft-tech/TANF-app/issues).
For the most up-to-date status please see the roadmap or backlog at [raft-tech fork of the TANF-app GitHub public repository](https://github.com/raft-tech/TANF-app/issues).
26 changes: 26 additions & 0 deletions docs/Security-Compliance/Security-Controls/at-2/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Awareness and Training (AT)
## AT-2c - Security Awareness Training
The organization provides basic security awareness training to information system(IS) users (including managers, senior executives, and contractors):
a. As part of initial training for new users;
b. When required by information system changes; and
c. [At least every 365 days] thereafter.

The content should:

- provide a basic understanding of the need for information security and user actions to maintain security and to respond to suspected security incidents
- address awareness of the need for operations security. Security awareness techniques can include, for example, displaying posters, offering supplies inscribed with security reminders, generating email advisories/notices from senior organizational officials, displaying logon screen messages, and conducting information security awareness events.
For CSP Only
AT-2(c) [at least annually]

## TDP Implementation

TDP's privileged users, as ACF staff, are required to complete the following trainings annually:
- HHS Records Management Training
- ACF Privacy 101 Training
- ACF Cybersecurity Awareness Training
- 2022 Introductory Role Based Training for IT Administrators

Per TDP's IPT meeting on April 6, 2022, the abovementioned trainings are sufficient for satisfying AT-2 as well as [Related controls](https://csrc.nist.gov/Projects/risk-management/sp800-53-controls/release-search#!/controls?version=4.0&family=AT): AT-01 (security training policy), AT-03 (role-based security), and AT-04 (security training record-keeping). Copies of certificates of completion are managed in [TDP's Sharepoint site](https://hhsgov.sharepoint.com/:f:/r/sites/TANFDataPortalOFA/Shared%20Documents/compliance/TDP%20System%20Admin%20Trainings?csf=1&web=1&e=cwteMG). :lock:


### Related Files
14 changes: 10 additions & 4 deletions docs/Security-Compliance/boundary-diagram.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,19 +6,25 @@

### Data flow

For the MVP, OFA admins will upload data on behalf of STTs and upload data files locally into the web application which will store the files in cloud.gov AWS S3 buckets. Developers will deploy new code through GitHub, initiating the continuous integration process through Circle CI.
Users with `OFA Admin` and (STT) `Data Analyst` roles can upload data on upload data files locally into the web application which will store the files in cloud.gov AWS S3 buckets only after the files are successfully scanned for viruses via [ClamAV](../Technical-Documentation/Architecture-Decision-Record/012-antivirus-strategy.md). Developers will deploy new code through GitHub, initiating the continuous integration process through Circle CI.

### Environments/Spaces

Production, Staging, and Dev spaces provide users with access to a shared location for app development, deployment, and maintenance. The frontend and backend application in each environment is scoped to a space. Roles provide access control for these resources and each space role applies only to a particular space. Developer access to the Dev space does not permit access to Production. Development and Staging environments will not contain any PII.
[Production, Staging, and Dev spaces](../Technical-Documentation/diagrams/tdp-environments.png) provide users with access to a shared location for app development, deployment, and maintenance. The frontend and backend application in each environment is scoped to a space. Roles provide access control for these resources and each space role applies only to a particular space. Developer access to the Dev space does not permit access to Production. Development and Staging environments will not contain any PII.

### User access

All web users, including OFA admins will log into the system through their web browsers using Login.gov and two factor authentication. Developers will also have access to the Dev Space using SSH. STT users will be required to be approved within the application by an administrator.
All web users will log into the system through their web browsers.
- all non-acf users will leverage Login.gov and two factor authentication.
- all ACF users will leverage ACF AMS and authenticate using PIV/CAC. Developers will also have access to the `dev` and `staging` spaces using SSH.

All users will be required to be approved within the application by an administrator.

### Access points

Beyond web based authentication through Login.gov, and developer SSH access to the Dev Space, CircleCI will also have access to the various environments to support deployments.
Beyond web-based authentication through ACF AMS or Login.gov, and developer SSH access to the dev Space, CircleCI will also have access to the various environments to support deployments.

TDP system administrators will trigger CircleCI-based deployments via GitHub.

### Configuration

Expand Down
Loading