feat 1133 files transfer to ACF Titan

[raftmsohani]

Summary of Changes

Provide a brief summary of changes
Pull request closes #1133

How to Test

List the steps to test the PR

To test the upload task:

1. A local SFTP server is created
2. A test file is uploaded
3. Check the uploaded file

As can be seen below, when the file is uploaded, the renamed file is going to be logged in the Legacy File transfer section with all the information and a link to the uploaded file.

Add the following lines to .env file:

# Whether or not to use Localstack for S3 storage
USE_LOCALSTACK=yes

ACFTITAN_HOST=
ACFTITAN_KEY=
ACFTITAN_USERNAME=

REDIS_URI=redis://redis-server:6379
REDIS_SERVER_LOCAL=TRUE

ACFTITAN_SFTP_PYTEST=-----BEGIN RSA PRIVATE KEY-----_MIIEpQIBAAKCAQEAsf16LYNvP6vAcxZhSueDCxIpgUbOr4Hj6f37B/nAU+EIbU+3_i54NXTMmtDjG+xzRNOH110h3o7P3A5B7cFu+IlG0AHzlsBbfO/JAGzcLbtKnxDwn_/C5gFpUNKWqvj9kgqRmrmqPLl7UjfYxZVUSRD8Gh/zIONvTl3C0/EsPn/77e0YH/_wC7QDyg4rikK9Cksekp9PiskgkeqCIJTk0ETiGGhOODUBOWrhHCZBXAvbbhBe/EV_diU0IUKU869Ci0dKKP3h6YjeDhNwrkNa7jZnXPYPRHdsmWi+RK70dsoZe/MkK2GE_yucjsh8gCHWvltlJ7NQJ0fkrcrEdTp5K9ddVAQIDAQABAoIBAHMMDYIHt8vm/1Ek_gSTCeiCYz70hAI3PHntr0Gv6UgelqCXH5jLXqXm5i2XoYS2FFfDhsV9DFxn4REzI_ghFBK6fROdEq6eglEIwV7Lvqm3g5r3kXDR1i+HcARu6jZ/FJ/mNvFU2yW/Gmgtcu_9bs4w670kPp751Y8e6sSj/dYK8hRCoQlN7B5Ald8bZpVcZA/aVmJLfGvzNBY0D+x_wsTRvZn5Q7XG/BJL0+ZWJoSEi4SPrkENUdlHueRHpwPlzBuiGmksrpYvlnnqwvZD_ld90hxkGgnDCmReTzBRKqSPh3wab+OKzePVrLGDFzhd4V5rSsOJl01uOHac/8QLZ_vHCQEDECgYEA2VHvIkmwu+BpDLXAmD8aiKfzcrEGT32SDVZaNdRcrADXKbtVmSKN_I/UCmtfwNVHZk/FMPqarnfWtqv0B62eovI8qaYNCzNUFCKW3aUcACnnMc9HKlYsf_25jJF2X3JzMv319xYnniy77fr6RsxFw2l/7J3dg0tNeFUvPVMPt8trUCgYEA0at/_eqzMowG/8dwFubY3t/hlwPT4HuvD1A+/yC5jThU280l7lCwRjk9Eh9024Ihy0XuP_+m6af1IMzhg5fYqrF7sU7Tqxx2m+47J5iQZ+m2n6jSLDslqTcAYKj9XOHMJ3/ANe_+eZKpXxpTstfX7A5Yug9a58PjleMjdm1VpDzKJ0CgYEAhOem58FJZJ0JocxFzNZK_0+hi6nF4+oRBHgcBhIorYsXg0JTQ9KY8yxC8VxZYwUMdXWzkxCwKKMBnRXsWAXGT_sD2eIok0ATEFsxQl5yyUydNTRkG3M12yTgpScQza6g5T6LfmD+Oa4CALjM9x9WSv_vqUDr7jaAv8Len/EkgA7dUECgYEAx5X5/pvZHF5BCgkIhjTXq09QBTLrsft56Tao_t/S4YQ6+xS4w7eZZO99m+/HvGCOrMI/viVOZzBMdz12t9Dx5C1jx3bTeoFWf+X3e_RTqicGycrZbnNLMV4DBQA4Vh82yG7KWE1luKuSbJ09CyVBMbPXSXawf5teTPDgSs_ot/OJ90CgYEAvWYQHspz7irepznaDSwnUqbfDPNQr16IN6s3KnR/MGKl6q54uQ80_2U/k4d3KxHMAFEUVBXSBOwFswoz0ZgR9bzMUDsoSwtnfXoyFvG1GiW20diNpjPIg_toSg3kO4adt6TdPPc1hRlbKah+Ft20qVhvaPAmQDJD3Ow8we7UAZIas=_-----END RSA PRIVATE KEY-----

• TODO 1: The upload task has to be tested on Cloud.gov and actual Titan file server
• TODO 2: Functional test using pytest-sftpserver

Deliverables

More details on how deliverables herein are assessed included here.

Deliverable 1: Accepted Features

Checklist of ACs:

• Per @ADPennington, we should log all steps within this process to ensure all uploaded files were transferred to ACF TITAN.
• Uploaded file(s) must be securely transferred to designated locations in ACF Titan.
• Testing Checklist has been run and all tests pass
• README is updated, if necessary

• lfrohlich and/or adpennington confirmed that ACs are met.

Deliverable 2: Tested Code

• Are all areas of code introduced in this PR meaningfully tested?
  • If this PR introduces backend code changes, are they meaningfully tested?
  • If this PR introduces frontend code changes, are they meaningfully tested?
• Are code coverage minimums met?
  • Frontend coverage: [insert coverage %] (see CodeCov Report comment in PR)
  • Backend coverage: [insert coverage %] (see CodeCov Report comment in PR)

Deliverable 3: Properly Styled Code

• Are backend code style checks passing on CircleCI?
• Are frontend code style checks passing on CircleCI?
• Are code maintainability principles being followed?

Deliverable 4: Accessible

• Does this PR complete the epic?
• Are links included to any other gov-approved PRs associated with epic?
• Does PR include documentation for Raft's a11y review?
• Did automated and manual testing with iamjolly and ttran-hub using Accessibility Insights reveal any errors introduced in this PR?

Deliverable 5: Deployed

• Was the code successfully deployed via automated CircleCI process to development on Cloud.gov?

Deliverable 6: Documented

• Does this PR provide background for why coding decisions were made?
• If this PR introduces backend code, is that code easy to understand and sufficiently documented, both inline and overall?
• If this PR introduces frontend code, is that code easy to understand and sufficiently documented, both inline and overall?
• If this PR introduces dependencies, are their licenses documented?
• Can reviewer explain and take ownership of these elements presented in this code review?

Deliverable 7: Secure

• Does the OWASP Scan pass on CircleCI?
• Do manual code review and manual testing detect any new security issues?
• If new issues detected, is investigation and/or remediation plan documented?

Deliverable 8: User Research

Research product(s) clearly articulate(s):

• the purpose of the research
• methods used to conduct the research
• who participated in the research
• what was tested and how
• impact of research on TDP
• (if applicable) final design mockups produced for TDP development

[raftmsohani]

To run Celery, run:
celery -A tdpservice.scheduling worker -l info

[n0remac]

I accidentally added commits from my branch so I removed them and force pushed.

[ADPennington]

thanks @raftmsohani. successfully tested this using dev and prod credentials to titan 🚀

Please note:

• the pilot files for states and tribes are properly renamed when landing on titan.
• any section 4 files submitted by STTs that do not have a "Stratum Data" section included in the STT model's filenames field have an invalid name upon landing on titan, as expected.
• any sections in the STT model's filenames field that are preceded by "SSP" will have invalid file names upon landing on titan, as expected at this stage. As an STT data analyst, I need to tell TDP which data report I am uploading #1755 will resolve this.
• territories' files have proper file names upon landing on titan, except in cases where section 4 data is erroneously submitted. this is expected.
• i have added the prod key value pairs to prod backend app in cloud.gov in preparation for merge to master.