As a grantee, I need my uploaded files transferred to ACF Titan

[andrew-jameson]

Description:
Provide a brief background and justification for this issue

As preparation for R2, we need to pilot grantee's data to be transferred to ACF's SFTP server aka ACF Titan. All such data files submitted in the past 24 hours should be transferred to this server in a dated folder (e.g. yyyymmdd) to prevent over-writing of data files since STTs follow standard naming conventions.

Acceptance Criteria:
Create a list of functional outcomes that must be achieved to complete this issue

• Per @ADPennington, we should log all steps within this process to ensure all uploaded files were transferred to ACF TITAN.
• Uploaded file(s) must be securely transferred to designated locations in ACF Titan.
• Testing Checklist has been run and all tests pass
• README is updated, if necessary

Tasks:
Create a list of granular, specific work items that must be completed to deliver the desired outcomes of this issue

• Test SFTP account credentials are valid (both locally and deployed app(s))
• Ensure folders are created for specific date and quarter as noted in the comments below
• Leverage Paramiko for sftp implementation
• Ensure error-handling for failed transfers is returned back to the user.
• Ensure that ClamAV file scan, s3 upload all succeed before the file is uploaded to Titan
• Set up a SFTP client like ClamAV for transferring to Titan (see tdpservice/security/clients.py)
• Run Testing Checklist and confirm all tests pass

Notes:
Add additional useful information, such as related issues and functionality that isn't covered by this specific issue, and other considerations that will be helpful for anyone reading this

• Relevant login credentials are accessible via Cloud.gov environment variables:

• ACFTITAN_HOST
• ACFTITAN_USERNAME
• ACFTITAN_KEY

Supporting Documentation:
Please include any relevant log snippets/files/screen shots

Open Questions:
Please include any questions or decisions that must be made before beginning work or to confidently call this issue complete

• What folder will the files be dropped in? While it just be our $HOME directory?

  • Yes. the credentials used in dev/staging will drop files directly into a folder called tanfdpdev. for prod, the folder is called tanfdp. From there, OFA will direct OCIO to this folder to pick up files for processing via TDRS.

• Will this need to be phased out for production? Which tribes does this apply to? For first draft we can just use all tribes since this will be test data.

  • This will be phased out when TDP has a backend that can process and store data in db.
  • the pilot users have not yet been selected yet (Define List of STTs and plan for V2 pilot #1239)

[ADPennington]

Open Questions:
Please include any questions or decisions that must be made before beginning work or to confidently call this issue complete

• What folder will the files be dropped in? While it just be our $HOME directory?

@abottoms-coder yep. folder name is same as value for ACFTITAN_USERNAME. Please note that we still do need to account for files with the same name being dropped in this folder.

ACFTITAN_USERNAME/

YYYYMMDD/

YYYY-Q1/

file1
file2

YYYY-Q2/

file1

• Will this need to be phased out for production? Which tribes does this apply to? For first draft we can just use all tribes since this will be test data.

UX team will be handling recruitment strategy for this pilot. Agree that for testing purposes, we can use any tribe organization that currently submits data via TDRS. cc: @ttran-hub @dk-ui

[andrew-jameson]

Thanks, @ADPennington! I had neglected the folder structure.

[ADPennington]

6/23/2022 update:

• emailed OCIO re: why we were provided with private keys instead of public per @raftmsohani request. standing by for follow-up and will loop Mo into email chain upon response
• I also noticed today that the staging and prod keys we were provided with and tested are no longer working, so have inquired about getting this addressed as well.

[raftmsohani]

Following up with our meeting @ADPennington with TANF team:

1. The connection is restricted to source IP address range
2. We will create pub keys and send it to TANF team to be added to the server

[raftmsohani]

We are looking into using Celery to schedule and task control for performing backend uploads.

[raftmsohani]

As per @ADPennington : the following directories have to be added on top of directory structure:

• For dev: ../tanfdpdev-data
• For Prod: ../tanfdp-data

[ADPennington]

@raftmsohani and I decided to remove an AC from this ticket because it would require fixing an issue presented in another.

• AC to be removed: Transferred files must be renamed on ACF Titan per convention noted in comments below.
• why: this renaming logic is incorrect. I will include more context in Transferred files must be renamed on ACF Titan #2090

[andrew-jameson]

Ready to demo per sprint review 8/30

[andrew-jameson]

Moved to blocked pending bugfix in #2090