Issue 2336: Documentation added for CircleCI IR (#2367)

* documentation added for CircleCI IR

* Update docs/Technical-Documentation/secret-key-rotation-steps.md

syntax

* Update docs/Security-Compliance/Incidence-Response/README.md

fixed internal ref again

* Update docs/Technical-Documentation/secret-key-rotation-steps.md

extra underscore

* fixed syntax

Co-authored-by: Andrew <84722778+andrew-jameson@users.noreply.github.com>

docs/Security-Compliance/Incidence-Response/CircleCI-IRP.md

@@ -0,0 +1,32 @@
+# Response Strategy for CircleCI Security Incident (Dec 2022)
+
+## Scenario: Security of CI/CD tool compromised
+
+TDP uses CircleCI as its CI/CD platform, and our team was recently informed of a [security incident](https://circleci.com/blog/january-4-2023-security-alert/) that potentially exposed the secret keys that we store as environment variables on the platform (see notification snapshot below).  
+
+While, there's been no indication from CircleCI or our audit logs of unusual activity on TDP, we were advised to rotate the `production` secret keys. 
+
+**This document captures the steps we followed to respond to this incident, from the point of initial notification to resolution. These steps should be repeated in the event of another incident on platform.**
+
+![CircleCIDec2022](https://user-images.githubusercontent.com/84722778/210823266-c3874fd7-f3d5-4eaa-bff9-99661db46397.png)
+
+
+
+## Mitigation steps
+1.  Follow [secret key incident response communication protocol](./Secret-Key-Mgmt.md/#communication-protocol-if-secret-keys-are-leaked).
+
+
+2. Identify and remove secret keys stored in `HHS/TANF-app` CircleCI as environment variables. 
+
+```
+CF_USERNAME_PROD + CF_PASSWORD_PROD
+JWT_KEY
+ACFTITAN_KEY
+AMS_CLIENT_SECRET
+```
+3. Review [historic logs via cloud.gov](https://cloud.gov/docs/deployment/logs/#web-based-logs-with-historic-log-data) for the backend production app (`tdp-backend-prod`) during the time period of the security incident. Check for anomalous activity and report back to the team. 
+
+4. Follow steps [here](../../Technical-Documentation/secret-key-rotation-steps.md) for rotating abovementioned keys.  
+
+5. Once all keys have been rotated the new ones should be added back to CircleCI and the production deployment workflow should be initiated to restage the production apps and confirm that the system is functioning normally. If there is any unintended downtime impacting system users, OFA product team should be informed immediately so that a contingency plan can be activated until the system is restored. 
+

docs/Security-Compliance/Incidence-Response/README.md

@@ -5,4 +5,5 @@ This subdirectory contains documentation that describes our protocols for respon
 
 ## Table of Contents
 * **[Secret Key Management](./Secret-Key-Mgmt.md)** - lays out TDP's standards for minimizing the risk of leaking secret keys (i.e. sensitive information that--if exposed--opens the door for unauthorized access to the system and/or data therein).
-* **[System Admin Account Restoration](./Sys-Admin-Acct-Mgmt.md)** - lays out steps for restoring permissions for a TDP sys admin if their account has been compromised. 
+* **[System Admin Account Restoration](./Sys-Admin-Acct-Mgmt.md)** - lays out steps for restoring permissions for a TDP sys admin if their account has been compromised. 
+* **[CircleCI Secret Key Rotation Tabletop](./CircleCI-IRP.md)** - lays out steps taken in response to CircleCI's security incident in December 2022. 

docs/Technical-Documentation/secret-key-rotation-steps.md

@@ -3,8 +3,10 @@
 ## Context
 
 To maintain good security, we will periodically rotate the following secret keys, which are used to control authentication and authorization to our application:
+- CF deployer keys (_for continuous delivery_)
 - JWT keys (_external user auth_)
 - ACF AMS keys (_internal user auth_)
+- ACF Titan server keys (_for file transfers between TDP and TDRS_)
 - Django secret keys ([_cryptographic signing_](https://docs.djangoproject.com/en/4.0/topics/signing/#module-django.core.signing))
 
 This document outlines the process for doing this for each set of keys. 
@@ -16,6 +18,61 @@ This document outlines the process for doing this for each set of keys.
 - As of June 2022, CircleCI supplies environment variable key-value pairs to multiple environments (e.g. Raft's CircleCI deploys applications to dev and staging environments). The values from CircleCI are expected to be unique per environment, so until [#1826](https://github.com/raft-tech/TANF-app/issues/1826) is researched and addressed, these values will need to be manually corrected in cloud.gov immediately following the execution of the [`<env>-deployment` CircleCI workflow](../../.circleci/config.yml). This workaround applies to backend applications in the TDP staging environment. 
 
 ## Rotation procedures
+
+**<details><summary>CF deployer keys</summary>**
+There are unique cloud foundry (CF) credentials for each cloud.gov space (`tanf-dev`, `tanf-staging`, `tanf-prod`) for deployments. These are stored in the `tanf-keys` service instance in cloud.gov. The steps below should be followed to rotate the credentials for the relevant space(s). **Note**: `<ENV>` := `DEV`, `STAGING`, or `PROD`.
+
+0. verify existing credentials for `deployer` key in `tanf-keys` service instance (these are the values for `CF_USERNAME_<ENV>` and `CF_PASSWORD_<ENV>` in circleci project settings
+
+```
+# target env space
+cf target -o hhs-acf-ofa -s tanf-<env>
+
+# verify deployment credentials
+cf service-key tanf-keys deployer
+```
+
+1. remove the current username associated the the `deployer` `tanf-keys`  service instance from cloud.gov space (this is the same value as `CF_USERNAME_<env>` ). This task can also be done from the dashboard.
+
+```
+cf delete-user <<insert USERNAME value for deployer key>>
+```
+
+
+2. delete the `deployer` service key associated with `tanf-keys` service instance ([reference](https://docs.cloudfoundry.org/devguide/services/service-keys.html))
+
+```
+# delete
+cf delete-service-key tanf-keys deployer
+
+# verify deletion
+cf service-keys tanf-keys 
+```
+
+3. create new `deployer` service key within `tanf-keys` instance (reference link above)
+
+```
+cf create-service-key tanf-keys deployer
+```
+
+4. add username for newly generated `deployer` service key to space as a user to relevant space and assign as an org user and space developer. This task can also be done from the dashboard.
+
+```
+# add user
+cf create-user <<insert USERNAME value for deployer key>>
+
+# add as a user to org
+cf set-org-role <<insert USERNAME value for deployer key>> hhs-acf-ofa OrgUser
+
+# add as developer to prod space
+cf set-space-role <<insert USERNAME value for deployer key>> hhs-acf-ofa tanf-prod SpaceDeveloper
+```
+
+5. Confirm that the new deployment credentials work in CircleCI (re-run deployment workflow after adding `CF_USERNAME_<env>` and `CF_PASSWORD_<env>` back to CircleCI with rotated values) 
+
+
+</details>
+
 **<details><summary>JWT Keys</summary>**
 
 #### The following steps are applicable for **lower environments (dev and staging) _only_**. See [here](#Production-Environment) for prod environment procedure. 
@@ -89,14 +146,70 @@ Production environment key generation, change requests, and distribution will be
 </details>
 
 **<details><summary>ACF AMS Keys</summary>**
-The ACF AMS Ops team manages these credentials for all environments (dev, staging, and prod), so we will need to submit a service request ticket whenever we need keys rotated. 
+The ACF OCIO Ops team manages these credentials for all environments (dev, staging, and prod), so we will need to submit a service request ticket whenever we need keys rotated. 
 
 Service requests tickets must be submitted by Government-authorized personnel with Government computers and PIV access (e.g. Raft tech lead for lower environments and TDP sys admins for production environment). Please follow the procedures below:
 
 1. Submit request tickets from government-issued email address and use the email template located on **page 2** of [this document.](https://hhsgov.sharepoint.com/:w:/r/sites/TANFDataPortalOFA/Shared%20Documents/compliance/Authentication%20%26%20Authorization/ACF%20AMS%20docs/OCIO%20OPERATIONS%20REQUEST%20TEMPLATES.docx?d=w5332585c1ecf49a4aeda17674f687154&csf=1&web=1&e=aQyIPz) cc OFA tech lead on lower environment requests. 
 2. Update environment variables in CircleCI and relevant cloud.gov backend applications after ticket completed by OCIO. [Restage applications](https://cloud.gov/docs/deployment/app-maintenance/#restaging-your-app).
 </details>
 
+**<details><summary>ACF Titan Server Keys</summary>**
+The ACF OCIO Ops team manages these credentials for all environments (dev, staging, and prod), so we will need to submit a service request ticket whenever we need keys rotated. 
+
+Service requests tickets must be submitted by Government-authorized personnel with Government computers and PIV access (e.g. Raft tech lead for lower environments and TDP sys admins for production environment). Please follow the procedures below:
+
+1. Generate new public/private key pair
+
+Below is an example of how to generate new titan public/private key pair from _Git BASH for Windows_. Two files called `filename_where_newtitan_keypair_saved` are created: one is the _private_ key and the other is a _public_ key (the latter is saved with a _.pub_ extention).
+(note: the info below is not associated with any real keys)
+
+```
+$ ssh-keygen -t rsa -b 4096
+Generating public/private rsa key pair.
+
+Enter file in which to save the key (/c/Users/username/.ssh/id_rsa): filename_where_newtitan_keypair_saved
+
+Enter passphrase (empty for no passphrase):
+
+Enter same passphrase again:
+
+Your identification has been saved in filename_where_newtitan_keypair_saved
+
+Your public key has been saved in filename_where_newtitan_keypair_saved.pub
+
+The key fingerprint is:
+SHA256:BY6Nl0hCjIrI9yZMBGH2vbDFLCTq2DsFQXQTmLydwjI 
+
+The key's randomart image is:
++---[RSA 4096]----+
+| X*B*.. .        |
+|+ O+=+ * o       |
+|=oo* *+ = .      |
+|Eo++B .. .       |
+|.+=oo.  S        |
+|   = o           |
+|  o o            |
+|   .             |
+|                 |
++----[SHA256]-----+
+```
+
+2. Submit request tickets from government-issued email address and use the email template located on **page 2** of [this document.](https://hhsgov.sharepoint.com/:w:/r/sites/TANFDataPortalOFA/Shared%20Documents/compliance/Authentication%20%26%20Authorization/ACF%20AMS%20docs/OCIO%20OPERATIONS%20REQUEST%20TEMPLATES.docx?d=w5332585c1ecf49a4aeda17674f687154&csf=1&web=1&e=aQyIPz) cc OFA tech lead on lower environment requests. 
+
+The request should include:
+- the titan service account name (i.e. `tanfdp` for prod; `tanfdpdev` for dev/staging) 
+- the newly generated public key from `filename_where_newtitan_keypair_saved.pub`
+
+3. When OCIO confirms that the change has been made, add the private key from `filename_where_newtitan_keypair_saved` to CircleCI as an environment variable. The variable name is `ACFTITAN_KEY`. **Please note**: the value needs must be edited before adding to CircleCI. It should be a one-line string with underscores ("_") replacing the spaces at the end of every line. See example below:
+
+```
+-----BEGIN OPENSSH PRIVATE KEY-----_somehashvalue_-----END OPENSSH PRIVATE KEY-----
+```
+
+4. Re-run the deployment workflow from CircleCI and confirm that the updated key value pair has been added to the relevant cloud.gov backend application.
+</details>
+
 **<details><summary>Django secret keys</summary>**
 
 `DJANGO_SECRET_KEY` is dynamically generated since [#1151](https://github.com/raft-tech/TANF-app/pull/1151), so all that needs to be done to rotate this key in any environment is to re-run the relevant environment's deployment workflow in CircleCI. These are as follows: