forked from HHS/TANF-app
-
Notifications
You must be signed in to change notification settings - Fork 3
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Issue 2336: Documentation added for CircleCI IR (#2367)
* documentation added for CircleCI IR * Update docs/Technical-Documentation/secret-key-rotation-steps.md syntax * Update docs/Security-Compliance/Incidence-Response/README.md fixed internal ref again * Update docs/Technical-Documentation/secret-key-rotation-steps.md extra underscore * fixed syntax Co-authored-by: Andrew <84722778+andrew-jameson@users.noreply.github.com>
- Loading branch information
1 parent
5981832
commit 70f47b7
Showing
3 changed files
with
148 additions
and
2 deletions.
There are no files selected for viewing
32 changes: 32 additions & 0 deletions
32
docs/Security-Compliance/Incidence-Response/CircleCI-IRP.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,32 @@ | ||
# Response Strategy for CircleCI Security Incident (Dec 2022) | ||
|
||
## Scenario: Security of CI/CD tool compromised | ||
|
||
TDP uses CircleCI as its CI/CD platform, and our team was recently informed of a [security incident](https://circleci.com/blog/january-4-2023-security-alert/) that potentially exposed the secret keys that we store as environment variables on the platform (see notification snapshot below). | ||
|
||
While, there's been no indication from CircleCI or our audit logs of unusual activity on TDP, we were advised to rotate the `production` secret keys. | ||
|
||
**This document captures the steps we followed to respond to this incident, from the point of initial notification to resolution. These steps should be repeated in the event of another incident on platform.** | ||
|
||
![CircleCIDec2022](https://user-images.githubusercontent.com/84722778/210823266-c3874fd7-f3d5-4eaa-bff9-99661db46397.png) | ||
|
||
|
||
|
||
## Mitigation steps | ||
1. Follow [secret key incident response communication protocol](./Secret-Key-Mgmt.md/#communication-protocol-if-secret-keys-are-leaked). | ||
|
||
|
||
2. Identify and remove secret keys stored in `HHS/TANF-app` CircleCI as environment variables. | ||
|
||
``` | ||
CF_USERNAME_PROD + CF_PASSWORD_PROD | ||
JWT_KEY | ||
ACFTITAN_KEY | ||
AMS_CLIENT_SECRET | ||
``` | ||
3. Review [historic logs via cloud.gov](https://cloud.gov/docs/deployment/logs/#web-based-logs-with-historic-log-data) for the backend production app (`tdp-backend-prod`) during the time period of the security incident. Check for anomalous activity and report back to the team. | ||
|
||
4. Follow steps [here](../../Technical-Documentation/secret-key-rotation-steps.md) for rotating abovementioned keys. | ||
|
||
5. Once all keys have been rotated the new ones should be added back to CircleCI and the production deployment workflow should be initiated to restage the production apps and confirm that the system is functioning normally. If there is any unintended downtime impacting system users, OFA product team should be informed immediately so that a contingency plan can be activated until the system is restored. | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters