Skip to content

Docker image for ckan app powering inventory.data.gov

License

Notifications You must be signed in to change notification settings

GSA/inventory-app

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

inventory-app

CircleCI

Is a Docker-based CKAN development environment for inventory.data.gov. For details on the system architecture, please see our data.gov systems list!

Note: this is currently a work in progress. We're mainly using this to manage the requirements-freeze.txt for production dependencies. Very little works beyond that.

Development

Prerequisites

Getting started

Build and bring up the containers.

make up
make up-with-data [_Gives development environment basic user, organization, and dataset_]

Open CKAN to verify it's working

open http://localhost:5000

If you would like to seed data into the system, examine the test framework (e2e/cypress/support/command.js) for some examples of creating organizations and/or datasets with resources.

docker compose commands

To enter into the app container in interactive mode as root, you will need to run the following:

docker compose exec app /bin/bash

To run a one off command inside the container:

docker compose exec app {command}

Update dependencies

Update the "lock" file for dependencies.

make build requirements

This freezes the requirements at requirements.txt. Run the tests with the updated dependencies.

make test

Live Editing

To edit CKAN or extension code live, the local code needs to be attached via a volume. Add a local extension folder path into the docker-compose.yml file that you would like to edit (see volume section for commented example). After editing the extension/ckan core (see below), run make up (the app will not start appropriately) then run make debug to restart the application with an interactive console.

TODO: tested --reload functionality of gunicorn, but does not work well with paster flag. Hopefully this option improves in the future.

Debugger

Add import ipdb; ipdb.set_trace() as a new line where you would like to start debugging. Then run make debug, and a new instance of the ckan app will be started that has an interactive console. Once the debugger statement is triggered, then a command prompt should display in the console. See documentation for available commands. ipdb is preferred for styling/readability reasons, but pdb will work as well. web-pdb was tested, but has various timing complications of it's own that causes unnecessary complications and failures.

The flask debugger is also imported as a dev requirement and turned on by default in the development.ini file (debug = true), which gives some UI tools on the webpage to parse stack traces and various other examination tools. The behavior is inconsistent, probably due to ckan serving pages as pylons sometimes and flask at others.

Make sure you remove all pdb statements before commiting to any repository!

Tests

make test

The tests utilize cypress. The above command runs in "headless" mode, and debugging capabilities are limited. To fully install and rapidly iterate on tests, install cypress locally with npm.

npm install cypress

Then, you should be able to run make cypress to turn on cypress in interactive mode. If you are using WSL you may need additional setup. Start at this walkthrough, and consider asking a team member for setup help. You will also need to install npx to use the make command.

Please be aware that the tests attempt to clean themselves after each spec file by specifically removing datasets and organizations. If datasets are manually added into the test organization, the dataset and organization may not be removed and this may have unintended consequences. If the system is in a bad state, you should be able to run make clean to restart in a clean environment.

Extension Tests

TODO: add details about running and editing extension tests here.

Deploying to cloud.gov

Copy vars.yml.template to vars.yml, and customize the values in that file. Then, assuming you're logged in for the Cloud Foundry CLI:

Create the database used by datastore. ${app_name} should be the same as what you have in vars.yml.

cf create-service aws-rds micro-psql ${app_name}-datastore --wait

Create the database used by CKAN itself. You have to wait a bit for the datastore DB to be available.

cf create-service aws-rds small-psql ${app_name}-db --wait

Create the s3 bucket for data storage.

cf create-service s3 basic-sandbox ${app_name}-s3 --wait

Create the Redis service for cache

cf create-service aws-elasticache-redis redis-dev ${app_name}-redis --wait

Deploy the Solr instance.

cf push --vars-file vars.yml ${app_name}-solr --wait

Create the secrets service to store secret environment variables. See Secrets below.

Deploy the CKAN app.

cf push --vars-file vars.yml ${app_name}

Ensure the inventory app can reach the Solr app.

cf add-network-policy ${app_name} ${app_name}-solr --protocol tcp --port 8983

You should now be able to visit https://[ROUTE], where [ROUTE] is the route reported by cf app ${app_name}.

Secrets

Tips on managing secrets. When creating the service for the first time, use create-user-provided-service instead of update.

cf update-user-provided-service ${app_name}-secrets -p "CKAN___BEAKER__SESSION__SECRET, CKAN___WTF_CSRF_SECRET_KEY, DS_RO_PASSWORD, NEW_RELIC_LICENSE_KEY, SAML2_PRIVATE_KEY"
Name Description Where to find
CKAN___BEAKER__SESSION__SECRET Session secret for encrypting CKAN sessions. pwgen -s 32 1
CKAN___WTF_CSRF_SECRET_KEY CSRF secret for generating CSRF tokens. pwgen -s 32 1
DS_RO_PASSWORD Read-only password to configure for the datastore user. Initially randomly generated #2839
NEW_RELIC_LICENSE_KEY New Relic License key. New Relic account settings.
SAML2_PRIVATE_KEY Base64 encoded SAML2 key matching the certificate configured for Login.gov Google Drive.

CI configuration

Create a GitHub environment for each application you're deploying. Each GH environment should be configured with secrets from a ci-deployer service account.

Secret name Description
CF_SERVICE_AUTH The service key password.
CF_SERVICE_USER The service key username.

Login.gov integration

We use Login.gov as our SAML2 Identity Provider (IdP). Production apps use the production Login.gov instance while other apps use the Login.gov identity sandbox.

Each year in March, Login.gov rotates their credentials. See our wiki for details.

Our Service Provider (SP) certificate and key are provided in through environment variable and user-provided service.

The Login.gov IdP metadata is stored in file under config/.

License and Contributing

We're so glad you're thinking about re-using and/or contributing to Data.gov!

Before contributing to Data.gov we encourage you to read our CONTRIBUTING guide, our LICENSE, and our README (you are here), all of which should be in this repository. If you have any questions, you can email the Data.gov team at datagov@gsa.gov.