dataskop-platform

The Data Donation Platform (DDP) of DataSkop, developed by AlgorithmWatch.

This project was initially bootstrapped with Django-Cookie-Cutter but heavily modified.

Development

Development setup

Get the code and create .ENV files for local development.

git clone git@github.com:algorithmwatch/dataskop-platform.git
cd dataskop-platform
cp -r docs/exampleenv .envs

Adjust .envs/.local to you needs. See docs/exampleenv.

Git hook

For consistent code formating, we provide a format script. Since git hooks are not stored with git, you have to configure it on your local machine.

Set up a pre-commit git hook in .git/hooks/pre-commit with the following content.

#!/usr/bin/env bash

# If any command fails, exit immediately with that command's exit status
set -xeo pipefail

docker-compose -f docker-compose.local.yml run --rm django ./scripts/check_format.sh

Ensure the file is executable: chmod +x .git/hooks/pre-commit and the following config is not empty: git config core.hooksPath. If it's empty, set it like this:

git config core.hooksPath .git/hooks

Development with VS Code Development Container

We recommend to use VS Code with the Docker-based VS Code Development Container. As an alternative, see below on how to use Docker without VS Code.

To start the development server: Open a new terminal and see how /start will get executed automatically to open two dev servers (Django, Wepback). To run management command, stop the dev servers and run ./manage.py $command, e.g., ./manage.py makemigrations.

If you add a new VS Code extension, you need to remove this named volume docker volume rm dataskop_extensions. (See more)

To get started

./manage migrate
./manage createsuperuser
./manage seeddata
/start

Development with Docker-Compose

Install and use Docker with Docker-Compose. (See more)

Start development setup:

./local.sh

To to run management commands:

./local.sh manage migrate

Testing

To run the tests, check your test coverage:

coverage run -m pytest
coverage report

To only run tests:

pytest

View sent E-Mail during development

In development, we use MailHog as a local SMTP server with a web interface. View sent emails at: http://localhost:8025

Preview E-Mails

We are using Django-Herald to send emails. To preview the generated texts, go to: http://localhost:8000/herald/

Celery

To run a celery worker:

cd dataskop
celery -A config.celery_app worker -l info -Q high_priority,low_priority,celery

Please note: For Celery's import magic to work, it is important where the celery commands are run. If you are in the same folder with manage.py, you should be right.

Production

Settings via environment varibales

See docs/exampleenv/.production/.django. Also the cookiecutter docs may help for some settings.

Deployment

Docker-Compose (originating from Django-Cookie-Cutter)

Sentry

Setup Sentry to monitor errors. Set the SENTRY_DSN as environment variable.

E-Mail service

Right now, we only support Mailjet but we could make any other email service from django-anymail work.

License

Affero General Public License 3.0