Adjustment

Check commits for detail, below is some major change:

• config: reduce SENTRY_EVENT_RETENTION_DAYS to 32 days
• config: reduce KAFKA_LOG_RETENTION_HOURS to 2 hours
• config: enable reverse SSL proxy config in sentry.conf.py
• nginx: listen on host 127.0.0.1, not 0.0.0.0
• nginx: pass host nginx X-Forward-* headers

Reclaim disk spaces

• run docker-compose --file docker-compose.yml exec worker sentry cleanup --days 30 from repo root (may not work)
• check docker log size ls -al {docker-data-root}/containers/*/*.log & truncate with echo "" > .../AAA/BBB.log (~10G after months of running)
• check: https://forum.sentry.io/t/sentry-disk-cleanup-kafka/11337

sentry container keep restarting after some high CPU running

• from log found error KafkaError{code=OFFSET_OUT_OF_RANGE, related container:
  • {...}_snuba-subscription-consumer-events_1
  • {...}_snuba-subscription-consumer-transactions_1
• from google search: https://forum.sentry.io/t/kafka-offset-issue-snuba-subscription-consumer-events/12825 then check: getsentry#478 (comment)
• Fix by reset both snuba-events-subscriptions-consumers & snuba-transactions-subscriptions-consumers with command like: kafka-consumer-groups --bootstrap-server 127.0.0.1:9092 --group snuba-events-subscriptions-consumers --topic events --reset-offsets --to-latest --execute and: kafka-consumer-groups --bootstrap-server 127.0.0.1:9092 --group snuba-transactions-subscriptions-consumers --topic events --reset-offsets --to-latest --execute

dockerd log rotate config

• default dockerd will keep all json log as long as the container exists, to prevent disk bust, add config in /etc/docker/daemon.json: (https://docs.docker.com/config/containers/logging/json-file/#usage)

  {
    ...
    "log-driver": "json-file",
    "log-opts": { "max-size": "8m", "max-file": "2" }
  }
  

• to apply config to container, all containers need to be re-created, and verify config by docker container inspect {hash}:

    "HostConfig": {
      ...
      "LogConfig": {
        "Type": "json-file",
        "Config": {
          "max-file": "2",
          "max-size": "8m"
        }
      },
  

Other tips

• Found in install.sh: you can create user from CLI, instead of sending mails with: docker-compose run --rm web createuser
• Mails may surly end up in the spam folder, check there if you're being invited through email.
• DO NOT RUN docker volume prune, since the volume is not linked, and this will remove them all, also delete/reset some of the volumes may cause the whole system stop working. So either do full backup, or prepare for fast clean reset.

host nginx config

  server {
    # https with nginx
    listen                      443;
    ssl                         on;
    ssl_certificate             /PATH/TO/CERT.pem;
    ssl_certificate_key         /PATH/TO/CERT.pem;

    client_max_body_size        64M;                # allow large uploads of files

    # https://github.com/getsentry/onpremise/issues/554
    proxy_set_header            Host                $host;
    proxy_set_header            X-Real-IP           $remote_addr;
    proxy_set_header            X-Forwarded-For     $proxy_add_x_forwarded_for;
    proxy_set_header            X-Forwarded-Proto   "https";

    # sentry
    location / {                # main entry
      proxy_pass                http://127.0.0.1:9000/;
    }
    
    # quick hack to mute error spamming
    # deny                      1.2.3.4;
    # deny                      1.2.3.4;
  }

trysen.js

Used to custom rate limit issue POST requests, so the sentry docker setup will not get huge backlog, and grind the machine to a stall with the unrestrained memory hogging (in redis & kafka mostly)

Check the source code, and some config scripts in ./x-trysen/ for details.

---

Self-Hosted Sentry 20.11.1

Official bootstrap for running your own Sentry with Docker.

Requirements

• Docker 19.03.6+
• Compose 1.24.1+

Minimum Hardware Requirements:

• You need at least 2400MB RAM

Setup

To get started with all the defaults, simply clone the repo and run ./install.sh in your local check-out.

If you like trying out new things, you can run SENTRY_PYTHON3=1 ./install.sh instead to use our brand new Python 3 images. Keep in mind that Python 3 support is experimental at this point

During the install, a prompt will ask if you want to create a user account. If you require that the install not be blocked by the prompt, run ./install.sh --no-user-prompt.

There may need to be modifications to the included example config files (sentry/config.example.yml and sentry/sentry.conf.example.py) to accommodate your needs or your environment (such as adding GitHub credentials). If you want to perform these, do them before you run the install script and copy them without the .example extensions in the name (such as sentry/sentry.conf.py) before running the install.sh script.

The recommended way to customize your configuration is using the files below, in that order:

• config.yml
• sentry.conf.py
• .env w/ environment variables

We currently support a very minimal set of environment variables to promote other means of configuration.

If you have any issues or questions, our Community Forum is at your service! Everytime you run the install script, it will generate a log file, sentry_install_log-<ISO_TIMESTAMP>.txt with the output. Sharing these logs would help people diagnose any issues you might be having.

Versioning

If you want to install a specific release of Sentry, use the tags/releases on this repo.

We continously push the Docker image for each commit made into Sentry, and other services such as Snuba or Symbolicator to our Docker Hub and tag the latest version on master as :nightly. This is also usually what we have on sentry.io and what the install script uses. You can use a custom Sentry image, such as a modified version that you have built on your own, or simply a specific commit hash by setting the SENTRY_IMAGE environment variable to that image name before running ./install.sh:

SENTRY_IMAGE=getsentry/sentry:83b1380 ./install.sh

Note that this may not work for all commit SHAs as this repository evolves with Sentry and its satellite projects. It is highly recommended to check out a version of this repository that is close to the timestamp of the Sentry commit you are installing.

Event Retention

Sentry comes with a cleanup cron job that prunes events older than 90 days by default. If you want to change that, you can change the SENTRY_EVENT_RETENTION_DAYS environment variable in .env or simply override it in your environment. If you do not want the cleanup cron, you can remove the sentry-cleanup service from the docker-compose.ymlfile.

Securing Sentry with SSL/TLS

If you'd like to protect your Sentry install with SSL/TLS, there are fantastic SSL/TLS proxies like HAProxy and Nginx. Our recommendation is running and external Nginx instance or your choice of load balancer that does the TLS termination and more. Read more over at our productionalizing self-hosted docs.

Updating Sentry

You need to be on at least Sentry 9.1.2 to be able to upgrade automatically to the latest version. If you are not, upgrade to 9.1.2 first by checking out the 9.1.2 tag on this repo.

We recommend (and sometimes require) you to upgrade Sentry one version at a time. That means if you are running 20.6.0, instead of going directly to 20.8.0, first go through 20.7.0. Skipping versions would work most of the time, but there will be times that we require you to stop at specific versions to ensure essential data migrations along the way.

Pull the version of the repository that you wish to upgrade to by checking out the tagged release of this repo. Make sure to check for any difference between the example config files and your current config files in use. There might be new configuration that has to be added to your adjusted files such as feature flags or server configuration.

The included install.sh script is meant to be idempotent and to bring you to the latest version. What this means is you can and should run install.sh to upgrade to the latest version available. Remember that the output of the script will be stored in a log file, sentry_install_log-<ISO_TIMESTAMP>.txt, which you may share for diagnosis if anything goes wrong.

For more information regarding updating your Sentry installation, please visit our documentation.

Resources

• Documentation
• Bug Tracker
• Community Forums