Improve Horizon Support

[jaydrogers]

Background

I've been working with @nckrtl on optimizing Horizon support

Related Discord Discussion

https://discord.com/channels/910287105714954251/1235899514867683430

Research Solution

Beta Give feedback

1. Ensure Horizon fix is stable
2. Test entrypoint script

Documentation

Beta Give feedback

1. Show example of Laravel Horizon
2. Show differences between Laravel Horizon and Laravel Queues (example discussion)
3. Create video how Horizon works
4. Post an update on Horizon can restart gracefully but cannot stop gracefully laravel/horizon#663

[jaydrogers]

@nckrtl: I just added some ToDos for myself above. Any other ideas that need to be added to this list, let me know.

Next week I will be partially out of the office, but I will start diving into this when I return the week after 👍

[nckrtl]

Looks complete to me. Still need to verify assumptions as discussed on Discord. Will give it this weekend a try.

[nckrtl]

Alright, I did a test and can confirm everything is working as expected with setting the horizon docker service order to start-first in docker-compose, but there are a few more requirements which need to be set properly in the Laravel project:

• In the horizon config file horizon.php you must set the timeout key of your queue config to an higher amount than your job may run. So if your job may run 5 minutes (300 seconds) the timeout should be at least set to 301

• In queue.php the retry_after key on your redis connection should be set higher than the timeout amount you set in your horizon config. Else your job will get retried too soon and it will get interrupted before it finished. So setting it to 302 should be sufficient but i'd recommend adding some margin (same goes for the timeout btw.

• In docker-compose the stop_grace_period should be set to an high enough amount, with 5 minute jobs 300 seconds should be sufficient but it wouldn't hurt to set this to 1 hour.

Because of the start-first a new container will start upon a new deployment while the old one will stay active and finish until horizon terminates gracefully. Made a video to demo it all. Might take a bit before the quality is bumped enough by YT for it to be watchable.

https://www.youtube.com/watch?v=NJ0Vj9nwgf4

[jaydrogers]

Hey @nckrtl,

I put this on our Discord thread, but it's probably more appropriate to ask on this GitHub Thread.

Just to confirm, after seeing your PR get merged (laravel/horizon#1454), we still need to run this script to gracefully bring the horizon processes down, correct?

https://github.com/jaydrogers/horizon-test/blob/main/entrypoint/horizon.sh

[jaydrogers]

✅ Big update & solution

So after months of testing, the solution to this is:

• Remove S6 overlay as ENTRYPOINT for our images. Instead we moved it to CMD so when someone passes command: we don't have S6 Overlay get in the way and try to start web servers or anything weird (Improve S6 experience with command handling and simplify entrypoint behavior #437)
• Set the stop signals to be SIGTERM.
• Set health checks for horizon

Background on stop signals

PHP-FPM requires SIGQUIT for a graceful shutdown, but S6 Overlay wants SIGTERM. I spent months trying to get this to work.

My solution to make FPM & S6 Overlay both happy

1. By default, any fpm-apache or fpm-nginx will ship with STOPSIGNAL SIGQUIT
2. If you need to run an advanced service, you can override the stop signal with your orchestrator (like Docker does with stop_signal:)

A full example

This shows how we can use stop signals where we need them and configure health checks with the new native health checks for Laravel:

services:
  php:
    image: my/laravel-app

  redis:
    image: redis:6
    command: "redis-server --appendonly yes --requirepass redispassword"

  horizon:
    image: my/laravel-app
    command: ["php", "/var/www/html/artisan", "horizon"]
    stop_signal: SIGTERM # Set this for graceful shutdown if you're using fpm-apache or fpm-nginx
    healthcheck:
      # This is our native healthcheck script for Horizon
      test: ["CMD", "healthcheck-horizon"]
      start_period: 10s

I just want to add a HUGE SHOUTOUT to @nckrtl for all of his help through this journey. This was one of the most complicated issues I've debugged in my 19 years of my system administration experience. I would've gotten nowhere without @nckrtl's help. 🙌