Horizon dies after few seconds

[spwin]

Hey,
I am trying to set up the local environment. I can see horizon dashboard with Inactive status.
When I am running php artisan horizon the status changes to Active for a second and then changes back to Inactive again. I tried using supervisor to run horizon but the same problem. From the supervisor logs I can see that it's running for a second, then restarting and starting again all the time. Is there any way to debug horizon and check why is it dying? I tried running php artisan horizon -vvv but every time the output is the same: Horizon started successfully. and then terminating the process. I can also confirm the queue is working fine when I run it using php artisan horizon:work.

Environment details:
Using Apache as a server on
Ubuntu 16.04.
Laravel version: 5.6.14
PHP version: 7.2
Redis server version: 4.0.11
Horizon version: 1.3

Configuration
config/queue.php

'redis' => [
            'driver' => 'redis',
            'connection' => 'default',
            'queue' => 'default',
            'retry_after' => 90,
            'block_for' => null,
        ],

config/horizon.php

'local' => [
            'supervisor-1' => [
                'connection' => 'redis',
                'queue' => ['default'],
                'balance' => 'auto',
                'processes' => 3,
                'tries' => 3,
            ],
        ],

.env

APP_ENV=local
QUEUE_DRIVER=redis
REDIS_HOST=127.0.0.1
REDIS_PASSWORD=null
REDIS_PORT=6379

Any suggestions?

[spwin]

Changed OS to Ubuntu 18.04 - this solved the problem. But some logging on horizon would be useful anyway.

[spwin]

If someone has the same problem, mine was crashing because of this:
https://github.com/laravel/horizon/blob/1.0/src/Listeners/MonitorMasterSupervisorMemory.php#L19

[mfn]

@spwin how did you fix it? Or did PHP due to the upgrade just use less memory and it auto-fixed itself.

[spwin]

@mfn I couldn't find other way than just comment MonitorMasterSupervisorMemory.php line 20 in vendor folder:

public function handle(MasterSupervisorLooped $event)
    {
        $master = $event->master;

        if ($master->memoryUsage() > 64) {
            // $master->terminate(12);
        }
    }

Works fine on local machine. If it's happening in production for you I think it's not a solution and you should increase memory on the server.
Cheers!

[mfn]

Hardcoded memory usage checks are never awesome; I think it would be very interesting what $master->memoryUsage() reports in your case. Do you think you will be able to get this value, e.g. via \Log::debug($master->memoryUsage());?

[spwin]

Now it's 44.018829345703, but this was happening for me periodically so I don't know what was the value before when it was crashing. This added line shows value every second so you can easily track it after php artisan horizon:

public function handle(MasterSupervisorLooped $event)
    {
        $master = $event->master;

        echo($master->memoryUsage() . "\n");
        if ($master->memoryUsage() > 64) {
            $master->terminate(12);
        }
    }

[Mohanrau]

Im having same issue any resolution team.Can anybody explain how this calculation calculating -->
$master->memoryUsage()

[mohammed1146]

I have the same issue, and I noticed $master->memoryUsage() is affected if your app loading many things in ServiceProviders, such as app/Console/kernel.php if you have many commands. @spwin if you disable theses things, the $master->memoryUsage() will be lower. @themsaid

[mfn]

Can anybody explain how this calculation calculating

You can see for yourself:

horizon/src/MasterSupervisor.php

Lines 322 to 325 in cc67213

	public function memoryUsage()
	{
	return memory_get_usage() / 1024 / 1024;
	}

I have the same issue, and I noticed $master->memoryUsage() is affected if your app loading many things in ServiceProviders,

What does memoryUsage() report in your case?

Maybe the hardcoded value in

horizon/src/Listeners/MonitorMasterSupervisorMemory.php

Lines 19 to 21 in cc67213

	if ($master->memoryUsage() > 64) {
	$master->terminate(12);
	}

needs to be simply configurable?

[driesvints]

Yeah we could maybe make this configurable.

[tomswinkels]

I have create a pull request.

#463

[spwin]

It would be nice to add an error of what’s happening as it just dies silently which is not helpful at all

[driesvints]

@spwin I agree. Maybe writing to a log could be useful. Can you create a separate issue and/or PR for this?

[driesvints]

The PR to make this configurable was merged.

[robclancy]

I wish the error was added... 3am last night I had to setup worker daemons because I couldn't work out why horizon was just shutting down. Then today only realizing the problem because of this issue.