Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add horizon:reset-metrics command #1318

Merged
merged 6 commits into from
Sep 18, 2023
Merged

Add horizon:reset-metrics command #1318

merged 6 commits into from
Sep 18, 2023

Conversation

trevorgehman
Copy link
Contributor

Problem

If a Job is renamed or moved, it will persist in Horizon under Metrics and never be removed. This ends up cluttering the Metrics with obsolete jobs. The problem is described in #1152

CleanShot 2023-09-15 at 00 40 06@2x

Solution

The simplest solution IMO is to just add a command to reset the metrics for Horizon entirely. I've given this command the signature horizon:reset-metrics.

It will remove all metric data in Horizon's Redis database, including:

  • List of jobs and queues that have metrics: measured_jobs / measured_queues
  • Current metrics (average runtime & throughput) for those jobs and queues: job:* / queue:*
  • Snapshots of metrics: snapshot:*
  • Timestamp of the last snapshot: last_snapshot_at
  • Snapshot lock key: metrics:snapshot

I believe I have covered everything relating to metrics without touching anything else.

Additionally, because I have to search Redis by wildcard to find some of the metrics data, I used SCAN instead of KEYS which is what Redis recommends to prevent locking issues:

Warning: consider KEYS as a command that should only be used in production environments with extreme care. It may ruin performance when it is executed against large databases. This command is intended for debugging and special operations, such as changing your keyspace layout. Don't use KEYS in your regular application code. If you're looking for a way to find keys in a subset of your keyspace, consider using SCAN...

https://redis.io/commands/keys/

After Running Command

CleanShot 2023-09-15 at 01 01 09@2x

CleanShot 2023-09-15 at 01 02 39@2x CleanShot 2023-09-15 at 01 02 42@2x

@taylorotwell taylorotwell merged commit 3a1474a into laravel:5.x Sep 18, 2023
11 checks passed
@wit3
Copy link

wit3 commented Sep 20, 2023

if i try to use the command ``
i receive this error : Call to undefined method Laravel\Horizon\Repositories\RedisMetricsRepository::reset()

Maybe the command must call ::clear() instead of ::reset()

EDIT:
already fixed with this commit ded3632

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants