Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Increase in runs getting lost #2465

Open
christad92 opened this issue Sep 5, 2024 · 4 comments
Open

Increase in runs getting lost #2465

christad92 opened this issue Sep 5, 2024 · 4 comments
Assignees
@stuartc
Labels
bug Newly identified bug

Comments

@christad92
Copy link

christad92 commented Sep 5, 2024 •
edited by taylordowns2000
Loading

Recently, it seems as though runs are getting lost a lot more.

https://app.openfn.org/projects/2fedca61-bb42-4e65-a908-7a60f1b25c1e/runs/95da66d2-f70f-4d28-b8bf-bd7b12da212a
@christad92 christad92 added the bug Newly identified bug label Sep 5, 2024
@christad92
Copy link
Author

This seems to be related to the GitHub sync issues and we have not reported any new lost runs since refreshing workflows to create new snapshots.

Seems related to #2460 and #2461

@christad92
Copy link
Author

christad92 commented Sep 17, 2024 •
edited by taylordowns2000
Loading

Note: #2460 went to prod on September 10th and we still are seeing a large number of lost runs: https://analytics.openfn.org/question/596-lost-runs-since-9th-september

@christad92
Copy link
Author

We have also noted that runs get lost when data clips are too large and are not returned to lightning from the worker.

@taylordowns2000 taylordowns2000 changed the title Jobs are still getting lost Increase in runs getting lost Sep 18, 2024
@taylordowns2000
Copy link
Member

taylordowns2000 commented Sep 18, 2024 •
edited
Loading

@stuartc , there are 5 in this list (all times UTC):

running with worker@1.5.1:

  1. sept 11, 410am - worker crashed/restarted
  2. sept 11, 630am - no restart detected
  3. sept 11, 910am - no restart detected
  4. sept 12, 410am - no restart detected

running with worker@1.6.1 or greater:

  1. sept 18, 610am - no restart detected

@taylordowns2000 taylordowns2000 mentioned this issue Sep 18, 2024
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@stuartc stuartc

Labels
bug Newly identified bug
Projects
v2
Status: In progress
Milestone
No milestone
Development

No branches or pull requests
3 participants
@stuartc @christad92 @taylordowns2000