cylc.get_suite_jobs_auths: database locked failures for rose-prune

[oliver-sanders]

We are hitting fairly regular database locked issues on Mac OS:

• https://github.com/metomi/rose/actions/runs/1986244844
• https://github.com/metomi/rose/actions/runs/1980854001

The relevant job logs can be found in the artefacts for those actions runs, here's a snippet:

20220315T104700Z/09-app-prune-work/3ymMFP', 'response': [True, 'Messages queued: 1']}]}}, 'user': 'runner'}
[FAIL] 2022-03-15T10:49:42+0000 database is locked
Traceback (most recent call last):
  File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/metomi/rose/task_run.py", line 176, in main
    sys.exit(runner(opts, args))
  File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/metomi/rose/run.py", line 201, in run
    return self.run_impl(opts, args, uuid, work_files)
  File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/metomi/rose/task_run.py", line 130, in run_impl
    return self.app_runner(opts, args)
  File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/metomi/rose/run.py", line 201, in run
    return self.run_impl(opts, args, uuid, work_files)
  File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/metomi/rose/app_run.py", line 400, in run_impl
    self, conf_tree, opts, args, uuid, work_files
  File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/metomi/rose/apps/rose_prune.py", line 101, in run
    suite_name, [(cycle, None) for cycle in cycle_set]
  File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/metomi/rose/suite_engine_procs/cylc.py", line 85, in get_suite_jobs_auths
    new_platforms = get_platforms_from_task_jobs(suite_name, cycle)
  File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/cylc/rose/platform_utils.py", line 77, in get_platforms_from_task_jobs
    dao = CylcWorkflowDAO(dbfilepath)
  File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/cylc/flow/rundb.py", line 331, in __init__
    self.create_tables()
  File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/cylc/flow/rundb.py", line 385, in create_tables
    ["table"]):
sqlite3.OperationalError: database is locked

[oliver-sanders]

Mac OS tests run on real (rather than virtual) hardware so it's not unsurprising that Mac OS would catch this issue and the Linux runners not.

[wxtim]

Will this replicate locally? Would I have to use my private laptop?

[oliver-sanders]

Haven't tried to replicate, seems odd that it only hits Mac OS.

[oliver-sanders]

Possible cause - cylc/cylc-rose#155

[oliver-sanders]

The traceback above was from a prune test

20220315T104700Z/09-app-prune-work/3ymMFP

Cylc-Rose was reading from the private DB which could cause DB locking. cylc/cylc-rose#155 changes this to use the public DB.