Skip to content

Commit

Permalink
ci: more printing about state of master (#9058)
Browse files Browse the repository at this point in the history
Two problems combined into one:
1. the session that `_wait_for_master` used had the system's default retry policy (which takes about 15 seconds before it gives up)
2. `_wait_for_master` reported every 60 iterations through its loop (which itself was limited by the number of iterations) rather than every 60 seconds. This meant that it only actually reported once every 16 minutes.

CircleCI had been killing workflows that didn't print text after 10 minutes. So when migrations took more than 10 minutes, the whole test would fail.

This PR fixes both of problems (1) and (2).
  • Loading branch information
wes-turner authored Apr 1, 2024
1 parent a3834ac commit 94e5d21
Showing 1 changed file with 8 additions and 4 deletions.
12 changes: 8 additions & 4 deletions .circleci/scripts/wait_for_perf_migration_upload_results.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,19 +16,23 @@
def _wait_for_master() -> None:
print("Checking for master")
cert = certs.Cert(noverify=True)
sess = api.UnauthSession("http://127.0.0.1:8080", cert)
sess = api.UnauthSession("http://127.0.0.1:8080", cert).with_retry(0)

# 2 hours is the most a migration can take, with this setup.
# If a migration takes longer than that we have hit an issue a customer will likely hit too.
for i in range(2 * 60 * 60):
time_start = time.time()
time_last_report = time_start
while time_start - time.time() < 2 * 60 * 60:
try:
r = sess.get("info")
if r.status_code == requests.codes.ok:
print(f"Master up and available after {int(time.time() - time_start) / 60} minutes")
return
except api.errors.MasterNotFoundException:
pass
if i % 60 == 0:
print("Waiting for master to be available...")
if time.time() - time_last_report > 60:
time_last_report = time.time()
print(f"Waiting for master ({int(time_last_report - time_start) / 60} minutes elapsed)")
time.sleep(1)
raise ConnectionError("Timed out connecting to Master")

Expand Down

0 comments on commit 94e5d21

Please sign in to comment.