storage controller: misc fixes

[jcsp]

Problem

Collection of small changes, batched together to reduce CI overhead.

Summary of changes

• Layer download messages include size -- this is useful when watching a pageserver hydrate its on disk cache in the log.
• Controller migrate API could put an invalid NodeId into TenantState
• Scheduling errors during tenant create could result in creating some shards and not others.
• Consistency check could give hard-to-understand failures in tests if a reconcile was in process: explicitly fail the check if reconciles are in progress instead.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

2490 tests run: 2368 passed, 0 failed, 122 skipped (full report)

---

Flaky tests (5)

Postgres 16

• test_wal_restore: release
• test_wal_restore_initdb: release

Postgres 15

• test_pageserver_key_reload: release
• test_fully_custom_config: release
• test_vm_bit_clear_on_heap_lock: debug

Code coverage* (full report)

• functions: 28.8% (6993 of 24258 functions)
• lines: 47.4% (43010 of 90711 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
b78c2a6 at 2024-03-06T15:14:46.538Z :recycle:}