pageserver: fixes to /location_config API

[jcsp]

Problem

I found some issues with the /location_config API when writing new tests.

Summary of changes

• Calling the API with the "Detached" state is now idempotent.
• Tenant::spawn_attach now takes a boolean to indicate whether to expect a marker file. Marker files are used in the old attach path, but not in the new location conf API. They aren't needed because in the New World, the choice of whether to attach via remote state ("attach") or to trust local state ("load") will be revised to cope with the transitions between secondary & attached (see pageserver: always attach Tenant from remote storage, remove local-storage load path #5550). It is okay to merge this change ahead of that ticket, because the API is not used in the wild yet.
• Instead of using schedule_local_tenant_processing, the location conf API handler does its own directory creation and calls spawn_attach directly.
• A new unsafe_create_dir_all is added. This differs from crashsafe::create_dir_all in two ways:
  • It is intentionally not crashsafe, because in the location conf API we are no longer using directory or config existence as the signal for any important business logic.
  • It is async and uses tokio::fs.

[github-actions]

2280 tests run: 2164 passed, 0 failed, 116 skipped (full report)

---

Flaky tests (2)

Postgres 16

• test_pageserver_restart[False]: release

Postgres 14

• test_crafted_wal_end[simple]: debug

Code coverage (full report)

• functions: 52.5% (8184 of 15597 functions)
• lines: 81.1% (47825 of 58998 lines)

---

_{The comment gets automatically updated with the latest test results
64fd597 at 2023-10-13T15:12:57.089Z :recycle:}