pageserver: simpler, stricter config error handling

[jcsp]

Problem

Tenant attachment has error paths for failures to write local configuration, but these types of local storage I/O errors should be considered fatal for the process. Related thread on an earlier PR that touched this code: #7947 (comment)

Summary of changes

• Make errors writing tenant config fatal (abort process)
• When reading tenant config, make all I/O errors except ENOENT fatal
• Replace use of bare anyhow errors with LoadConfigError

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

3025 tests run: 2910 passed, 0 failed, 115 skipped (full report)

---

Flaky tests (2)

Postgres 14

• test_location_conf_churn[3]: debug
• test_pg_regress[None]: debug

Code coverage* (full report)

• functions: 32.7% (6915 of 21148 functions)
• lines: 50.0% (54197 of 108318 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
facdd2f at 2024-07-02T12:53:16.746Z :recycle:}

[problame]

general pet peeve: your error contexts / messages use present progressive tense. style guide says present tense, for maximum terseness at no information loss

neon/docs/error-handling.md

Lines 200 to 247 in 4a50483

	### Neon Rust code
	#### Anyhow Context
	When adding anyhow `context()`, use form `present-tense-verb+action`.
	Example:
	- Bad: `file.metadata().context("could not get file metadata")?;`
	- Good: `file.metadata().context("get file metadata")?;`
	#### Logging Errors
	When logging any error `e`, use `could not {e:#}` or `failed to {e:#}`.
	If `e` is an `anyhow` error and you want to log the backtrace that it contains,
	use `{e:?}` instead of `{e:#}`.
	#### Rationale
	The `{:#}` ("alternate Display") of an `anyhow` error chain is concatenation fo the contexts, using `: `.
	For example, the following Rust code will result in output
	```
	ERROR failed to list users: load users from server: parse response: invalid json
	```
	This is more concise / less noisy than what happens if you do `.context("could not ...")?` at each level, i.e.:
	```
	ERROR could not list users: could not load users from server: could not parse response: invalid json
	```
	```rust
	fn main() {
	match list_users().context("list users") else {
	Ok(_) => ...,
	Err(e) => tracing::error!("failed to {e:#}"),
	}
	}
	fn list_users() {
	http_get_users().context("load users from server")?;
	}
	fn http_get_users() {
	let response = client....?;
	response.parse().context("parse response")?; // fails with serde error "invalid json"
	}
	```

(We should extend that style guide to include thiserror error messages)