feat(resource-value-verification): extend xce-deploy-ha with apiserver step + start-all.sh (task 3.1)

deploy-ha.sh Step 3 gains an opt-in-by-presence apiserver step (gated on a
pre-built /tmp/apiserver binary, not built by this script) that generates
start-apiserver.sh with a debug-level logger.yaml, plus an always-generated
start-all.sh sequencing controller-health-check-then-worker/scheduler
(-vvv)/apiserver. Purely additive - callers without a built apiserver
(e.g. xce-test-ha) see zero behavior change.

Live-confirmed both scenarios end-to-end: no-apiserver correctly skips the
step; with-apiserver reaches full backend registration with debug logging
active and scheduler emitting DEBUG3 output.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

.claude/skills/xce-deploy-ha/SKILL.md

@@ -37,6 +37,12 @@ disown`) before relying on subagent dispatch for unattended use.
   `cluster.bin` from `src/configs/*.yaml` fresh on every deploy
 - `openssl` must be on `PATH` (provided by `nix develop`) — `deploy-ha.sh`
   generates a fresh self-signed TLS cert/key for every deploy
+- Optional: run `cd src/frontend && make build-server` first if you want an
+  apiserver step (Step 4d) — a separate Go build system from the meson build
+  controller/worker/scheduler share, producing `/tmp/apiserver`.
+  `deploy-ha.sh` never runs this itself; without it, Step 3 simply skips
+  generating `scripts/start-apiserver.sh` (soft-skip, not an error), same as
+  the existing worker/scheduler convention.
 
 No data directory needed: TLS cert/key and `cluster.bin` are both generated
 directly into the deploy dir (`$TOP`) by `deploy-ha.sh` itself, not read from
@@ -145,6 +151,14 @@ This:
   `$TOP/scripts/xceadmin` **if** a wheel exists at
   `<same build dir>/xceadmin/dist/*.whl` — same soft-skip-with-warning
   convention. See Step 4c.
+- Also generates `$TOP/apiserver/config/{app.yaml,auth.yaml,rbac.yaml,logger.yaml}`
+  and `$TOP/scripts/start-apiserver.sh` **if** a pre-built apiserver binary
+  exists at `/tmp/apiserver` (`make build-server`'s default output path —
+  **not** alongside `$BINARY`, since the apiserver is a separate Go build,
+  not part of this meson build dir) — same soft-skip-with-warning
+  convention as worker/scheduler. See Step 4d.
+- Also **always** generates `$TOP/scripts/start-all.sh`, a combined shortcut
+  for Steps 4/4b/4c — see below.
 
 ## Step 4 — Start all nodes
 
@@ -217,6 +231,67 @@ a warning if no wheel exists in the same build dir as `$BINARY`, same as
   `--controller` silently tried port 9991 and failed with a connection-
   refused that had nothing to do with the actual deployment.
 
+## Step 4d — Start the apiserver (optional)
+
+`deploy-ha.sh` also generates `$TOP/scripts/start-apiserver.sh` **if** a
+pre-built apiserver binary exists at `/tmp/apiserver` (soft-skipped with a
+warning, not a hard error, if missing — see Prerequisites). Start it once at
+least one controller is up, the same way as the worker/scheduler:
+
+```bash
+"$BASH_BIN" "$TOP/scripts/start-apiserver.sh" &
+```
+
+- Runs the apiserver binary with `-d "$TOP/apiserver"` as its top directory,
+  pointing it at `$TOP/apiserver/config/{app.yaml,auth.yaml,rbac.yaml,logger.yaml}`
+  — a **minimal boot-only config**, not a production one:
+  - `auth.yaml`/`rbac.yaml` are copied verbatim from the repo's own shipped
+    dev-mode templates (`src/frontend/apiserver/config/`) — functional
+    auth/RBAC is not required for this deployment (no REST query is ever
+    issued against it here), only enough to load without erroring.
+  - `app.yaml` is copied from the same shipped template, with its
+    `backend:` section replaced to point `peers:` at this deployment's own
+    `ctrl-N=host:port` candidate list (the same one every other generated
+    script here already uses) — without this, the apiserver never connects
+    to the controller at all.
+  - `logger.yaml` is generated fresh (not copied) with `logging.level:
+    debug` explicitly set — required for any future apiserver-side
+    trace-log observability to actually be emitted; at any less-verbose
+    level it is silently dropped.
+- Logs to `$TOP/logs/apiserver.log` — same shared `logs/` directory
+  `ctrl-N.log`/`scheduler.log` already use (an absolute `outputPath` in
+  `logger.yaml`, even though the apiserver's own topDir is the separate
+  `$TOP/apiserver`).
+- Verify it started and connected to the controller:
+  `grep "Backend connection registered" "$TOP/logs/apiserver.log"`.
+- Does not attempt HA/multi-instance apiserver deployment — single instance
+  only.
+
+## Step 4e — `start-all.sh`: Steps 4/4b/4c/4d in one shot
+
+`deploy-ha.sh` **always** generates `$TOP/scripts/start-all.sh`, a combined
+shortcut that automates the manual recipe above end to end:
+
+```bash
+"$BASH_BIN" "$TOP/scripts/start-all.sh"
+```
+
+It: starts every controller node (`start.sh ctrl-N -vvv`, staggered 0.5s
+apart, same as Step 4); waits up to 30s for the same primary-election
+health-check Step 5 checks for; then starts whichever of
+`start-worker.sh`/`start-scheduler.sh`/`start-apiserver.sh` were actually
+generated for this deploy — checked via `[ -x ... ]` at `start-all.sh`'s own
+runtime, so it correctly skips any step whose binary wasn't built, with no
+regeneration needed. The scheduler is started at `-vvv` specifically (not
+just `-v`) so its `TRACE`/`debug3`-level output is reliably emitted. Every
+process is backgrounded via `setsid nohup … </dev/null >/dev/null 2>&1 &
+disown`, so the whole cluster survives `start-all.sh`'s own exit — safe to
+run from a subagent dispatch the same way Steps 1–5 already are (see the
+note at the top of this file).
+
+`start-all.sh` is regenerated fresh on every Step 3 deploy, same as every
+other generated script — nothing to keep in sync by hand.
+
 ## Step 5 — Verify cluster health
 
 Wait ~15 seconds for election and config replication, then check:
@@ -303,3 +378,7 @@ If you need a full redeploy (different peers or new binary), run steps 3–5 ins
 | A generated script fails with `bad interpreter` or silently uses the wrong bash | Redeploying with a stale `$BASH_BIN` captured outside `nix develop`, or a shell that shadows `bash` | Re-resolve `BASH_BIN=$(nix develop ./src --command which bash)` right before Step 3 and re-deploy |
 | `scripts/xceadmin ...` (no `--controller`) fails `cannot connect to localhost:9991: Connection refused` | bug-1566, fixed — a wheel built before this fix, or a `deploy-ha.sh` from before it, doesn't export `XCEADMIN_CONTROLLER` | Rebuild (`/xce-build xceadmin --incremental`) and redeploy (Step 3); check `$TOP/scripts/xceadmin` — it should `export XCEADMIN_CONTROLLER="<first --peers candidate>"` before the `exec` line |
 | `scripts/xceadmin` not generated | No wheel in `$BINARY`'s build dir (`deploy-ha.sh` logs a `Warning:`, not an error) | `/xce-build xceadmin` (add `--incremental` if you don't want to lose other already-built binaries in that dir — see `/xce-build`'s SKILL.md), then redeploy (Step 3) |
+| `scripts/start-apiserver.sh` not generated | No binary at `/tmp/apiserver` (`deploy-ha.sh` logs a `Warning:`, not an error) — this is a separate Go build, not part of `$BINARY`'s meson build dir | `cd src/frontend && make build-server`, then redeploy (Step 3) |
+| Apiserver exits immediately / `Failed to initialize configuration files` | `$TOP/apiserver/config/app.yaml` failed to parse or is missing a required field (e.g. `file_upload.max_size`) — most likely the shipped `src/frontend/apiserver/config/app.yaml` template this is copied from changed shape | Inspect `$TOP/logs/apiserver.log` / `$TOP/apiserver/logs/logger_error.log`; diff `$TOP/apiserver/config/app.yaml` against the shipped template |
+| Apiserver boots but never logs `Backend connection registered` | `app.yaml`'s `backend.peers` doesn't match the controller cluster (stale redeploy), or no controller is up yet | Confirm `$TOP/apiserver/config/app.yaml`'s `backend.peers` matches `$TOP/scripts/start.sh`'s `--peers`; start (or wait for) a controller first |
+| `start-all.sh` reports "no primary election observed... proceeding anyway" | Same causes as `election_timeout` above — no controller became primary within 30s | Check controller logs directly (Step 5); worker/scheduler/apiserver may still fail to connect until a primary exists |