myxce/.claude/skills/xce-deploy-ha/SKILL.md

name	description	model
xce-deploy-ha	Deploy and manage a local XCE HA controller cluster for testing. Usage: /xce-deploy-ha [peers] — e.g. /xce-deploy-ha localhost:9991 localhost:9992 localhost:9993	haiku

XCE HA Cluster Deployment

Deploy a multi-node XCE controller cluster for local testing using src/tools/deploy-ha.sh. Covers: build, deploy, start, verify, stop, and cleanup.

No context: fork, intentionally. That Skill-tool isolation mechanism is confirmed broken for launching long-lived controller processes (same root cause as xce-test-ha's soak driver, fixed 2026-07-08): the forked session's own teardown SIGHUPs the whole process group, killing the cluster this skill just started. Since this skill's whole point is a cluster that stays running after the skill returns for later interactive use, that failure mode is fatal here — do not add context: fork back.

Agent-tool subagent dispatch is fine, though (different mechanism, empirically verified safe 2026-07-09 — see xce-test-ha/SKILL.md): dispatch an Agent subagent whose job is only to run Steps 1-5 below, ending with start.sh ctrl-N & for each node backgrounded with the usual shell & inside that subagent's own Bash call, then have the subagent return once health-check (Step 5) passes. The started controllers reparent to init and keep running independently of the subagent's own lifetime, same as the setsid nohup … & disown pattern verified for the soak driver. Prefer adding setsid to the start.sh invocations here too (belt-and-suspenders — this skill currently uses plain &, weaker than the soak driver's setsid nohup … & disown) before relying on subagent dispatch for unattended use.

Prerequisites

• Build must be current: run /xce-build controller first if binary is stale
• Run /xce-build xceadmin first (or nix build .#xcecore) so a built xceadmin binary exists — deploy-ha.sh (Step 3) uses it to build cluster.bin from src/configs/*.yaml fresh on every deploy
• openssl must be on PATH (provided by nix develop) — deploy-ha.sh generates a fresh self-signed TLS cert/key for every deploy
• Optional: run cd src/frontend && make build-server first if you want an apiserver step (Step 4d) — a separate Go build system from the meson build controller/worker/scheduler share, producing /tmp/apiserver. deploy-ha.sh never runs this itself; without it, Step 3 simply skips generating scripts/start-apiserver.sh (soft-skip, not an error), same as the existing worker/scheduler convention.
• Run /xce-build worker so both worker binaries exist in the same build dir as $BINARY: xcecd_virtual (default) and xcecd (the real, root-gated production daemon). Step 3 auto-picks which one scripts/start-worker.sh targets — see Step 4b.

No data directory needed: TLS cert/key and cluster.bin are both generated directly into the deploy dir ($TOP) by deploy-ha.sh itself, not read from any pre-existing ha/data/-style fixture.

Arguments

Arguments are space-separated host:port pairs, one per node, in priority order (index 0 = highest priority = primary candidate):

/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993

If no arguments are given, default to localhost:9991 localhost:9992 localhost:9993.

Step 1 — Resolve paths and the deploy environment's bash

BINARY="src/xce.build/controller/xce_controller"
TOP="/tmp/xce-deploy"

Verify the binary exists:

ls -lh "$BINARY"

If missing, tell the user to run /xce-build controller first.

Also resolve the bash that actually lives in the deploy environment (the nix dev shell) instead of assuming a bare bash on the invoker's own PATH means the same thing:

BASH_BIN=$(nix develop ./src --command which bash)

Why not just type bash: deploy-ha.sh and every script it generates are bash (arrays, +=, <<<) even though deploy-ha.sh itself carries a #!/bin/sh shebang in the repo today — a real incident, not a hypothetical: a working-tree edit once flipped that shebang while the script still used ${BASH_SOURCE[0]} and bash-only array syntax, which only kept working here because this environment's /bin/sh happens to be symlinked to bash. On a system where /bin/sh is dash (common on Debian/Ubuntu), that same file would fail immediately. Resolving $BASH_BIN once via which bash inside the nix dev shell — the same shell that provides openssl and the correct runtime libs for the binaries — and using it explicitly for every script invocation below sidesteps trusting any file's shebang line at all, in either direction (a stale one, or an ambient bash that might not resolve to the intended interpreter).

Step 2 — Stop any running instances

kill -9 $(pgrep xce_controller) 2>/dev/null || true
kill -9 $(pgrep -f xcecd_virtual) 2>/dev/null || true
kill -9 $(pgrep xce_scheduler) 2>/dev/null || true
sudo kill -9 $(pgrep -x xcecd) 2>/dev/null || true   # real worker runs as root — plain kill can't touch it

Step 3 — Deploy cluster

Join peers as a comma-separated string and call the deploy script — always via nix develop (needed for openssl) and the resolved $BASH_BIN (never a bare bash, never ./deploy-ha.sh relying on its own shebang):

PEERS="localhost:9991,localhost:9992,localhost:9993"  # from args
nix develop ./src --command "$BASH_BIN" src/tools/deploy-ha.sh "$BINARY" "$TOP" "$PEERS"

Every invocation is a fresh, self-contained deploy — no flag needed to force a rebuild, nothing is ever reused across deploys:

• cluster.bin is always rebuilt from src/configs/*.yaml via xceadmin, resolved from the meson build dir (xce.build/xceadmin/.venv/bin/xceadmin by default, also checks build.release/build.dbg), a nix-built result/bin/xceadmin, or PATH — in that order. If none is found, the script errors and tells you to run /xce-build xceadmin or nix build .#xcecore first.
• The TLS cert/key pair is always freshly generated via openssl (self-signed, RSA 2048, 10y validity — long-lived since it's per-deploy anyway; loses no real security property since TLS verification is SSL_VERIFY_NONE everywhere in this codebase). Errors if openssl isn't on PATH.
• The bash baked into the shebang of every script deploy-ha.sh generates (start.sh, cleanup.sh, start-worker.sh, start-scheduler.sh) is whatever command -v bash resolves to inside deploy-ha.sh's own invocation — i.e. $BASH_BIN from Step 1, since you invoked it with that interpreter under nix develop. This skill always still invokes each generated script explicitly as "$BASH_BIN" script.sh … below (the shebang is belt-and-suspenders for a human running one directly later, not load-bearing for this skill or /xce-test-ha, which calls the same deploy-ha.sh).

This:

• Creates $TOP/certs/, $TOP/controller/ctrl-{1..N}/conf, $TOP/controller/ctrl-{1..N}/data, $TOP/logs/
• Generates xce.crt/xce.key directly into $TOP/certs/
• Places cluster.bin.<tag> + conf/current in controller/ctrl-1/conf/ only (standbys receive config from primary via OPCode_replConfigPush)
• Generates $TOP/scripts/start.sh (with --workdir $TOP/controller --certdir $TOP/certs) and cleanup.sh
• Also generates $TOP/scripts/start-worker.sh and $TOP/scripts/start-scheduler.sh if the relevant binary exists alongside $BINARY (same build dir) — soft-skipped with a warning, not a hard error, if missing (so /xce-test-ha, which never builds worker/scheduler and calls this same script, is unaffected). Which worker binary — xcecd_virtual or the real xcecd — start-worker.sh targets is auto-detected per deploy (/run/systemd/system present → real xcecd; absent, e.g. this sandbox → xcecd_virtual), overridable with XCE_REAL_WORKER=0|1 before Step 3. See Step 4b.
• Also extracts a standalone xceadmin into $TOP/xceadmin-site and generates $TOP/scripts/xceadmin if a wheel exists at <same build dir>/xceadmin/dist/*.whl — same soft-skip-with-warning convention. See Step 4c.
• Also generates $TOP/apiserver/config/{app.yaml,auth.yaml,rbac.yaml,logger.yaml} and $TOP/scripts/start-apiserver.sh if a pre-built apiserver binary exists at /tmp/apiserver (make build-server's default output path — not alongside $BINARY, since the apiserver is a separate Go build, not part of this meson build dir) — same soft-skip-with-warning convention as worker/scheduler. See Step 4d.
• Also always generates $TOP/scripts/start-all.sh, a combined shortcut for Steps 4/4b/4c — see below.

Step 4 — Start all nodes

Start each controller in the background, via $BASH_BIN. Use -vvv for raft trace logs:

"$BASH_BIN" "$TOP/scripts/start.sh" ctrl-1 -vvv &
sleep 0.5
"$BASH_BIN" "$TOP/scripts/start.sh" ctrl-2 -vvv &
sleep 0.5
"$BASH_BIN" "$TOP/scripts/start.sh" ctrl-3 -vvv &

The startup stagger (0.5s between nodes) allows higher-priority nodes to begin their election before lower-priority ones start.

Step 4b — Start worker and scheduler (optional)

Once at least one controller is up (no need to wait for a full election — both fail over across the whole candidate list on their own), start a worker and/or the scheduler with the identical --peers configuration the controllers were deployed with — deploy-ha.sh bakes the exact same ctrl-1=host:port,ctrl-2=host:port,... string into both generated scripts, so there is nothing to keep in sync by hand:

"$BASH_BIN" "$TOP/scripts/start-worker.sh" worker-1 -vvv &   # name defaults to worker-1
"$BASH_BIN" "$TOP/scripts/start-scheduler.sh" -v &            # scheduler is a cluster singleton, no name

• start-worker.sh runs whichever binary Step 3 auto-detected — xcecd_virtual (ordinary user, no sudo) in this sandbox, or the real, production xcecd (wrapped in sudo by the generated script itself) on a host where /run/systemd/system exists. See "Real worker" below for what changes when it's the real binary.
• start-scheduler.sh runs xce_scheduler; its identity is fixed internally to xce_scheduler (no name argument). Logs to $TOP/logs/scheduler.log.
• Verify a worker joined: grep "Successfully sent .* load indices" "$TOP/logs/xcecd-worker-1.log" (virtual — logs are xcecd-<name>.log) or sudo grep "Successfully sent .* load indices" "$TOP/logs/xcecd.log" (real — always just xcecd.log, not per-name, and root-owned 640, not world-readable — a plain grep as yourself gets Permission denied; see below).
• Verify the scheduler joined: grep "sched_init: joined cluster" "$TOP/logs/scheduler.log".

Real worker (production xcecd)

Only relevant when Step 3 picked the real binary (/run/systemd/system present, or XCE_REAL_WORKER=1 forced it). Two things are structurally different from xcecd_virtual, both because of how the production binary itself is built, not this skill's choice:

• Single instance only. xcecd's db_path (/var/run/xcecd.db), domain_sock_file (/var/run/.xcecd.sock), and log file ($TOP/logs/xcecd.log) are all hardcoded in worker/src/main.c — unlike xcecd_virtual, -n/the name argument does not change any of them (it only sets the HA cluster identity used to join --peers). Only run one real worker per host at a time; you cannot do a real-worker equivalent of start-worker.sh worker-2.
• Job dispatch needs a real systemd+dbus host. Actually running a job goes through sd_bus_open_system() → org.freedesktop.systemd1.Manager.StartTransientUnit (worker/src/xcecd_cgroups.c) — a live call to the system dbus, answered by a real systemd manager. That's exactly the /run/systemd/system signal Step 3 auto-detects on, so on a host that picked the real binary this should just work; if you forced XCE_REAL_WORKER=1 somewhere that signal is absent (e.g. this sandbox), the daemon still starts, joins the cluster, and exercises its root-gated sqlite db-init/recovery and domain socket — but job submission will fail at the dispatch step regardless of sudo.

The generated start-worker.sh itself handles the rest — sudo mkdir -p /var/run (one-time preflight; xcecd exits immediately with "Failed to initialize database" if that directory doesn't exist, since neither hardcoded path is a flag) and wrapping the daemon exec in sudo. Nothing extra to do at call time beyond what's shown above, provided sudo is passwordless (confirmed sudo -n true in this sandbox) — on a host where it isn't, run this step interactively, not via the unattended Agent-tool dispatch described at the top of this file (a password prompt there would just hang).

Verify it's running as root: ps -o pid,user,cmd -C xcecd.

Reading its log needs sudo too — confirmed live: $TOP/logs/xcecd.log comes out owned root:root, mode 640 (unlike xcecd_worker.log and the domain socket, both of which the daemon explicitly chmods 0666; the main log file gets no such treatment). A plain grep/tail against it as yourself fails with Permission denied — use sudo grep .../sudo tail ... instead, same as the sudo kill already needed to stop the process.

Step 4c — Use xceadmin against this deployment (optional)

deploy-ha.sh also installs a standalone xceadmin at $TOP/xceadmin-site (extracted from the /xce-build xceadmin-produced wheel — soft-skipped with a warning if no wheel exists in the same build dir as $BINARY, same as start-worker.sh/start-scheduler.sh) and generates a wrapper at $TOP/scripts/xceadmin:

"$BASH_BIN" "$TOP/scripts/xceadmin" config push "$TOP/controller/ctrl-1/conf/cluster.bin.<tag>"
"$BASH_BIN" "$TOP/scripts/xceadmin" scheduler record status

• Genuinely standalone: works with zero nix/network dependency at runtime ($PYTHONPATH is baked into the wrapper at deploy time — see deploy-ha.sh's own comment for why a --system-site-packages venv does NOT work for this, a real dead end already ruled out). No need to prefix with "$BASH_BIN"/nix develop to run it, only to generate it in Step 3.
• Defaults --controller to this deployment's own first --peers candidate, not xceadmin's generic bare fallback (localhost:9991, unrelated to any real deployment's actual ports). The wrapper exports XCEADMIN_CONTROLLER="<first --peers candidate>", which every --controller-accepting xceadmin subcommand (config push, scheduler record start/stop/status) reads via envvar= fallback — an explicit --controller host:port on the command line still overrides it. Before this (bug-1566), a bare scripts/xceadmin scheduler record status with no --controller silently tried port 9991 and failed with a connection- refused that had nothing to do with the actual deployment.

Step 4d — Start the apiserver (optional)

deploy-ha.sh also generates $TOP/scripts/start-apiserver.sh if a pre-built apiserver binary exists at /tmp/apiserver (soft-skipped with a warning, not a hard error, if missing — see Prerequisites). Start it once at least one controller is up, the same way as the worker/scheduler:

"$BASH_BIN" "$TOP/scripts/start-apiserver.sh" &

• Runs the apiserver binary with -d "$TOP/apiserver" as its top directory, pointing it at $TOP/apiserver/config/{app.yaml,auth.yaml,rbac.yaml,logger.yaml} — a minimal boot-only config, not a production one:
  • auth.yaml/rbac.yaml are copied verbatim from the repo's own shipped dev-mode templates (src/frontend/apiserver/config/) — functional auth/RBAC is not required for this deployment (no REST query is ever issued against it here), only enough to load without erroring.
  • app.yaml is copied from the same shipped template, with its backend: section replaced to point peers: at this deployment's own ctrl-N=host:port candidate list (the same one every other generated script here already uses) — without this, the apiserver never connects to the controller at all.
  • logger.yaml is generated fresh (not copied) with logging.level: debug explicitly set — required for any future apiserver-side trace-log observability to actually be emitted; at any less-verbose level it is silently dropped.
• Logs to $TOP/logs/apiserver.log — same shared logs/ directory ctrl-N.log/scheduler.log already use (an absolute outputPath in logger.yaml, even though the apiserver's own topDir is the separate $TOP/apiserver).
• Verify it started and connected to the controller: grep "Backend connection registered" "$TOP/logs/apiserver.log".
• Does not attempt HA/multi-instance apiserver deployment — single instance only.

Step 4e — start-all.sh: Steps 4/4b/4c/4d in one shot

deploy-ha.sh always generates $TOP/scripts/start-all.sh, a combined shortcut that automates the manual recipe above end to end:

"$BASH_BIN" "$TOP/scripts/start-all.sh"

It: starts every controller node (start.sh ctrl-N -vvv, staggered 0.5s apart, same as Step 4); waits up to 30s for the same primary-election health-check Step 5 checks for; then starts whichever of start-worker.sh/start-scheduler.sh/start-apiserver.sh were actually generated for this deploy — checked via [ -x ... ] at start-all.sh's own runtime, so it correctly skips any step whose binary wasn't built, with no regeneration needed. The scheduler is started at -vvv specifically (not just -v) so its TRACE/debug3-level output is reliably emitted. Every process is backgrounded via setsid nohup … </dev/null >/dev/null 2>&1 & disown, so the whole cluster survives start-all.sh's own exit — safe to run from a subagent dispatch the same way Steps 1–5 already are (see the note at the top of this file).

start-all.sh is regenerated fresh on every Step 3 deploy, same as every other generated script — nothing to keep in sync by hand.

Step 5 — Verify cluster health

Wait ~15 seconds for election and config replication, then check:

# 1. Confirm ctrl-1 became primary and pushed config
grep -i "CTRL_PRIMARY\|push_config\|Config committed" "$TOP/logs/ctrl-1.log" | grep -v DEBUG3

# 2. Confirm ctrl-2 and ctrl-3 received config
find "$TOP/controller/ctrl-2/conf" "$TOP/controller/ctrl-3/conf" -type f | sort

# 3. Confirm standbys set primary_ctrl_fd
grep "primary control channel established" "$TOP/logs/ctrl-2.log" "$TOP/logs/ctrl-3.log"

Healthy cluster signs:

• ctrl-1.log: HA FSM (raft): → CTRL_PRIMARY (term=...) (or legacy HA FSM: → CTRL_PRIMARY (term=...))
• ctrl-1.log: Config committed on majority (tag=..., pushed=2/2 standbys)
• controller/ctrl-2/conf/cluster.bin.* and controller/ctrl-3/conf/cluster.bin.* exist
• ctrl-2.log / ctrl-3.log: handle_repl_config_push: config saved and loaded

Step 6 — Raft trace (DEBUG3 logs with -vvv)

With -vvv, raft vote events appear at DEBUG3:

RAFT[p0] term=2 event=election_start
RAFT[p0] term=2 event=vote_reply_in
RAFT[p0] term=2 event=quorum_reached
RAFT[p0] term=2 event=became_leader

Clean restart

Stop all instances, clean up state, and restart without redeploying:

# 1. Stop running controllers (and worker/scheduler, if you started them)
kill -9 $(pgrep xce_controller) 2>/dev/null || true
kill -9 $(pgrep -f xcecd_virtual) 2>/dev/null || true
kill -9 $(pgrep xce_scheduler) 2>/dev/null || true
sudo kill -9 $(pgrep -x xcecd) 2>/dev/null || true   # real worker runs as root

# 2. Run the generated cleanup script
#    Keeps: controller/ctrl-1/conf/* (primary config + current pointer), $TOP/certs/
#    Wipes: controller/ctrl-{1..N}/data/*, controller/ctrl-{2..N}/conf/*, logs/*.log
"$BASH_BIN" "$TOP/scripts/cleanup.sh"

# 3. Restart all nodes
"$BASH_BIN" "$TOP/scripts/start.sh" ctrl-1 -vvv &
sleep 0.5
"$BASH_BIN" "$TOP/scripts/start.sh" ctrl-2 -vvv &
sleep 0.5
"$BASH_BIN" "$TOP/scripts/start.sh" ctrl-3 -vvv &

# 4. Restart worker/scheduler if you had them running (Step 4b) — the
#    generated scripts still carry the same --peers, nothing to redo there
"$BASH_BIN" "$TOP/scripts/start-worker.sh" worker-1 -vvv &
"$BASH_BIN" "$TOP/scripts/start-scheduler.sh" -v &

What cleanup.sh preserves:

• $TOP/certs/ — shared TLS cert and key
• controller/ctrl-1/conf/cluster.bin.* + conf/current — primary config (standbys re-receive via push)

What cleanup.sh removes:

• All controller/ctrl-{1..N}/data/ dirs (WAL, snapshots, ha-term.bin) — forces fresh election next start
• All controller/ctrl-{2..N}/conf/ — standbys rebuild config from primary's OPCode_replConfigPush
• All logs/*.log — clean log slate

Not touched by cleanup.sh, if you ran the real worker: /var/run/xcecd.db and /var/run/.xcecd.sock live outside $TOP entirely (hardcoded, not parameterized by this deploy), so they persist across cleanup.sh runs — that's what lets the real worker's db-recovery path actually be exercised on restart. Remove them yourself (sudo rm -f /var/run/xcecd.db /var/run/.xcecd.sock) if you specifically want a from-scratch real-worker state.

If you need a full redeploy (different peers or new binary), run steps 3–5 instead.

Troubleshooting

Symptom	Cause	Fix
cluster_open_listener: no port	Peers missing from --peers	Ensure name=host:port format in generated start.sh
controller/ctrl-2/3/conf empty	push_config_to_standbys bug or OPCode_replConfigPush not handled	Verify binary is latest build
repl_ack_decode_and_dispatch: unexpected opcode	Heartbeat NOP on promoted ep	Fixed in current build; update binary
election_timeout loops without primary	All nodes have config_tag=0 (cold-start) or network issue	Check peers string matches actual listen ports
ha-term.bin causes "already voted" on restart	Old state from previous run	Run clean restart above
SSL_do_handshake: Broken pipe flooding from a worker/scheduler	A typo'd hostname in the --peers you passed (e.g. local instead of localhost) — the connect reaches SSL before failing, so it doesn't error out cleanly	Diff the failing --peers string against $TOP/scripts/start.sh's (they must be byte-identical); or just use start-worker.sh/start-scheduler.sh, which always carry the exact string the controllers were deployed with
start-worker.sh/start-scheduler.sh not generated	Whichever worker binary auto-detection (or XCE_REAL_WORKER) selected — xcecd_virtual or xcecd — doesn't exist in $BINARY's build dir, or xce_scheduler doesn't (deploy-ha.sh logs a Warning:, not an error)	/xce-build worker and/or /xce-build scheduler, then redeploy (Step 3) — /xce-build worker produces both xcecd_virtual and xcecd together
xcecd should be run as root	XCE_WORKER_BIN was pointed at real xcecd directly (bypassing the generated start-worker.sh, which already wraps it in sudo)	Invoke start-worker.sh itself rather than the binary directly, or prefix your own invocation with sudo
Real worker exits immediately: Failed to initialize database / Failed to open database	/var/run doesn't exist on this host — xcecd hardcodes db_path=/var/run/xcecd.db, not a flag	start-worker.sh's real-worker branch already runs sudo mkdir -p /var/run; if you invoked xcecd directly instead, run that yourself first
Job submitted to a real worker never starts / dispatch silently fails	Real xcecd dispatches via sd_bus_open_system() → org.freedesktop.systemd1.Manager.StartTransientUnit — no system dbus / systemd manager reachable (no /run/dbus/system_bus_socket, no dbus-daemon/systemctl)	Not fixable inside this skill on a host without a real systemd manager; the real worker still exercises root-gated startup (sqlite db init/recovery, domain socket, controller registration/reconnect) there, just not job execution — use xcecd_virtual (XCE_REAL_WORKER=0) for anything that needs jobs to actually run
Two real workers won't start / second clobbers the first	Real xcecd's db_path/domain_sock_file/log file are hardcoded, not parameterized by -n like xcecd_virtual is	Only run one real-worker instance per host; use xcecd_virtual (XCE_REAL_WORKER=0) for multiple named workers
kill -9 $(pgrep xcecd) doesn't stop the real worker	It's running as root (via sudo); an unprivileged kill against another user's PID fails silently	Use sudo kill -9 $(pgrep -x xcecd) (already in Step 2 / Clean restart above)
grep/tail against $TOP/logs/xcecd.log says Permission denied	Confirmed live: the real worker's log file comes out root:root, mode 640 — not chmod'd permissive like xcecd_worker.log/the domain socket are	Prefix with sudo (sudo grep .../sudo tail ...)
Deploy picked the wrong worker binary (real when you wanted virtual, or vice versa)	Auto-detection follows /run/systemd/system, which may not match what you actually want on this host	Set XCE_REAL_WORKER=0 or XCE_REAL_WORKER=1 before Step 3 to force it, then redeploy
A generated script fails with bad interpreter or silently uses the wrong bash	Redeploying with a stale $BASH_BIN captured outside nix develop, or a shell that shadows bash	Re-resolve BASH_BIN=$(nix develop ./src --command which bash) right before Step 3 and re-deploy
scripts/xceadmin ... (no --controller) fails cannot connect to localhost:9991: Connection refused	bug-1566, fixed — a wheel built before this fix, or a deploy-ha.sh from before it, doesn't export XCEADMIN_CONTROLLER	Rebuild (/xce-build xceadmin --incremental) and redeploy (Step 3); check $TOP/scripts/xceadmin — it should export XCEADMIN_CONTROLLER="<first --peers candidate>" before the exec line
scripts/xceadmin not generated	No wheel in $BINARY's build dir (deploy-ha.sh logs a Warning:, not an error)	/xce-build xceadmin (add --incremental if you don't want to lose other already-built binaries in that dir — see /xce-build's SKILL.md), then redeploy (Step 3)
scripts/start-apiserver.sh not generated	No binary at /tmp/apiserver (deploy-ha.sh logs a Warning:, not an error) — this is a separate Go build, not part of $BINARY's meson build dir	cd src/frontend && make build-server, then redeploy (Step 3)
Apiserver exits immediately / Failed to initialize configuration files	$TOP/apiserver/config/app.yaml failed to parse or is missing a required field (e.g. file_upload.max_size) — most likely the shipped src/frontend/apiserver/config/app.yaml template this is copied from changed shape	Inspect $TOP/logs/apiserver.log / $TOP/apiserver/logs/logger_error.log; diff $TOP/apiserver/config/app.yaml against the shipped template
Apiserver boots but never logs Backend connection registered	app.yaml's backend.peers doesn't match the controller cluster (stale redeploy), or no controller is up yet	Confirm $TOP/apiserver/config/app.yaml's backend.peers matches $TOP/scripts/start.sh's --peers; start (or wait for) a controller first
start-all.sh reports "no primary election observed... proceeding anyway"	Same causes as election_timeout above — no controller became primary within 30s	Check controller logs directly (Step 5); worker/scheduler/apiserver may still fail to connect until a primary exists