myxce/.claude/skills/xce-test-ha/SKILL.md

name	description
xce-test-ha	End-to-end HA failover soak test for an XCE controller cluster: deploy 3 nodes, submit jobs, then repeatedly kill the primary and verify zero committed-job loss across 5-10 rounds. Usage: /xce-test-ha [rounds] [dir]

XCE HA Failover Soak Test

Deploys a 3-node XCE controller cluster, submits jobs, then drives repeated primary kills + restarts, verifying after every failover that no committed job is ever lost. PASS only if all rounds complete with zero loss.

This is the live regression for the HA failover work. It deterministically exercises the full single-failover lifecycle (kill → elect → discover → verify → restart → reconverge) and has surfaced/validated these bugs:

• bug-828 committed-job data safety,
• bug-854 client primary discovery (incl. dead-first-master initial connect),
• bug-1028 post-failover availability (streamer term-source),
• bug-1042 standby-rejoin never converges (raft log_len not synced after recovery),
• bug-1043 catch-up apply gap (replicated entries not applied to the job table).

A green run is PASS: N rounds, M jobs submitted, zero job loss (exit 0).

Arguments

• Optional [rounds] — number of kill/restart rounds (clamped to 5–10, default 6).
• Optional [dir] — deploy directory for the 3-node cluster (default /home/claude/tmp/xce-ha-deploy). Maps to XCE_HA_TOP / run-ha-soak.sh's positional dir arg. Relative paths are resolved to absolute. Safety: the driver rm -rfs this dir on each run, so it refuses to wipe a non-empty directory that wasn't created by a prior /xce-test-ha run (it looks for a .xce-ha-deploy marker) — point it at a new/empty/scratch path, not a real source or home directory.

/xce-test-ha                      # 6 rounds, default deploy dir
/xce-test-ha 10                   # 10 rounds, default deploy dir
/xce-test-ha 10 /tmp/my-ha-run    # 10 rounds, deploy to /tmp/my-ha-run

Step 1 — Ensure binaries are built

The test needs xce_controller and xce_test:

BIN=src/xce.build/controller/xce_controller
XT=src/xce.build/controller/xce_test
ls -lh "$BIN" "$XT"

If either is missing, build them first with the /xce-build skill:

/xce-build controller

(Do NOT proceed to the soak until both binaries exist.)

Step 2 — Run the soak driver

The full workflow (deploy → start → elect → submit → kill-loop → verify) is encoded in run-ha-soak.sh in this skill directory. It MUST run under the nix dev shell so the controller/xce_test runtime libs (openssl, etc.) are on the library path. Run it in the background (it takes several minutes) and tail the log:

SKILL_DIR=.claude/skills/xce-test-ha
nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" 2>&1 | tee /tmp/xce-test-ha.log

Pass the round count and other knobs via env. The deploy directory may be given either as the script's first positional argument or via XCE_HA_TOP (positional wins):

# 10 rounds, default deploy dir
XCE_HA_ROUNDS=10 nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh"

# 10 rounds, deploy to a specific directory (positional arg)
XCE_HA_ROUNDS=10 nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" /tmp/my-ha-run

# equivalent via env
XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh"

Env var	Meaning	Default
XCE_HA_TOP	deploy directory	/home/claude/tmp/xce-ha-deploy
XCE_HA_DATADIR	config/certs source (cluster.bin, xce.crt, xce.key)	<repo>/ha/data
XCE_HA_ROUNDS	kill/restart rounds (5–10)	6
XCE_HA_JOBS	jobs per submit batch	3
XCE_HA_WAITMS	wait-primary budget per probe	30000

What the driver does (maps to the requested steps)

1. Preconditions — verifies xce_controller, xce_test, cluster.bin, and certs exist.
2. Deploy — src/tools/deploy-ha.sh into XCE_HA_TOP with nodes ctrl-1=localhost:9991, ctrl-2=localhost:9992, ctrl-3=localhost:9993, using config/certs from XCE_HA_DATADIR.
3. Start — scripts/start.sh ctrl-N -vvv for all three, 0.5s staggered.
4. Elect — waits (≤45s) for raft_cb_become_leader … CTRL_PRIMARY in any logs/ctrl-N.log, then confirms the client can reach the primary via ctrlstatus.
5. Initial workload — drives xce_test: set masters … / client / wait-primary / jobsub ×N / joblist (expect-job) / walcompare. Every acked JobId is tracked. 6–11. Kill loop (5–10 rounds): discover the live primary → kill -9 it → restart the node downed in the previous round (so the cluster returns to 3 members) → wait for a new primary → verify_no_loss (expect-job every tracked JobId) → submit another batch → verify_no_loss again → walcompare. Each elected primary is recorded for the leadership-coverage check. 11b. Leadership coverage — every node must serve as primary at least once. The natural kill loop only ever elects the two highest-priority nodes (ctrl-1/ctrl-2): the election timeout is priority-staggered (my_priority * election_priority_step_ms — p0 immediate, p1 +500ms, p2 +1000ms), so the lowest-priority node (ctrl-3) never wins while a lower-index node survives alongside it. After the loop, the driver restores 3-of-3 and forces any never-yet-primary node to leadership via a deterministic, loss-free maneuver that exploits the RequestVote freshness gate (a behind node cannot win): kill the spoiler standby → commit a batch so it falls behind → kill the primary (target holds all commits) → restart the stale spoiler so the freshness-fenced quorum elects the target → verify no loss → reconverge. A node staying behind cannot catch up while there is no primary, so the target wins deterministically, not by timing.
6. Verdict — exits 0 (PASS) only if every round verified zero job loss and all three nodes (ctrl-1, ctrl-2, ctrl-3) acted as primary; any missing job, failed election, unreachable primary, or a node that never led exits non-zero (FAIL). The PASS line names the nodes that led.

Step 3 — Report the result

• Exit 0 and a final PASS: N rounds, M jobs submitted, zero job loss line ⇒ test PASSED.
• Any FAIL: … line / non-zero exit ⇒ test FAILED. Surface the failing round and reason; the per-node logs under $XCE_HA_TOP/logs/ and the deploy log $XCE_HA_TOP/deploy.log hold the detail.

Goal-driven loop integration (/goal)

This skill is designed to be the success oracle for a goal-driven loop whose goal is "the HA soak test passes". The driver is built for that contract:

• Single machine-checkable verdict. Exactly one terminal line — either [xce-test-ha] PASS: … (exit 0) or [xce-test-ha] FAIL: … (exit ≠ 0). A loop decides done/continue by grepping that line / checking the exit code; no human-style parsing of intermediate output is needed.
• Fully idempotent & self-contained. Every run rm -rfs $XCE_HA_TOP, redeploys from ha/data, kills any stale xce_controller, and cleans up on exit (trap). A loop can invoke it any number of times with no manual reset.
• Deterministic-enough. Cold-start election ~15s, each round ~20-30s; a 5-round run is ~3-5 min, 10-round ~6-10 min. Budget the loop's per-iteration timeout to rounds × 60s + 60s.

Loop shape (the goal is reached when the driver exits 0):

goal: `/xce-test-ha` exits 0 with a "PASS:" line
each iteration:
  1. run the driver (background, capture log + exit code)
  2. if exit == 0 and log has "PASS:"  → GOAL REACHED, stop
  3. else: read the FAIL line + $XCE_HA_TOP/logs to find the root cause
  4. fix the underlying bug in src/ (controller or libxce)
  5. /xce-build controller   (rebuild the binaries the next run uses)
  6. repeat

The driver itself does NOT fix code — it is the oracle. The loop's job each round is: run oracle → if FAIL, diagnose from the logs, patch src/, rebuild, re-run. The goal is reached the first time the oracle returns PASS.

Job-loss model (why expect-job is the authority)

walcompare --committed checks the committed WAL prefix is byte-identical (term+crc) across members; it is logged as a note. The authoritative loss check is expect-job <JobId> against the live primary for every job ever acked — a committed job that an elected primary cannot list is a real loss (bug-828 class), regardless of walcompare's term-mapping tolerance.

The driver runs xce_test with ASAN_OPTIONS=detect_leaks=0: the debug/ASAN build reports harmless heap leaks at exit (non-zero exit code) that are NOT job loss. Verdicts are therefore judged on command output (expect-job: ok count and not found), never the leak-polluted process exit code.

Manual driving (debug)

To poke the cluster by hand after the driver deploys it (or instead of it):

# discover primary + submit one job
printf 'set masters localhost:9991,localhost:9992,localhost:9993\nset myname m\nclient\nwait-primary 30000\nctrlstatus\njobsub -u root -q normal -c /bin/true\njoblist\nquit\n' \
  | nix develop ./src --command "$XT"

# dump a node's WAL
printf 'waldump %s/controller/ctrl-1/data\nquit\n' "$XCE_HA_TOP" \
  | nix develop ./src --command "$XT"

Troubleshooting

Symptom	Cause	Fix
controller binary missing	not built	/xce-build controller
no primary elected within 45s	cold-start election stalled / port clash	check logs/ctrl-*.log; ensure ports 9991-9993 free
primary elected in logs but client could not reach it	client discovery (bug-854 regression)	check client.c rotation fix is in the binary
a committed job was LOST after failover	data-safety regression (bug-828 class)	inspect WAL via waldump; this is a real FAIL
driver hangs	a controller wedged; xtest 60s timeout should break it	check for stuck xce_test/xce_controller procs