myxce/.claude/skills/xce-test-ha/SKILL.md

name	description
xce-test-ha	End-to-end HA failover soak test for an XCE controller cluster: deploy 3 nodes, submit jobs, then repeatedly kill the primary and verify zero committed-job loss across 5-10 rounds. Usage: /xce-test-ha [rounds] [dir]

XCE HA Failover Soak Test

Deploys a 3-node XCE controller cluster, submits jobs, then drives repeated primary kills + restarts, verifying after every failover that no committed job is ever lost. PASS only if all rounds complete with zero loss.

This is the live regression for the HA failover work. It deterministically exercises the full single-failover lifecycle (kill → elect → discover → verify → restart → reconverge) and has surfaced/validated these bugs:

• bug-828 committed-job data safety,
• bug-854 client primary discovery (incl. dead-first-master initial connect),
• bug-1028 post-failover availability (streamer term-source),
• bug-1042 standby-rejoin never converges (raft log_len not synced after recovery),
• bug-1043 catch-up apply gap (replicated entries not applied to the job table).

A green run is PASS: N rounds, M jobs submitted, zero job loss (exit 0).

Arguments

• Optional [rounds] — number of kill/restart rounds (clamped to 5–10, default 6).
• Optional [dir] — deploy directory for the 3-node cluster (default /home/claude/tmp/xce-ha-deploy). Maps to XCE_HA_TOP / run-ha-soak.sh's positional dir arg. Relative paths are resolved to absolute. Safety: the driver rm -rfs this dir on each run, so it refuses to wipe a non-empty directory that wasn't created by a prior /xce-test-ha run (it looks for a .xce-ha-deploy marker) — point it at a new/empty/scratch path, not a real source or home directory.

Modes: sequential (default) vs chaotic

The driver has two kill schedules, selected by a flag (or XCE_HA_MODE):

• --sequential (default) — supported single-failover lifecycle: kill the primary → wait for a new primary → restart the downed node → wait for full 3-of-3 reconvergence before the next kill. Any committed-job loss or failed election is a hard FAIL (exit ≠ 0). Backward-compatible; the leadership- coverage maneuver (all three nodes must lead) runs only in this mode.
• --chaotic — overlapping kills that land inside election / catch-up windows: the previously-downed node is restarted but the driver does not wait for it to reconverge before killing the next primary. This deliberately exercises the unsupported double-fault-within-one-reconvergence window that surfaces committed-prefix divergence (bug-851 family). Quorum is preserved — only the single current primary is killed per round and the prior node is restarted, so at most one node is down at a kill instant.

SKILL_DIR=.claude/skills/xce-test-ha
# chaotic soak, 6 rounds, scratch dir
XCE_HA_ROUNDS=6 nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --chaotic /tmp/ha-chaos
# equivalent via env
XCE_HA_MODE=chaotic XCE_HA_ROUNDS=6 nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" /tmp/ha-chaos

run-ha-soak.sh also accepts a bare-integer positional as the round count (matching /xce-test-ha [rounds] [dir]): a non-integer positional is the deploy dir, an integer positional is rounds. So the release-binary short run is:

# 2-round chaotic soak on real-fs release binaries (no nix shell needed for the
# release build), positional rounds + dir:
XCE_HA_BIN=$PWD/src/build.release/controller/xce_controller \
XCE_HA_XT=$PWD/src/build.release/controller/xce_test \
  bash .claude/skills/xce-test-ha/run-ha-soak.sh --chaotic 2 ~/tmp/ha-1p5

For the acceptance gate (task 7.2) run chaotic mode for ≥6 rounds against real-fs (ext4) release binaries (point XCE_HA_BIN / XCE_HA_XT at the release xce_controller / xce_test).

Chaotic verdict (bug-854 / bug-924 isolated)

Chaotic mode does not hard-fail on data outcomes; it counts them and emits a single machine-parseable verdict line (soak-verdict.sh):

[xce-test-ha] VERDICT mode=chaotic rounds=6 jobs=18 committed_divergence=0 committed_job_loss=0 excluded_bug854_discovery=2 excluded_bug924_fdatasync=0 result=PASS

• committed_divergence — committed-prefix walcompare divergences, counted from task 1.1's WALCMP_DIVERGENCE … class=committed machine line (walcompare --report). Uncommitted-tail divergences are non-fatal and not counted.
• committed_job_loss — a tracked, acked job reported not found on a reachable primary (a real server-side loss, bug-828 class).
• excluded_bug854_discovery — the client could not LOCATE the primary (libxce primary-discovery fault). A discovery miss, not a server losing committed state — reported but excluded from the verdict.
• excluded_bug924_fdatasync — overlayfs fdatasync FAILED (errno=5/EIO) ERROR lines in the controller logs (environment defect). Reported but excluded; the real-fs acceptance run does not hit it.

result=PASS iff committed_divergence == 0 and committed_job_loss == 0, regardless of the excluded bug-854/bug-924 counts. The driver also prints the usual terminal PASS: / FAIL: oracle line (exit 0 / non-zero) so a goal-loop can branch on it unchanged.

--factstream mode: kill-mid-fact-stream (raft-apply-decoupling task 7.2)

Drives a continuous worker-fact stream — jstart/jfinish/jfail, real wire-protocol worker-report notifications (xcecli_notify_jobstart/ _jobfinish/_jobfail in libxce), sent from xce_test's worker REPL context — against a batch of pre-submitted jobs, kills the primary partway through the stream, lets a new primary take over, and asserts:

• zero committed-state divergence (same walcompare --committed mechanism chaotic mode uses), and
• zero phantom fact reactions — a job whose live materialized state (jobinfo) implies a fact was applied (and so its reactions — worker slot release, scheduler dispatch trigger, client/apiserver notify — already fired, since apply_committed_entry() is the single seam that both mutates state and fires the reaction hook) with no committed WAL fact entry backing it on some node (waldump, extended by this task to decode jobid/state for every worker-fact WAL type, not just WAL_JOB_SUBMIT).

This is a different traffic shape than every other mode: jobsub (existing modes) is a client command (job-submission); jstart/jfinish/ jfail are worker facts (asynchronous job-lifecycle reports) — the two travel different code paths in the controller (handle_jobsubmit vs. on_job_start/on_job_finish/on_job_fail) and this mode is the only one that exercises the fact path under a kill.

SKILL_DIR=.claude/skills/xce-test-ha
# 2-round kill-mid-fact-stream soak, scratch dir
nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --factstream ~/tmp/xce-ha-factstream
# tune job count / kill timing / round count
XCE_HA_FACTSTREAM_JOBS=40 XCE_HA_FACTSTREAM_KILL_DELAY=2 XCE_HA_FACTSTREAM_ROUNDS=3 \
  nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --factstream ~/tmp/xce-ha-factstream

Verdict line (soak-verdict.sh's verdict_emit_factstream):

[xce-test-ha] VERDICT mode=factstream rounds=2 jobs=3 facts_sent=40 committed_divergence=0 phantom_facts=0 excluded_bug924_fdatasync=0 result=PASS

result=PASS iff committed_divergence == 0 and phantom_facts == 0. jobs is the jobsub-tracked count from the shared initial-submit step; facts_sent is the number of fact-stream jobs (each gets a jstart + jfinish/jfail pair, so the actual fact-message count is 2 × facts_sent). A fact genuinely lost in the kill window (fire-and-forget notify, no reply) is not a phantom and is not counted — that is the accepted block-but-durable outcome (Requirement 6) for a fact that never reached majority commit; only the reverse (a reaction fired with no committed WAL entry backing it) counts.

Verdict self-test (no cluster)

The verdict parser has a deterministic, pure-bash self-test that feeds fixture output (a committed divergence, a bug-854 discovery failure, and bug-924 fdatasync lines) and asserts the divergence/job-loss counts are unaffected by bug-854/bug-924. It needs no binaries and runs in under a second:

bash .claude/skills/xce-test-ha/run-ha-soak.sh --selftest   # or:
bash .claude/skills/xce-test-ha/soak-verdict-selftest.sh

/xce-test-ha                      # 6 rounds, default deploy dir
/xce-test-ha 10                   # 10 rounds, default deploy dir
/xce-test-ha 10 /tmp/my-ha-run    # 10 rounds, deploy to /tmp/my-ha-run

Step 1 — Ensure binaries are built

The test needs xce_controller and xce_test:

BIN=src/xce.build/controller/xce_controller
XT=src/xce.build/controller/xce_test
ls -lh "$BIN" "$XT"

If either is missing, build them first with the /xce-build skill:

/xce-build controller

(Do NOT proceed to the soak until both binaries exist.)

Step 2 — Run the soak driver

The full workflow (deploy → start → elect → submit → kill-loop → verify) is encoded in run-ha-soak.sh in this skill directory. It MUST run under the nix dev shell so the controller/xce_test runtime libs (openssl, etc.) are on the library path. Run it in the background (it takes several minutes) and tail the log:

SKILL_DIR=.claude/skills/xce-test-ha
nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" 2>&1 | tee /tmp/xce-test-ha.log

Pass the round count and other knobs via env. The deploy directory may be given either as the script's first positional argument or via XCE_HA_TOP (positional wins):

# 10 rounds, default deploy dir
XCE_HA_ROUNDS=10 nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh"

# 10 rounds, deploy to a specific directory (positional arg)
XCE_HA_ROUNDS=10 nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" /tmp/my-ha-run

# equivalent via env
XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh"

Env var	Meaning	Default
XCE_HA_TOP	deploy directory	/home/claude/tmp/xce-ha-deploy
XCE_HA_DATADIR	config/certs source (cluster.bin, xce.crt, xce.key)	<repo>/ha/data
XCE_HA_MODE	sequential, chaotic, force-snapshot, or factstream (its --flag wins)	sequential
XCE_HA_BIN	xce_controller binary (point at real-fs release for the gate)	<repo>/src/xce.build/controller/xce_controller
XCE_HA_XT	xce_test binary	<repo>/src/xce.build/controller/xce_test
XCE_HA_ROUNDS	kill/restart rounds (5–10)	6
XCE_HA_JOBS	jobs per submit batch	3
XCE_HA_WAITMS	wait-primary budget per probe	30000
XCE_HA_FACTSTREAM_ROUNDS	--factstream kill-mid-fact-stream rounds	2
XCE_HA_FACTSTREAM_JOBS	--factstream jobs per round (each = 1 jstart + 1 jfinish/jfail)	30
XCE_HA_FACTSTREAM_KILL_DELAY	--factstream seconds before killing the primary after the stream starts	3
XCE_HA_FACTSTREAM_WAVE_SIZE	--factstream jobs per paced wave (own xce_test connection)	3
XCE_HA_FACTSTREAM_WAVE_GAP	--factstream seconds slept between waves	0.5
XCE_HA_FACTSTREAM_SETTLE	--factstream seconds slept after convergence, before traffic starts (bug-1416 mitigation)	10

What the driver does (maps to the requested steps)

1. Preconditions — verifies xce_controller, xce_test, cluster.bin, and certs exist.
2. Deploy — src/tools/deploy-ha.sh into XCE_HA_TOP with nodes ctrl-1=localhost:9991, ctrl-2=localhost:9992, ctrl-3=localhost:9993, using config/certs from XCE_HA_DATADIR.
3. Start — scripts/start.sh ctrl-N -vvv for all three, 0.5s staggered.
4. Elect — detects the elected primary via a client probe (xce_test wait-primary + ctrlstatus), reading live controller state. Primary, per-failover term, and convergence are ALL detected through the client path, never by grepping a controller logfile: the RELEASE build block-buffers its --logfile and flushes only on signal, so a log grep lags reality and would false-abort ("no primary within Ns") even though a primary is serving (task 1.3). The client probe is immune to that buffering.
5. Initial workload — drives xce_test: set masters … / client / wait-primary / jobsub ×N / joblist (expect-job) / walcompare. Every acked JobId is tracked. 6–11. Kill loop (5–10 rounds): discover the live primary → kill -9 it → restart the node downed in the previous round (so the cluster returns to 3 members) → wait for a new primary → verify_no_loss (expect-job every tracked JobId) → submit another batch → verify_no_loss again → walcompare. Each elected primary is recorded for the leadership-coverage check. 11b. Leadership coverage — every node must serve as primary at least once. The natural kill loop only ever elects the two highest-priority nodes (ctrl-1/ctrl-2): the election timeout is priority-staggered (my_priority * election_priority_step_ms — p0 immediate, p1 +500ms, p2 +1000ms), so the lowest-priority node (ctrl-3) never wins while a lower-index node survives alongside it. After the loop, the driver restores 3-of-3 and forces any never-yet-primary node to leadership via a deterministic, loss-free maneuver that exploits the RequestVote freshness gate (a behind node cannot win): kill the spoiler standby → commit a batch so it falls behind → kill the primary (target holds all commits) → restart the stale spoiler so the freshness-fenced quorum elects the target → verify no loss → reconverge. A node staying behind cannot catch up while there is no primary, so the target wins deterministically, not by timing.
6. Verdict — exits 0 (PASS) only if every round verified zero job loss and all three nodes (ctrl-1, ctrl-2, ctrl-3) acted as primary; any missing job, failed election, unreachable primary, or a node that never led exits non-zero (FAIL). The PASS line names the nodes that led.

Step 3 — Report the result

• Exit 0 and a final PASS: N rounds, M jobs submitted, zero job loss line ⇒ test PASSED.
• Any FAIL: … line / non-zero exit ⇒ test FAILED. Surface the failing round and reason; the per-node logs under $XCE_HA_TOP/logs/ and the deploy log $XCE_HA_TOP/deploy.log hold the detail.

Goal-driven loop integration (/goal)

This skill is designed to be the success oracle for a goal-driven loop whose goal is "the HA soak test passes". The driver is built for that contract:

• Single machine-checkable verdict. Exactly one terminal line — either [xce-test-ha] PASS: … (exit 0) or [xce-test-ha] FAIL: … (exit ≠ 0). A loop decides done/continue by grepping that line / checking the exit code; no human-style parsing of intermediate output is needed.
• Fully idempotent & self-contained. Every run rm -rfs $XCE_HA_TOP, redeploys from ha/data, kills any stale xce_controller, and cleans up on exit (trap). A loop can invoke it any number of times with no manual reset.
• Deterministic-enough. Cold-start election ~15s, each round ~20-30s; a 5-round run is ~3-5 min, 10-round ~6-10 min. Budget the loop's per-iteration timeout to rounds × 60s + 60s.

Loop shape (the goal is reached when the driver exits 0):

goal: `/xce-test-ha` exits 0 with a "PASS:" line
each iteration:
  1. run the driver (background, capture log + exit code)
  2. if exit == 0 and log has "PASS:"  → GOAL REACHED, stop
  3. else: read the FAIL line + $XCE_HA_TOP/logs to find the root cause
  4. fix the underlying bug in src/ (controller or libxce)
  5. /xce-build controller   (rebuild the binaries the next run uses)
  6. repeat

The driver itself does NOT fix code — it is the oracle. The loop's job each round is: run oracle → if FAIL, diagnose from the logs, patch src/, rebuild, re-run. The goal is reached the first time the oracle returns PASS.

Job-loss model (why expect-job is the authority)

walcompare --committed checks the committed WAL prefix is byte-identical (term+crc) across members; it is logged as a note. The authoritative loss check is expect-job <JobId> against the live primary for every job ever acked — a committed job that an elected primary cannot list is a real loss (bug-828 class), regardless of walcompare's term-mapping tolerance.

The driver runs xce_test with ASAN_OPTIONS=detect_leaks=0: the debug/ASAN build reports harmless heap leaks at exit (non-zero exit code) that are NOT job loss. Verdicts are therefore judged on command output (expect-job: ok count and not found), never the leak-polluted process exit code.

Manual driving (debug)

To poke the cluster by hand after the driver deploys it (or instead of it):

# discover primary + submit one job
printf 'set masters localhost:9991,localhost:9992,localhost:9993\nset myname m\nclient\nwait-primary 30000\nctrlstatus\njobsub -u root -q normal -c /bin/true\njoblist\nquit\n' \
  | nix develop ./src --command "$XT"

# dump a node's WAL
printf 'waldump %s/controller/ctrl-1/data\nquit\n' "$XCE_HA_TOP" \
  | nix develop ./src --command "$XT"

Troubleshooting

Symptom	Cause	Fix
controller binary missing	not built	/xce-build controller
no primary elected within ~90s via client probe	cold-start election stalled / port clash (NOT a log-buffer artifact — detection is client-probe based)	check logs/ctrl-*.log; ensure ports 9991-9993 free
primary elected in logs but client could not reach it	client discovery (bug-854 regression)	check client.c rotation fix is in the binary
a committed job was LOST after failover	data-safety regression (bug-828 class)	inspect WAL via waldump; this is a real FAIL
driver hangs	a controller wedged; xtest 60s timeout should break it	check for stuck xce_test/xce_controller procs