Now that xce_test auto-loads a personal rc file at startup (this spec),
harden the three existing non-interactive xce_test callers so a future
developer's ~/.config/xce_test/rc can never silently affect a CI/soak run:
run-ha-soak.sh, run-resource-verify.sh, and clean_failover_preservation.sh
all now pass --norc to their xtest() invocation. No behavior change today
(no rc file exists in this environment yet, per xce-test-rc-file task 3.3's
own verification) -- this is forward-looking reproducibility hardening.
Independently re-verified end-to-end: run-ha-soak.sh (--sequential, 2
rounds, zero job loss) and run-resource-verify.sh (1 resource, 3 layers,
zero mismatches) both PASS unchanged with --norc added.
clean_failover_preservation.sh could not be run in this sandbox -- its
DATADIR precondition (<repo>/ha/data/cluster.bin + certs) was deleted by an
unrelated earlier commit (ee8473bd, admin-config-tool) and never restored;
confirmed via git history this is pre-existing and unrelated to this edit
(the precondition check runs before xtest() and fails identically on the
unmodified script). Logged as bug-1721, not fixed here -- out of this
spec's scope.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
The post-failover wait-primary/expect-job probes used the full masters list led
by the just-killed node, triggering libxce bug-854 (positional discovery can't
fail over from a dead-first master). Now they use a live-first list excluding the
killed primary.
NOTE: this exposes a deeper, separate post-failover AVAILABILITY issue (not data
loss): the new primary stays stuck in a catch-up loop with a lagging standby
(standby[0] lsn=5 < committed_lsn=6) and does not serve clients, so wait-primary
still times out. That is a catch-up/serving-readiness liveness concern orthogonal
to bug-828's committed-job DATA SAFETY, which is fixed and verified (walcompare
committed-prefix identical, jobs preserved on disk).
The commit-engine committed_lsn advance was TRACE-only; promoted to INFO (fires only
on a real advance, not noisy) so operators/tests can observe the authoritative primary
committed watermark. The clean-failover harness now polls this 'commit: committed_lsn
N -> M' line (not the lagging ha_process_standby_state heartbeat) with generous catch-up
headroom.
This made the live RED->GREEN visible: with Option A, committed_lsn reaches 5 on a
stable cluster ('jobs committed on a majority (committed_lsn=5 >= 5)') even via catch-up
delivery — the commit-stall is fixed.
Ran clean_failover_preservation.sh on a real fs ($HOME/tmp/xce-ha-deploy, ext4).
PAYLOAD run CONFIRMED bug-990 fixed live: standbys persist the leader's real
term (2) for the committed prefix; walcompare OK.
Harness fixes: robust failover detection (any post-kill CTRL_PRIMARY promotion,
45s settle) + a wait-for-commit-before-kill gate (jobsub returns on accept not
commit, so the prior harness raced replication vs kill).
That commit-wait gate exposed bug-991: on a STABLE cluster, jobs replicate to
all members (durable LSN 5) but committed_lsn pins at 1. Catch-up-delivered
entries don't advance the library commit_index, so task-2.4's ceiling
min(durable-majority, raft_commit_index) starves commit below the durable
majority. A real regression from the commit-source switch (2.4+3.3), invisible
to the steady-state-only unit tests.
Task 5.2 harness deploys a real 3-node localhost cluster, kills the primary,
promotes a standby, restarts the old primary, and asserts convergence via
walcompare --committed + per-job expect-job. Runs GREEN in PAYLOAD mode
(guards 848/849/850 + no committed-job loss).
The term-identity half of R5.1 is BLOCKED (bug-990): live replication still
uses term-0 repl_broadcast; the library send loop (on_send_append_entries) has
no live caller, so standbys persist hdr.term=0. The live-send retirement was
deferred across tasks 3.2/3.3 into verification tasks 5.2/5.4 that cannot
contain the production cutover. Needs a dedicated implementation task.
