myxce/.claude/skills/xce-test-ha
Rongsong Shen 0206b9e166 fix(xce-test-ha): retry chaotic-mode verify_classify before counting job loss (bug-1487)
30-round chaotic soak reported committed_job_loss=1 at round 25, but the
flagged job was confirmed present on every later verify and the final
sweep -- a real loss never comes back. Root-caused from ctrl-1/ctrl-2/ctrl-3
logs, no controller change needed: a newly-elected primary cannot fold a
prior-term entry into its own committed_lsn merely because it holds it
durably (Raft Figure-8 safety via ha_arm_own_term_gate) -- it needs a fresh
majority ack for its own-term WAL_NOOP first. That normally closes in
<200ms, but chaotic mode's overlapping-restart design let it stretch to
6.232s this run (the peer whose ack was needed was itself mid-catch-up from
its own recent restart). verify_classify ran its expect-job check
immediately on primary detection with no settle/retry, unlike verify_no_loss
(bug-1422) or wait_for_full_convergence, so it landed inside that window.

Added a bounded retry (5 attempts, 2s settle) around the LOSS classification
only, mirroring verify_no_loss's precedent -- a job still missing after that
still counts as a genuine loss. Verified with a fresh 30-round chaotic
re-soak: committed_divergence=0 committed_job_loss=0 result=PASS.
2026-07-31 05:16:14 +00:00
..
run-ha-soak.sh fix(xce-test-ha): retry chaotic-mode verify_classify before counting job loss (bug-1487) 2026-07-31 05:16:14 +00:00
SKILL.md feat(config-upgrade-snapshot): task 8.9 — live WAL-trim-under-chaos soak infra, blocked on bug-1474 2026-07-25 21:10:59 +00:00
soak-verdict-selftest.sh test(job-array-single-wal-record): large-array HA soak scenario 2026-07-16 04:43:55 +00:00
soak-verdict.sh test(job-array-single-wal-record): chaotic-failover array retry scenario 2026-07-16 05:27:24 +00:00