---
id: "bug-1487"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "unknown"
severity: "unknown"
timestamp: "2026-07-31T04:50:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: [".claude/skills/xce-test-ha/run-ha-soak.sh"]
tags: ["xce-test-ha", "chaotic-soak", "verify_classify", "raft-commit-barrier", "apply-decoupling", "false-positive", "test-harness"]
related: ["bug-1422", "bug-1431", "bug-1432"]
legacy: true
---

# bug-1487: 30-round chaotic HA soak (run-ha-soak.sh --chaotic 30): committed_job_loss=1 at round 25/30…

## Description
30-round chaotic HA soak (run-ha-soak.sh --chaotic 30): committed_job_loss=1 at round 25/30 (VERDICT result=FAIL). expect-job reported job 0-6-1785473261-216067 not found on newly-elected primary ctrl-2 (term=27) immediately after the round-25 kill of ctrl-1 (term=26). committed_divergence stayed 0 for the entire run; the job (and all others) was confirmed present on every subsequent verify (rounds 26-30) and the final post-run sweep of all 93 tracked jobs.

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
NOT a controller bug and NOT genuinely-uncommitted-at-kill-time -- test-harness false-positive, root-caused with log/timestamp evidence from the preserved run (/home/claude/tmp/xce-ha-chaos30-run1/logs/ctrl-{1,2,3}.log). ctrl-1 (primary, term=26) majority-committed all 3 round-25 jobs (lsn 101,102,103; self + ctrl-2 ack) between 04:47:41.839 and 04:47:42.052 -- confirmed via commit_advance_locked log lines -- moments before being killed at 04:47:42+. ctrl-2 was elected primary at term=27 at 04:47:52.470 with ha_compute_safe_committed_lsn reporting safe_committed_lsn=99 "(from 2 granted voters)" -- the conservative Raft floor, since ctrl-3 (the only other reachable voter) was itself mid-restart/catch-up from round 25s own overlapping-kill design and had not yet caught up past lsn 99. Per the documented Raft Figure-8 safety mechanism (ha_arm_own_term_gate, ha.c:4997-5046: "the universal commit barrier still requires per-entry majority of a new-term entry"), a newly-elected primary cannot fold a PRIOR-term entry into its own committed_lsn watermark merely because it holds it durably -- it must first get a fresh majority ack, under its OWN term, for its own-term WAL_NOOP (armed at promotion). ctrl-2s committed_lsn stayed at 100 (excluding lsn 101-103) until ctrl-3s WAL catch-up stream let it ack the term-27 noop at lsn=104, at 04:47:58.702 -- 6.232s after promotion (commit_advance_locked: "committed_lsn 100 -> 104"). expect-job (joblist query) reads c->jobmgr, which the apply pump only populates for entries that have reached this committed_lsn watermark -- so the job was genuinely invisible, by design, for that 6.232s window. run-ha-soak.shs verify_classify() (chaotic mode) runs its expect-job check immediately once wait_for_new_primary detects the new primary is SERVING at a higher term -- unlike verify_no_loss (sequential mode; already carries a 3-attempt/2s-settle retry from bug-1422 for a related but distinct client-retry-storm race) and unlike wait_for_full_convergence (which chaotic mode intentionally skips between kills, by design, to land kills inside overlap windows) -- so its single, instantaneous query landed inside the 6.232s barrier window and counted a false LOSS. Comparison: round 24s election (ctrl-1, term=26) closed the identical barrier window in only 167ms because the other voting peer was not mid-catch-up at that moment -- the window duration is directly proportional to how far behind the second required peer is, which chaotic modes overlapping-restart design can occasionally push to several seconds.

## Fix
Added a bounded retry (5 attempts, 2s settle between) around the expect-job/verdict-classify check in verify_classify() (.claude/skills/xce-test-ha/run-ha-soak.sh) -- mirrors verify_no_loss/bug-1422s "retry the whole check" precedent, only retrying on a LOSS classification (BUG854/OK still resolve on the first attempt, matching prior semantics). ~8s of total settle margin comfortably covers the measured 6.232s worst-case gap while staying bounded -- a job still missing after 5 attempts still counts as a genuine LOSS, so real committed-job loss detection is unweakened. Verified with a full, fresh 30-round chaotic re-soak (see run2 VERDICT).

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-1422, bug-1431, bug-1432