fix(xce-test-ha): retry chaotic-mode verify_classify before counting job loss (bug-1487)

30-round chaotic soak reported committed_job_loss=1 at round 25, but the
flagged job was confirmed present on every later verify and the final
sweep -- a real loss never comes back. Root-caused from ctrl-1/ctrl-2/ctrl-3
logs, no controller change needed: a newly-elected primary cannot fold a
prior-term entry into its own committed_lsn merely because it holds it
durably (Raft Figure-8 safety via ha_arm_own_term_gate) -- it needs a fresh
majority ack for its own-term WAL_NOOP first. That normally closes in
<200ms, but chaotic mode's overlapping-restart design let it stretch to
6.232s this run (the peer whose ack was needed was itself mid-catch-up from
its own recent restart). verify_classify ran its expect-job check
immediately on primary detection with no settle/retry, unlike verify_no_loss
(bug-1422) or wait_for_full_convergence, so it landed inside that window.

Added a bounded retry (5 attempts, 2s settle) around the LOSS classification
only, mirroring verify_no_loss's precedent -- a job still missing after that
still counts as a genuine loss. Verified with a fresh 30-round chaotic
re-soak: committed_divergence=0 committed_job_loss=0 result=PASS.

.wolf/buglog.json

@@ -13795,9 +13795,11 @@
       "ha-soak",
       "workflow"
     ],
-    "related_bugs": [],
-    "occurrences": 1,
-    "last_seen": "2026-07-23T09:50:00Z"
+    "related_bugs": [
+      "bug-1486"
+    ],
+    "occurrences": 2,
+    "last_seen": "2026-07-31T03:48:00Z"
   },
   {
     "id": "bug-1465",
@@ -14412,5 +14414,78 @@
     ],
     "occurrences": 1,
     "last_seen": "2026-07-29T21:10:00Z"
+  },
+  {
+    "id": "bug-1485",
+    "timestamp": "2026-07-30T14:35:00Z",
+    "error_message": "test_e2e_live_ha_push.py::test_live_config_push_against_primary_reaches_success_path flaky (near-100% fail rate across ~25 live runs during admin-config-tool task 6.3 round-2 remediation): TimeoutError: controller never opened localhost:<port>: [Errno 111] Connection refused, raised from a fresh wait_for_port re-probe of the elected primary's own address performed AFTER _wait_for_primary (CTRL_PRIMARY marker) + a new _wait_for_standbys (both other nodes show CTRL_STANDBY) + a 3s settle sleep all already succeeded.",
+    "file": "src/xceadmin/tests/test_e2e_live_ha_push.py (live_ha_cluster fixture); root cause is in src/controller (exact site not yet isolated -- C-side investigation out of this task's boundary)",
+    "root_cause": "NOT a pure test-side readiness race as round-1 review assumed. Live forensic capture (ps aux mid-run) caught the freshly-elected ctrl-1 as a <defunct> zombie process moments after its own log recorded the full raft_cb_become_leader promotion sequence ending at 'HA: repl registered standby ep[1] (slot 1, ...)' -- the log then goes silent (no crash trace, no further lines) and the process is gone. stderr was DEVNULL'd in the fixture so no ASAN/signal diagnostic was captured from a real failing pytest run. Confusingly, 3 separate standalone (non-pytest) repro scripts using byte-identical spawn code (same xce_controller build.dbg binary, same deploy-ha.sh cluster, same peers/certs/workdir args) ran the exact same primary-promotion sequence successfully every single time with the primary staying alive and the port reachable for 20-30s post-election -- while pytest-invoked runs of the identical fixture code failed roughly 24/25 times. Ruled out during investigation: pytest's default fd-capture (reproduced with -s / capturing disabled, still failed 3/3), ulimits (NPROC/NOFILE/AS/STACK identical between nix-develop plain-shell and pytest-invoked-via-python contexts), and orphaned-process port contention (ps aux clean before each batch). True root cause not isolated -- prime suspects are a genuine timing-sensitive controller-side crash/race in the CTRL_STANDBY->CTRL_PRIMARY raft promotion path (raft_cb_become_leader / repl registration) that pytest's process tree somehow makes far more likely to trigger, or a process-group/session/signal-mask difference inherited by subprocess.Popen children when the parent is pytest vs a bare python3 script.",
+    "fix": "NOT FIXED. Task 6.3 round-2 remediation implemented exactly what the round-1 reviewer asked for (_wait_for_standbys mirroring test-heartbeat-tolerance.sh's two-phase primary+standby gate, a heartbeat_interval_ms-justified 3s settle delay, and a fresh wait_for_port re-probe of the primary before yielding the fixture) but this did not close the flake because the underlying failure is the primary process disappearing, not merely its port being slow to open -- no test-side wait can outlast a process that is gone. Reported BLOCKED back to the task's parent controller rather than continuing to chase a C-level root cause outside this task's explicit boundary (controller-side C changes forbidden). Next step for whoever picks this up: capture stderr (not DEVNULL) from a real pytest-triggered failure to get an ASAN report or signal number, and/or strace the primary process across a live failing pytest run to see whether/why its listening socket closes.",
+    "tags": [
+      "flaky-test",
+      "ha",
+      "raft-election",
+      "controller-crash",
+      "admin-config-tool",
+      "task-6.3",
+      "e2e-live",
+      "unresolved",
+      "pytest-specific"
+    ],
+    "related_bugs": [
+      "bug-1477",
+      "bug-1479",
+      "bug-606"
+    ],
+    "occurrences": 24,
+    "last_seen": "2026-07-30T14:35:00Z"
+  },
+  {
+    "id": "bug-1486",
+    "timestamp": "2026-07-31T03:48:00Z",
+    "error_message": "Backgrounded `xce-build.sh all` run went silently stale mid-flight ([360/362], no ninja FAILED/error line, no completion marker, log just stopped growing at 03:38:35) with no matching build process left in `ps aux`",
+    "file": "src/tools/xce-build.sh",
+    "root_cause": "Multiple DIFFERENT Claude agent sessions independently invoked xce-build.sh against the same shared worktree's src/xce.build directory at the same time (confirmed via `ps auxf`: sibling `bash src/tools/xce-build.sh controller` and a separate `... all` invocation running under different parent `claude --output-format stream-json` PIDs). xce-build.sh has no lock/mutex around its unconditional `rm -rf \"$BUILD_DIR\"` + `meson setup` step (same code path as bug-1464), so a peer session's rm -rf tore the build directory out from under this session's own in-progress ninja/meson compile with no error propagation -- the backgrounded process's output just stops, indistinguishable at first glance from a hang.",
+    "fix": "No code change; workflow fix. Before trusting a backgrounded xce-build.sh log that has gone stale, run `ps auxf | grep -E 'xce-build|meson|ninja|cc1'` to check for a sibling xce-build.sh invocation from a different session targeting the same worktree. If found, do not start a third concurrent build -- bounded-poll (`while kill -0 $PEER_PID; do sleep 5; done`, not a passive wait for a background-task notification) until all peer invocations exit, then run exactly one clean blocking `xce-build.sh all` and treat only that run's own output as ground truth. Real fix would be a flock/lockfile around setup_build()'s rm-rf+meson-setup step in xce-build.sh itself so concurrent sessions serialize instead of clobbering.",
+    "tags": [
+      "build",
+      "xce-build.sh",
+      "meson",
+      "ninja",
+      "race-condition",
+      "concurrent-sessions",
+      "multi-agent",
+      "workflow"
+    ],
+    "related_bugs": [
+      "bug-1464"
+    ],
+    "occurrences": 1,
+    "last_seen": "2026-07-31T03:48:00Z"
+  },
+  {
+    "id": "bug-1487",
+    "timestamp": "2026-07-31T04:50:00Z",
+    "error_message": "30-round chaotic HA soak (run-ha-soak.sh --chaotic 30): committed_job_loss=1 at round 25/30 (VERDICT result=FAIL). expect-job reported job 0-6-1785473261-216067 not found on newly-elected primary ctrl-2 (term=27) immediately after the round-25 kill of ctrl-1 (term=26). committed_divergence stayed 0 for the entire run; the job (and all others) was confirmed present on every subsequent verify (rounds 26-30) and the final post-run sweep of all 93 tracked jobs.",
+    "file": ".claude/skills/xce-test-ha/run-ha-soak.sh",
+    "root_cause": "NOT a controller bug and NOT genuinely-uncommitted-at-kill-time -- test-harness false-positive, root-caused with log/timestamp evidence from the preserved run (/home/claude/tmp/xce-ha-chaos30-run1/logs/ctrl-{1,2,3}.log). ctrl-1 (primary, term=26) majority-committed all 3 round-25 jobs (lsn 101,102,103; self + ctrl-2 ack) between 04:47:41.839 and 04:47:42.052 -- confirmed via commit_advance_locked log lines -- moments before being killed at 04:47:42+. ctrl-2 was elected primary at term=27 at 04:47:52.470 with ha_compute_safe_committed_lsn reporting safe_committed_lsn=99 \"(from 2 granted voters)\" -- the conservative Raft floor, since ctrl-3 (the only other reachable voter) was itself mid-restart/catch-up from round 25s own overlapping-kill design and had not yet caught up past lsn 99. Per the documented Raft Figure-8 safety mechanism (ha_arm_own_term_gate, ha.c:4997-5046: \"the universal commit barrier still requires per-entry majority of a new-term entry\"), a newly-elected primary cannot fold a PRIOR-term entry into its own committed_lsn watermark merely because it holds it durably -- it must first get a fresh majority ack, under its OWN term, for its own-term WAL_NOOP (armed at promotion). ctrl-2s committed_lsn stayed at 100 (excluding lsn 101-103) until ctrl-3s WAL catch-up stream let it ack the term-27 noop at lsn=104, at 04:47:58.702 -- 6.232s after promotion (commit_advance_locked: \"committed_lsn 100 -> 104\"). expect-job (joblist query) reads c->jobmgr, which the apply pump only populates for entries that have reached this committed_lsn watermark -- so the job was genuinely invisible, by design, for that 6.232s window. run-ha-soak.shs verify_classify() (chaotic mode) runs its expect-job check immediately once wait_for_new_primary detects the new primary is SERVING at a higher term -- unlike verify_no_loss (sequential mode; already carries a 3-attempt/2s-settle retry from bug-1422 for a related but distinct client-retry-storm race) and unlike wait_for_full_convergence (which chaotic mode intentionally skips between kills, by design, to land kills inside overlap windows) -- so its single, instantaneous query landed inside the 6.232s barrier window and counted a false LOSS. Comparison: round 24s election (ctrl-1, term=26) closed the identical barrier window in only 167ms because the other voting peer was not mid-catch-up at that moment -- the window duration is directly proportional to how far behind the second required peer is, which chaotic modes overlapping-restart design can occasionally push to several seconds.",
+    "fix": "Added a bounded retry (5 attempts, 2s settle between) around the expect-job/verdict-classify check in verify_classify() (.claude/skills/xce-test-ha/run-ha-soak.sh) -- mirrors verify_no_loss/bug-1422s \"retry the whole check\" precedent, only retrying on a LOSS classification (BUG854/OK still resolve on the first attempt, matching prior semantics). ~8s of total settle margin comfortably covers the measured 6.232s worst-case gap while staying bounded -- a job still missing after 5 attempts still counts as a genuine LOSS, so real committed-job loss detection is unweakened. Verified with a full, fresh 30-round chaotic re-soak (see run2 VERDICT).",
+    "tags": [
+      "xce-test-ha",
+      "chaotic-soak",
+      "verify_classify",
+      "raft-commit-barrier",
+      "apply-decoupling",
+      "false-positive",
+      "test-harness"
+    ],
+    "related_bugs": [
+      "bug-1422",
+      "bug-1431",
+      "bug-1432"
+    ],
+    "occurrences": 1,
+    "last_seen": "2026-07-31T04:50:00Z"
   }
 ]