---
id: "bug-1422"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "unknown"
severity: "unknown"
timestamp: "2026-07-09T10:14:14.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: [".claude/skills/xce-test-ha/run-ha-soak.sh"]
tags: ["xce-test-ha", "false-positive", "client-failover", "retry-budget-exhaustion", "E_XCE_STANDBY", "test-harness"]
related: ["bug-854", "bug-1421"]
legacy: true
---

# bug-1422: 40-round sequential HA soak FAIL at round 36: verify_no_loss reported '0/111 jobs confirmed…

## Description
40-round sequential HA soak FAIL at round 36: verify_no_loss reported '0/111 jobs confirmed present' right after a fresh election (new primary ctrl-1, term=40) — xce_test's expect-job got E_XCE_STANDBY (10014) from a fresh client session whose own wait-primary had JUST confirmed ctrl-1 as serving 45ms earlier

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
False-positive test-harness failure, NOT real committed-job loss: ctrl-1's own controller log confirms it stayed CTRL_PRIMARY throughout the window and successfully committed the 3 new post-failover jobs (log_append/ae_reply_success/commit_advanced at 10:14:22-23). ha_dispatch_gate's CTRL_PRIMARY branch (ha.c:3451-3531) unconditionally permits OPCode_get (joblist) even while fenced, so ctrl-1 itself could not have returned E_XCE_STANDBY for this query — the reply must have come from a standby (ctrl-2, just killed, or ctrl-3) that the client's failover retry loop landed on. Root cause traced to client_failover.c's xcecli_retry_next/xcecli_retry_decide (client_failover.c:305-319): E_XCE_STANDBY classifies as XCE_FO_RETRY_NOW (client_failover.c:59-60, prompt rotate, no backoff), and the 20-attempt retry budget (3 masters) should reliably reach the real primary — but when the WHOLE budget is exhausted while every recent classification was RETRY_NOW (as opposed to RETRY_BACKOFF), xcecli_retry_decide's budget-exhaustion branch (line 312-318) does NOT override *rc the way it does for RETRY_BACKOFF exhaustion (which maps to the distinct -E_XCE_NOT_READY, line 316-317) — it just returns false with *rc left at whatever the last attempt's raw status was (here, a stray E_XCE_STANDBY from a standby hit during the rapid post-election retry storm). verify_no_loss() (run-ha-soak.sh) then did a single one-shot check with zero retry at the driver level, unlike the job-submit loop just above it in the same file which already retries on this exact 'primary likely just promoted, regaining write-quorum' condition (run-ha-soak.sh:483-486). Only surfaced at round 36 of a 40-round run — narrow enough timing that shorter 6-10 round soaks across many prior sessions never hit it.

## Fix
Added a bounded retry (3 total attempts, 2s settle sleep between) around the WHOLE verify_no_loss() check in run-ha-soak.sh, mirroring the existing submit-retry loop's rationale: a job still missing after a real settle window + 2 retries is a genuine loss, not this client-side retry-storm race. Did not touch client_failover.c (the battle-tested libxce-failover retry/rotation library with many other production consumers) — fixing at the test-driver level is safer and fully local. Re-ran both 40-round soaks after the fix: sequential PASS (123 jobs, zero loss, all 3 nodes led) and chaotic PASS (123 jobs, zero committed divergence, zero committed job loss).

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-854, bug-1421