---
id: "bug-1436"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "lib"
severity: "unknown"
timestamp: "2026-07-12T05:50:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 2
files: ["src/lib/client_internal.h (new XCECLI_FAILOVER_CALL_COUNT macro)", "src/lib/client.c (xcecli_reslist/xcecli_workerlist/xcecli_joblist/xcecli_resval/xcecli_jobinfo switched to it). .claude/skills/xce-test-ha/run-ha-soak.sh's node_pattern()/run_pattern() workdir-scoping fix (prior pass) remains a legitimate", "independent improvement -- KEPT", "not reverted."]
tags: ["ha", "connection-reset", "econnreset", "xce-test-ha", "sequential-mode", "round-36-recurrence", "libxce-failover", "client_failover.c", "xce_failover_classify", "xcecli_joblist", "status-code-magnitude-collision", "ECONNREFUSED-111", "reconnect-storm", "ROOT-CAUSED", "FIXED", "live-repro"]
related: ["bug-1435", "bug-1418", "bug-854", "bug-646"]
legacy: true
---

# bug-1436: 40-round sequential /xce-test-ha FAIL at round 36/40: xce_joblist: Connection reset by peer…

## Description
40-round sequential /xce-test-ha FAIL at round 36/40: xce_joblist: Connection reset by peer, immediately after wait-primary confirmed ctrl-2 serving in 46ms. RE-OPENED 2026-07-12 ~14:00-16:00Z after a live re-test (fresh isolated 40-round run, no concurrent soak) reproduced the IDENTICAL failure with ctrl-2 confirmed NEVER killed (same PID throughout, continuous RAFT/heartbeat log activity, still answering other clients after the failure) -- disproving the process-collision theory as the (sole) root cause. ROOT-CAUSED FOR REAL via live capture (tcpdump unavailable in sandbox -- no CAP_NET_RAW; used direct -vvv log correlation on a live reproduction instead) 2026-07-12 ~16:00-16:15Z. FIXED.

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
TWO ROOT CAUSES, both now addressed. (1) SUPERSEDED/PARTIAL (kept as a real but non-exclusive fix): the original pass diagnosed this as run-ha-soak.sh's unscoped `pkill -9 -x xce_controller` letting a second concurrent soak invocation silently SIGKILL the first run's live primary. That fix (node_pattern()/run_pattern() workdir-scoped pgrep/pkill) is a genuine improvement and stays in place, but a FRESH, FULLY ISOLATED 40-round soak re-run (deploy dir xce-test-ha-40-v2, confirmed via `ps aux` before/during/after that no second soak was ever running) STILL FAILED at round 36/40 with the byte-identical signature, and ctrl-2's own log proved it was never killed (constant PID, unbroken RAFT event stream, still serving OTHER clients seconds after the failure) -- so process collision was real but was NOT the (sole) mechanism behind every occurrence. (2) THE ACTUAL ROOT CAUSE (found via live reproduction, not archaeology): `xcecli_joblist`/`xcecli_reslist`/`xcecli_workerlist`/`xcecli_resval`/`xcecli_jobinfo` (src/lib/client.c) are all built on `xce_get_with_condition` (src/lib/xceapi.c), which returns a NON-NEGATIVE ROW COUNT on success (`rc = getreply.nrows`) -- NOT the uniform '0 = success, negative = XCE/errno-style status' convention every OTHER xcecli_* wrapper follows. All of them were routed through the generic `XCECLI_FAILOVER_CALL` macro (client_internal.h), which unconditionally feeds the raw EXPR return value into `xce_failover_classify()` (client_failover.c) -- a function that recognizes retriable/terminal outcomes purely by MAGNITUDE (E_XCE_* codes >=10001, and POSIX errno values it deliberately accepts in EITHER sign via `fo_abs()`: ECONNRESET=104, ETIMEDOUT=110, ECONNREFUSED=111). A joblist reply whose row count happens to equal one of these magnitudes is silently misclassified as a transport failure on EVERY call with that exact count, even though the read fully succeeded -- forcing a spurious `xce_failover_rotate()` to the next configured master. Since get-class ops are primary-only, landing on a standby correctly (but pointlessly) fails with E_XCE_STANDBY and rotates again; the whole cycle repeats for every subsequent same-sized call in the same verify_no_loss batch, burning the finite `max_retries` budget (20, XCE_RETRY_DEFAULTS) on self-inflicted reconnect churn. run-ha-soak.sh's sequential mode submits NJOBS=3 per round, so the tracked job count follows the arithmetic progression 3+3*round -- of the three dangerous POSIX magnitudes (104, 110, 111), ONLY 111 is a multiple of 3, and it is hit for the FIRST time at EXACTLY round 36 (3+3*36=111=ECONNREFUSED) -- explaining both why this is genuinely rare (requires an exact numeric collision, not a generic race) and why it always surfaced at round 36 specifically for this test's job-count cadence. LIVE PROOF: a fresh, fully isolated 40-round soak (deploy dir xce-test-ha-40-bug1436-live) failed AGAIN at round 36/40 with 111 tracked jobs; the target primary's own -vvv log (ctrl-2, PID confirmed continuously alive via a parallel 1s-resolution forensic process/connection poller) showed a clear, damning pattern for the ENTIRE 'verify' client session: `cluster.c:786 server sock 4 accepted 10` (a BRAND NEW TCP connection) immediately before EVERY SINGLE `opcode get` request, followed by two `xce_reply` sends (the >100-row chunked-get path, xce_reply_get pktlimit=100) and then `cluster.c:64` (the TRACE line inside `handle_endpoint_error`, i.e. the connection being torn down) -- one full accept-serve-close cycle PER expect-job call, ~1799 accepts total on that one short-lived primary PID alone -- confirming the client was silently reconnecting on every single (mostly-successful) get call, exactly the reconnect storm this mechanism predicts, not a misreading of the log by a prior investigation pass. The run's own retry-storm messages ('verify_no_loss attempt 1/3: recheck failed... likely post-election client retry-storm race') were themselves an accurate but previously-misinterpreted symptom of this bug. Confirmed via codegraph_explore + direct reads of src/lib/client.c, src/lib/client_internal.h, src/lib/client_failover.c, src/lib/xceapi.c, src/lib/request.c (xce_reply_get chunking), src/controller/opget.c (do_reply_joblist/check_controller), src/lib/netio.c (endpoint_enq/handle_error/handle_output -- confirmed NOT independently implicated: zero `endpoint_update_epoll` WARN lines anywhere in the failing run's ctrl-2.log), and the existing `src/lib/testsuite/test_libxce_retry.c` StatusClassifier taxonomy tests (which confirm `xce_failover_classify` is INTENTIONALLY designed to accept genuine status codes in either sign -- the bug is that 5 call sites fed it a row count instead of a status code, not that the classifier itself is wrong). Two purpose-built synthetic repro harnesses (hammer2.sh: fixed 60-job list, ~5400+ calls with periodic real primary kills; hammer3.sh: fixed 495-job list, ~1500+ calls with periodic real primary kills) both ran clean with ZERO failures, because neither ever held the tracked job count at exactly 104/110/111 -- which in hindsight is the confirming, not disconfirming, result: the bug requires an EXACT count collision, not raw connection-churn volume, so a fixed-size synthetic repro that never lands on the magic number cannot hit it no matter how much volume it generates. That negative result is what motivated pivoting to the real /xce-test-ha harness (which naturally sweeps through every job count via its round-based growth) as the reproduction vehicle, per the live-capture mandate.

## Fix
Added `XCECLI_FAILOVER_CALL_COUNT(ac, EXPR)` to src/lib/client_internal.h -- a sibling of `XCECLI_FAILOVER_CALL` for xce_get-family calls. It only enters the retry/classify loop when EXPR's raw result is NEGATIVE (`while ((_fc_rc < 0) && xcecli_retry_next(...))`); any non-negative result (0 or a genuine row count of ANY magnitude) is unconditionally treated as success and returned as-is, so it can never be misread as a retriable/terminal status code. A dead ctx (XCE_CTX_ERR) is still handled correctly: the per-op XCE_CTX_RUN guard in xceapi.c makes EXPR itself return a negative -EINVAL in that state, which enters the loop and is reclassified by xcecli_retry_decide's dead-connection branch exactly as before. Switched the 5 count-returning call sites in src/lib/client.c to the new macro: xcecli_reslist, xcecli_workerlist, xcecli_joblist (the confirmed live-reproduced offender), xcecli_resval (same xce_get_with_condition pattern, same latent risk), and xcecli_jobinfo (returns a fixed 1 on success -- lower risk since it never collides with a recognized magnitude and was previously misclassified as TERMINAL, which is harmless because TERMINAL does not rotate/rewrite rc, but fixed anyway for correctness/consistency rather than relying on that coincidence). Did NOT touch `xce_failover_classify` itself or any of the other 14 XCECLI_FAILOVER_CALL call sites (hello, join, jobdispatch, jobsubmit, clusterconf, ctrlstatus, update_resvals, update_jusage, jobctrl, queuectrl, hostctrl, runtask, set_pend_reason, clear_reason) -- all of those genuinely follow the 0/status-code convention the classifier is designed for, confirmed via test_libxce_retry.c's existing StatusClassifier taxonomy tests, which still pass unmodified (xcelib_test: 293 Tests 0 Failures 0 Ignored). controller_unit_test: 926 PASS / 0 FAIL / 0 Ignored (direct-binary run, ANSI-stripped grep census, exit 0). Did NOT touch run-ha-soak.sh further (its bug-1436-prior-pass workdir-scoping fix stays, unrelated to this mechanism) and did NOT add a client-side retry-more band-aid -- the actual defect was a classification-layer bug misinterpreting legitimate success data as a transport error, not a genuinely unavoidable transient condition, so the correct fix is at the classification boundary, not the retry budget.

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-1435, bug-1418, bug-854, bug-646