Rongsong Shen
d26ff8870d
The buglog was one 27K-line JSON file whose fields covered only symptom,
root cause and fix, as free text: 2% of entries named the code version a
bug was found on, 12% the build profile, 36% a way to reproduce, and 58% of
the file was OpenWolf hook auto-detections ("Missing guard clause",
"Function not marked async" on .md files).
- .claude/skills/buglog/bug-record.md: the record standard. Eight
sections (Summary as the title, Description, Environment, Reproduction;
Root cause, Fix, Verification, References once fixed), each with the
content it must contain -- e.g. Environment needs the OS, the libc the
failing binary loads (from its ELF interpreter: glibc 2.42 in the nix
shell vs 2.40 on the host), toolchain, build profile and commit.
- .wolf/bugs/<id>.md: 500 records migrated from buglog.json; every field
verified to round-trip. They carry legacy: true and an explicit "not
recorded" where the old log had no such field; component derived from
file paths, severity from keywords (else unknown), fix_commits from
hashes in status_note/fix that git resolves (91 of 463 fixed).
- .wolf/buglog-archive.json: the 710 hook auto-detections, verbatim.
- .wolf/buglog.json is now an index generated from the records, keeping
the fields OpenWolf's hooks and `openwolf bug search` read.
- scripts/buglog.py rewritten: new (template + Environment collected from
the machine and --binary), check, status (refuses a status the record
doesn't support; fixed needs Root cause, Fix, Verification and
--commit hashes git resolves), note, recur, set, show, list, triage,
env, index. 29 unittest cases in scripts/tests/test_buglog.py.
- scripts/git-hooks/pre-commit runs `check --staged`: incomplete staged
records, a legacy record hand-changed to fixed without the fix-time
sections, a new record claiming legacy: true, or a stale index refuse
the commit.
- scripts/gitea-sync.py reads the records: issue body = the record's
sections plus linked fix commits (origin remote, GITEA_COMMIT_REPO to
override), and a body hash so an issue's text is refreshed when the
record changes.
- bug-1866 rewritten to the full standard as the worked example.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
9.7 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | legacy | ||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1436 | fixed | 2026-09-24 | lib | unknown | 2026-07-12T05:50:00Z | 2026-09-19T09:00:00Z | 2 |
|
|
|
true |
bug-1436: 40-round sequential /xce-test-ha FAIL at round 36/40: xce_joblist: Connection reset by peer…
Description
40-round sequential /xce-test-ha FAIL at round 36/40: xce_joblist: Connection reset by peer, immediately after wait-primary confirmed ctrl-2 serving in 46ms. RE-OPENED 2026-07-12 ~14:00-16:00Z after a live re-test (fresh isolated 40-round run, no concurrent soak) reproduced the IDENTICAL failure with ctrl-2 confirmed NEVER killed (same PID throughout, continuous RAFT/heartbeat log activity, still answering other clients after the failure) -- disproving the process-collision theory as the (sole) root cause. ROOT-CAUSED FOR REAL via live capture (tcpdump unavailable in sandbox -- no CAP_NET_RAW; used direct -vvv log correlation on a live reproduction instead) 2026-07-12 ~16:00-16:15Z. FIXED.
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
TWO ROOT CAUSES, both now addressed. (1) SUPERSEDED/PARTIAL (kept as a real but non-exclusive fix): the original pass diagnosed this as run-ha-soak.sh's unscoped pkill -9 -x xce_controller letting a second concurrent soak invocation silently SIGKILL the first run's live primary. That fix (node_pattern()/run_pattern() workdir-scoped pgrep/pkill) is a genuine improvement and stays in place, but a FRESH, FULLY ISOLATED 40-round soak re-run (deploy dir xce-test-ha-40-v2, confirmed via ps aux before/during/after that no second soak was ever running) STILL FAILED at round 36/40 with the byte-identical signature, and ctrl-2's own log proved it was never killed (constant PID, unbroken RAFT event stream, still serving OTHER clients seconds after the failure) -- so process collision was real but was NOT the (sole) mechanism behind every occurrence. (2) THE ACTUAL ROOT CAUSE (found via live reproduction, not archaeology): xcecli_joblist/xcecli_reslist/xcecli_workerlist/xcecli_resval/xcecli_jobinfo (src/lib/client.c) are all built on xce_get_with_condition (src/lib/xceapi.c), which returns a NON-NEGATIVE ROW COUNT on success (rc = getreply.nrows) -- NOT the uniform '0 = success, negative = XCE/errno-style status' convention every OTHER xcecli_* wrapper follows. All of them were routed through the generic XCECLI_FAILOVER_CALL macro (client_internal.h), which unconditionally feeds the raw EXPR return value into xce_failover_classify() (client_failover.c) -- a function that recognizes retriable/terminal outcomes purely by MAGNITUDE (E_XCE_* codes >=10001, and POSIX errno values it deliberately accepts in EITHER sign via fo_abs(): ECONNRESET=104, ETIMEDOUT=110, ECONNREFUSED=111). A joblist reply whose row count happens to equal one of these magnitudes is silently misclassified as a transport failure on EVERY call with that exact count, even though the read fully succeeded -- forcing a spurious xce_failover_rotate() to the next configured master. Since get-class ops are primary-only, landing on a standby correctly (but pointlessly) fails with E_XCE_STANDBY and rotates again; the whole cycle repeats for every subsequent same-sized call in the same verify_no_loss batch, burning the finite max_retries budget (20, XCE_RETRY_DEFAULTS) on self-inflicted reconnect churn. run-ha-soak.sh's sequential mode submits NJOBS=3 per round, so the tracked job count follows the arithmetic progression 3+3round -- of the three dangerous POSIX magnitudes (104, 110, 111), ONLY 111 is a multiple of 3, and it is hit for the FIRST time at EXACTLY round 36 (3+336=111=ECONNREFUSED) -- explaining both why this is genuinely rare (requires an exact numeric collision, not a generic race) and why it always surfaced at round 36 specifically for this test's job-count cadence. LIVE PROOF: a fresh, fully isolated 40-round soak (deploy dir xce-test-ha-40-bug1436-live) failed AGAIN at round 36/40 with 111 tracked jobs; the target primary's own -vvv log (ctrl-2, PID confirmed continuously alive via a parallel 1s-resolution forensic process/connection poller) showed a clear, damning pattern for the ENTIRE 'verify' client session: cluster.c:786 server sock 4 accepted 10 (a BRAND NEW TCP connection) immediately before EVERY SINGLE opcode get request, followed by two xce_reply sends (the >100-row chunked-get path, xce_reply_get pktlimit=100) and then cluster.c:64 (the TRACE line inside handle_endpoint_error, i.e. the connection being torn down) -- one full accept-serve-close cycle PER expect-job call, ~1799 accepts total on that one short-lived primary PID alone -- confirming the client was silently reconnecting on every single (mostly-successful) get call, exactly the reconnect storm this mechanism predicts, not a misreading of the log by a prior investigation pass. The run's own retry-storm messages ('verify_no_loss attempt 1/3: recheck failed... likely post-election client retry-storm race') were themselves an accurate but previously-misinterpreted symptom of this bug. Confirmed via codegraph_explore + direct reads of src/lib/client.c, src/lib/client_internal.h, src/lib/client_failover.c, src/lib/xceapi.c, src/lib/request.c (xce_reply_get chunking), src/controller/opget.c (do_reply_joblist/check_controller), src/lib/netio.c (endpoint_enq/handle_error/handle_output -- confirmed NOT independently implicated: zero endpoint_update_epoll WARN lines anywhere in the failing run's ctrl-2.log), and the existing src/lib/testsuite/test_libxce_retry.c StatusClassifier taxonomy tests (which confirm xce_failover_classify is INTENTIONALLY designed to accept genuine status codes in either sign -- the bug is that 5 call sites fed it a row count instead of a status code, not that the classifier itself is wrong). Two purpose-built synthetic repro harnesses (hammer2.sh: fixed 60-job list, ~5400+ calls with periodic real primary kills; hammer3.sh: fixed 495-job list, ~1500+ calls with periodic real primary kills) both ran clean with ZERO failures, because neither ever held the tracked job count at exactly 104/110/111 -- which in hindsight is the confirming, not disconfirming, result: the bug requires an EXACT count collision, not raw connection-churn volume, so a fixed-size synthetic repro that never lands on the magic number cannot hit it no matter how much volume it generates. That negative result is what motivated pivoting to the real /xce-test-ha harness (which naturally sweeps through every job count via its round-based growth) as the reproduction vehicle, per the live-capture mandate.
Fix
Added XCECLI_FAILOVER_CALL_COUNT(ac, EXPR) to src/lib/client_internal.h -- a sibling of XCECLI_FAILOVER_CALL for xce_get-family calls. It only enters the retry/classify loop when EXPR's raw result is NEGATIVE (while ((_fc_rc < 0) && xcecli_retry_next(...))); any non-negative result (0 or a genuine row count of ANY magnitude) is unconditionally treated as success and returned as-is, so it can never be misread as a retriable/terminal status code. A dead ctx (XCE_CTX_ERR) is still handled correctly: the per-op XCE_CTX_RUN guard in xceapi.c makes EXPR itself return a negative -EINVAL in that state, which enters the loop and is reclassified by xcecli_retry_decide's dead-connection branch exactly as before. Switched the 5 count-returning call sites in src/lib/client.c to the new macro: xcecli_reslist, xcecli_workerlist, xcecli_joblist (the confirmed live-reproduced offender), xcecli_resval (same xce_get_with_condition pattern, same latent risk), and xcecli_jobinfo (returns a fixed 1 on success -- lower risk since it never collides with a recognized magnitude and was previously misclassified as TERMINAL, which is harmless because TERMINAL does not rotate/rewrite rc, but fixed anyway for correctness/consistency rather than relying on that coincidence). Did NOT touch xce_failover_classify itself or any of the other 14 XCECLI_FAILOVER_CALL call sites (hello, join, jobdispatch, jobsubmit, clusterconf, ctrlstatus, update_resvals, update_jusage, jobctrl, queuectrl, hostctrl, runtask, set_pend_reason, clear_reason) -- all of those genuinely follow the 0/status-code convention the classifier is designed for, confirmed via test_libxce_retry.c's existing StatusClassifier taxonomy tests, which still pass unmodified (xcelib_test: 293 Tests 0 Failures 0 Ignored). controller_unit_test: 926 PASS / 0 FAIL / 0 Ignored (direct-binary run, ANSI-stripped grep census, exit 0). Did NOT touch run-ha-soak.sh further (its bug-1436-prior-pass workdir-scoping fix stays, unrelated to this mechanism) and did NOT add a client-side retry-more band-aid -- the actual defect was a classification-layer bug misinterpreting legitimate success data as a transport error, not a genuinely unavoidable transient condition, so the correct fix is at the classification boundary, not the retry budget.
Verification
Not recorded (migrated from buglog.json).
References
- Related: bug-1435, bug-1418, bug-854, bug-646