From 16f4ba67c7828915afb7b7e297322bdcacfd29c3 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Wed, 17 Jun 2026 04:46:45 +0000 Subject: [PATCH] docs(libxce-failover): log bug-609 (known-dead EINVAL) + record 6.2 live-E2E partial status --- .kiro/specs/libxce-failover/tasks.md | 1 + .wolf/anatomy.md | 38 +- .wolf/buglog.json | 634 +++++++++++++++++++++++++++ .wolf/memory.md | 102 +++++ 4 files changed, 770 insertions(+), 5 deletions(-) diff --git a/.kiro/specs/libxce-failover/tasks.md b/.kiro/specs/libxce-failover/tasks.md index 57fd427..7961fb1 100644 --- a/.kiro/specs/libxce-failover/tasks.md +++ b/.kiro/specs/libxce-failover/tasks.md @@ -111,3 +111,4 @@ - 5.2: `XCECLI_FF_CALL` macro (single-shot, NO retry loop/backoff/budget): lock → if known-dead (xcecli_ff_needs_reconnect = ctx.state==XCE_CTX_ERR) one rotate+recover → one send → unlock. Applied at the 3 wire chokepoints xcecli_notify/heartbeat/nop (notify_* delegate to xcecli_notify). wakeup/try_wakeup are LOCAL-only (cbuf_tryput own wbuf, no wire send) so correctly NOT wrapped. - 5.3: `fill_redirect_reply(xce_reply_t*, const ha_state_t*)` (non-static, join.c) sets redirect_host=ha.primary_host + redirect_port=(uint16_t)ha.primary_port; called ONLY on the E_XCE_REDIRECT branch before xce_reply. Audit: join.c is the SOLE E_XCE_REDIRECT emitter. NOMASTER path unchanged (8.2). host alias (no copy) safe: ha lives cluster-lifetime, xce_reply serializes synchronously. New test_join_redirect.c registered in controller_test.c + meson.build. - 6.1: full lib suite green (161 tests). All 14 reqs (2.4,2.5,3.2,3.3,4.1-4.5,5.2,6.2,6.3,8.3,8.4) map to genuine passing tests. Gap-filled 6.3 with `test_decide_jitter_staggers_many_clients` (64 distinct-seed carriers → staggered delays, anti-thundering-herd). No production bugs found. +- 6.2 (LIVE E2E — partial): Live 3-node cluster (deploy-ha.sh) was brought up and DID elect a primary; killing the primary triggered server-side failover (ctrl-2 self-elected term=2). The run REPRODUCED the original `Invalid argument` client bug → root-caused + FIXED (bug-609, known-dead-ctx override) + regression unit test added. The clean within-session client transparent-recovery could NOT be visually demonstrated in this sandbox (background controller processes are reaped between tool calls; nix runtime env needed for backgrounded procs; lib build cleans the controller binary). RECOMMEND final visual confirmation in a stable env: build all, `/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993`, run xce_test (set masters / client / jobsub / joblist), kill the primary, confirm joblist reports cluster-not-ready then transparently recovers (no EINVAL). diff --git a/.wolf/anatomy.md b/.wolf/anatomy.md index 33bcaa1..2404e65 100644 --- a/.wolf/anatomy.md +++ b/.wolf/anatomy.md @@ -1,7 +1,7 @@ # anatomy.md -> Auto-maintained by OpenWolf. Last scanned: 2026-06-17T02:32:09.045Z -> Files: 510 tracked | Anatomy hits: 0 | Misses: 0 +> Auto-maintained by OpenWolf. Last scanned: 2026-06-17T04:38:41.384Z +> Files: 523 tracked | Anatomy hits: 0 | Misses: 0 ## ./ @@ -365,11 +365,11 @@ ## .kiro/specs/libxce-failover/ - `brief.md` — Brief: libxce-failover (~2238 tok) -- `design.md` — Design Document — libxce-failover (~8277 tok) +- `design.md` — Design Document — libxce-failover (~8279 tok) - `requirements.md` — Requirements Document (~2933 tok) - `research.md` — Research Log — libxce-failover (~2344 tok) - `spec.json` (~128 tok) -- `tasks.md` — Implementation Plan (~2305 tok) +- `tasks.md` — Implementation Plan (~3305 tok) ## .kiro/specs/raft-library/ @@ -723,8 +723,36 @@ - `6180aec3d86b1cb2980831e6c75a8ac6216fd14493f52a1539ad73a750bd10b7.json` (~5053 tok) - `61f1333461274a206f965221fa31fd7ce681795ce2c4271c9455759288e83655.json` (~4360 tok) +## src/controller/ + +- `join.c` — include "config.h" (~2783 tok) +- `meson.build` (~1216 tok) + +## src/controller/testsuite/ + +- `controller_test.c` — include (~1487 tok) +- `test_join_redirect.c` — include (~937 tok) + +## src/include/ + +- `client.h` — if !defined(_CLIENT_H_) (~1164 tok) +- `mylock.h` — if !defined(_MYLOCK_H_) (~273 tok) +- `xcectx.h` — if !defined(_XCECTX_H_) (~678 tok) + +## src/include/xce/ + +- `client_failover.h` — ifndef XCE_CLIENT_FAILOVER_H (~618 tok) + +## src/lib/ + +- `client_failover.c` — define _GNU_SOURCE (~4435 tok) +- `client_internal.h` — if !defined(_CLIENT_INTERNAL_H_) (~3500 tok) +- `client.c` — include (~4370 tok) +- `meson.build` (~1535 tok) +- `xcectx.c` — include (~2223 tok) + ## src/lib/testsuite/ - `failover_harness.h` — Declares xce_cluster_config_t (~1393 tok) - `failover_harness.h` — test-only header for the libxce-failover harness: fo_master_cursor_t, FO_STATUS_TABLE rows, fo_run_threads(); no production failover symbols (~900 tok) -- `test_libxce_retry.c` — libxce-failover reusable harness (fake-master cursor, status taxonomy table, N-thread runner) + GREEN self-tests; consumed by later failover tasks 2.x/3.x/5.x (~2600 tok) +- `test_libxce_retry.c` — Declares struct (~18140 tok) diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 24df7b5..0fa5c8d 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -9812,6 +9812,640 @@ "related_bugs": [], "occurrences": 1, "last_seen": "2026-06-16T15:55:00Z" + }, + { + "id": "bug-605", + "timestamp": "2026-06-16T16:30:00Z", + "error_message": "3-node fresh HA deploy never elects a primary: all nodes stay CTRL_STANDBY, config-holder (ctrl-1) loops PreVote LOST every ~11s forever; config-less peers idle as voter-only. REPRODUCED via /xce-deploy-ha + clean restart 2026-06-16.", + "file": "src/controller/ha.c (ha_dispatch_gate); test: src/controller/testsuite/test_ha_role.c", + "root_cause": "ha_dispatch_gate() admitted OPCode_requestVote in CTRL_INIT/CTRL_ELECTING/CTRL_STANDBY but NEVER OPCode_preVoteRequest (opcode 38). ha-prevote added the new opcode + handler (cluster.c:440) but missed wiring it into the role-admission gate. So a STANDBY voter rejected an incoming PreVote with -XCE_ERR_STANDBY (-10014) BEFORE the handler ran -> no preVoteReply sent -> candidate's failed/empty exchange tallied as a denial (pre_vote_reply_decode_and_dispatch counts ok=0 as non-grant; vote_ep_err_handler synthesizes a denied vote). In a cold cluster (config-holder + config-less voter-only peers) every PreVote 'lost' with NO voter ever making a liveness decision (zero 'PreVote DENIED' logs), so no primary was ever elected and the config-holder looped PreVote forever. Confirmed via gdb: myreq_handler ha_dispatch_gate rc=-10014 on opcode 38. Design invariant: a PreVote (strictly non-mutating) must be at least as admissible as a real requestVote in every role.", + "fix": "Added OPCode_preVoteRequest alongside OPCode_requestVote in ha_dispatch_gate() for CTRL_INIT, CTRL_ELECTING, and the CTRL_STANDBY switch (src/controller/ha.c). PreVote now reaches the cluster.c handler and a proper preVoteReply is sent. Added TDD test test_dispatch_gate_allows_prevote_in_voter_roles (test_ha_role.c) asserting gate(preVoteRequest)==gate(requestVote)==0 across voter roles. VERIFIED: controller unit suite OK (new test PASS); fresh /xce-deploy-ha 3-node cold start now elects ctrl-1 PRIMARY (term=2), standbys establish primary channel + config committed, prevote-LOST=0, connections 96->8.", + "tags": [ + "ha", + "prevote", + "ha-dispatch-gate", + "opcode-38", + "cold-start", + "bootstrap-deadlock", + "FIXED", + "XCE_ERR_STANDBY" + ], + "related_bugs": [ + "bug-339" + ], + "occurrences": 1, + "last_seen": "2026-06-16T09:38:00Z" + }, + { + "id": "bug-606", + "timestamp": "2026-06-16T17:46:00Z", + "error_message": "After killing the primary (ctrl-1), the two surviving standbys (ctrl-2,ctrl-3) livelock in synchronized elections: PreVote win \u2192 CTRL_ELECTING \u2192 become_follower (term bumped) \u2192 ~10s \u2192 repeat, term climbing unbounded (3\u21929\u2192\u221e), no new primary ever elected. Reproduce: deploy 3-node HA, elect primary, submit 3-5 jobs, kill ctrl-1.", + "file": "src/controller/ha.c (ha_begin_failover, ha_election_stagger_ms, ha_failover_stagger_fn); src/controller/ha.h; test: src/controller/testsuite/test_ha_role.c", + "root_cause": "Dueling-candidates split-vote livelock \u2014 Raft \u00a75.2 randomized election timeout is missing on the FAILOVER path. Both survivors lose the primary at the same instant; their liveness polls fire on the identical fixed heartbeat_timeout, so they start PreVote+real-election in perfect lock-step (ms-aligned timestamps). Both win PreVote (neither sees a primary, so each grants the other \u2014 PreVote cannot break this tie), then both campaign at the same term, each votes for itself and denies the other (quorum 2, only 2 live nodes) \u2192 split vote \u2192 election times out (fixed election_timeout_ms, raft.c:105, NO jitter) \u2192 become_follower, term+1 \u2192 repeat forever. The priority/jitter config (election_jitter_max_ms, election_priority_step_ms) is applied ONLY at startup (ha_wait_for_heartbeat_or_timeout); the failover re-election path (liveness_poll_fn \u2192 ha_on_primary_ctrl_fd_rdhup \u2192 PreVote \u2192 ha_start_election_async \u2192 raft_start_election) has no de-synchronization. raft_on_leader_lost DOES have a priority stagger (raft.c:421) but the failover loop does not go through it.", + "fix": "Added ha_election_stagger_ms() (ha.c/ha.h): per-node failover delay = my_priority*election_priority_step_ms + random[0,election_jitter_max_ms], re-randomized each call (mirrors the startup stagger). ha_begin_failover() now schedules the PreVote start via delaymgr after that stagger (production only; c==NULL/no-delaymgr stays synchronous so existing tests are unaffected). The delaymgr callback ha_failover_stagger_fn guards with role==STANDBY && ha_heartbeat_is_fresh \u2192 skips PreVote if a primary connected during the stagger. Effect: two survivors no longer campaign in lock-step; the higher-priority survivor wins and its heartbeats reset the other. TDD: test_election_stagger_desyncs_by_priority (bounds + step>jitter ordering). VERIFIED on kill-primary repro: ctrl-2 (stagger 630ms) won \u2192 PRIMARY term=1; ctrl-3 (stagger 1072ms) received ctrl-2's ctrlhello first \u2192 STANDBY, then its stagger timer hit the guard and skipped PreVote ('a primary is alive \u2014 skipping PreVote'). No term inflation, no livelock. NOTE: optional raft.c:105 give-up-watchdog randomization NOT needed \u2014 the failover-start stagger alone resolves it; left as future defense-in-depth.", + "tags": [ + "ha", + "raft", + "election", + "split-vote", + "dueling-candidates", + "livelock", + "randomized-timeout", + "failover", + "stagger", + "jitter", + "FIXED" + ], + "related_bugs": [ + "bug-605" + ], + "occurrences": 1, + "last_seen": "2026-06-16T10:04:00Z" + }, + { + "id": "bug-607", + "timestamp": "2026-06-16T10:37:37.021Z", + "error_message": "Significant refactor of ", + "file": ".kiro/specs/libxce-failover/requirements.md", + "root_cause": "5 lines replaced/restructured", + "fix": "Rewrote 5\u21928 lines (5 removed) | Also: 1. If a controller returns a not-ready (no current; shall retry the request with backoff.", + "tags": [ + "auto-detected", + "refactor", + "md" + ], + "related_bugs": [], + "occurrences": 2, + "last_seen": "2026-06-16T10:38:03.618Z" + }, + { + "id": "bug-608", + "timestamp": "2026-06-16T10:36:00Z", + "error_message": "libxce client never fails over after primary killed; xce_joblist returns 'Invalid argument' (EINVAL) indefinitely", + "file": "src/lib/request.c", + "root_cause": "Two gaps: (1) xce_request_with_retry() (request.c:1956) is implemented+unit-tested but NO production path calls it - every xce_*_req wrapper calls plain xce_request() (request.c:1385 etc). (2) The retry primitive keys on E_XCE_STANDBY + reply.redirect_host/port (request.c:2010), but the real controller emits a dedicated E_XCE_REDIRECT status with data_which=nodata (join.c:264-266) and never populates those trailer fields. Controller non-primary taxonomy is E_XCE_REDIRECT / E_XCE_NOMASTER / E_XCE_NOT_READY / E_XCE_STANDBY (errnox.h:22-25). So the client both bypasses the wrapper AND, even if wired, would not detect the controller's actual redirect signal.", + "fix": "Spec libxce-failover: wire all production request paths through transparent failover; classify on actual controller statuses (REDIRECT=fast-rotate to find existing primary, address-if-present else rotate master list; NOMASTER/NOT_READY=backoff then cluster-not-ready; STANDBY=rotate); realign the existing retry unit tests to the emitted statuses. Implementation pending (requirements phase).", + "tags": [ + "libxce", + "failover", + "ha", + "E_XCE_REDIRECT", + "retry", + "client", + "protocol-mismatch" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-16T10:36:00Z" + }, + { + "id": "bug-609", + "timestamp": "2026-06-16T10:52:01.800Z", + "error_message": "Significant refactor of ", + "file": ".kiro/specs/libxce-failover/requirements.md", + "root_cause": "2 lines replaced/restructured", + "fix": "Rewrote 6\u21929 lines (2 removed)", + "tags": [ + "auto-detected", + "refactor", + "md" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-16T10:52:01.800Z" + }, + { + "id": "bug-610", + "timestamp": "2026-06-16T10:52:36.535Z", + "error_message": "Function not marked async", + "file": ".kiro/steering/roadmap.md", + "root_cause": "Function uses await but wasn't declared async", + "fix": "Added async modifier", + "tags": [ + "auto-detected", + "async-fix", + "md" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-16T10:52:36.535Z" + }, + { + "id": "bug-611", + "timestamp": "2026-06-16T13:02:01.004Z", + "error_message": "Significant refactor of ", + "file": ".kiro/specs/libxce-failover/design.md", + "root_cause": "3 lines replaced/restructured", + "fix": "Rewrote 20\u219227 lines (3 removed) | Also: - On every reply, copy `reply.status` and **deep-c; with the reply \u2014 see request.c:243-250 hazard) + ` | Also: - Preconditions: `ctx` initialized; `conf->nmaster; exhaustion returns `-E_XCE_NOT_READY`; ctx left us", + "tags": [ + "auto-detected", + "refactor", + "md" + ], + "related_bugs": [], + "occurrences": 3, + "last_seen": "2026-06-16T13:02:46.348Z" + }, + { + "id": "bug-612", + "timestamp": "2026-06-16T13:09:39.339Z", + "error_message": "Significant refactor of ", + "file": ".kiro/specs/libxce-failover/design.md", + "root_cause": "4 lines replaced/restructured", + "fix": "Rewrote 6\u219210 lines (4 removed)", + "tags": [ + "auto-detected", + "refactor", + "md" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-16T13:09:39.339Z" + }, + { + "id": "bug-613", + "timestamp": "2026-06-16T13:15:11.048Z", + "error_message": "Function not marked async", + "file": ".kiro/specs/libxce-failover/design.md", + "root_cause": "Function uses await but wasn't declared async", + "fix": "Added async modifier", + "tags": [ + "auto-detected", + "async-fix", + "md" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-16T13:15:11.048Z" + }, + { + "id": "bug-614", + "timestamp": "2026-06-16T13:18:29.113Z", + "error_message": "Incorrect value in code", + "file": ".kiro/specs/libxce-failover/design.md", + "root_cause": "Had `xce_ctx_call`", + "fix": "Changed to `xcecli_retry_next`", + "tags": [ + "auto-detected", + "wrong-value", + "md" + ], + "related_bugs": [], + "occurrences": 2, + "last_seen": "2026-06-16T13:18:35.874Z" + }, + { + "id": "bug-615", + "timestamp": "2026-06-16T13:18:59.337Z", + "error_message": "Significant refactor of ", + "file": ".kiro/specs/libxce-failover/research.md", + "root_cause": "4 lines replaced/restructured", + "fix": "Rewrote 4\u219210 lines (4 removed)", + "tags": [ + "auto-detected", + "refactor", + "md" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-16T13:18:59.337Z" + }, + { + "id": "bug-616", + "timestamp": "2026-06-16T13:36:08.340Z", + "error_message": "Significant refactor of ", + "file": ".kiro/specs/libxce-failover/design.md", + "root_cause": "2 lines replaced/restructured", + "fix": "Rewrote 12\u219220 lines (2 removed) | Also: F->>E: reconnect to address; F->>E: reconnect to next master | Also: | 5.1, 5.3 | Reads+writes; no silent drop | Xcecli; | XcecliRetryLoop | client (`client.c`) | `do/whil | Also: - `src/lib/client.c` \u2014 wrap each `xcecli_*` op bod; (xcecli_retry_next(ac,&rc,is_write,&st));` loop; a", + "tags": [ + "auto-detected", + "refactor", + "md" + ], + "related_bugs": [], + "occurrences": 4, + "last_seen": "2026-06-16T13:39:20.214Z" + }, + { + "id": "bug-617", + "timestamp": "2026-06-16T23:38:24.000Z", + "error_message": "Incorrect value in code", + "file": ".kiro/specs/libxce-failover/design.md", + "root_cause": "Had `redirect_host/port`", + "fix": "Changed to `host/port`", + "tags": [ + "auto-detected", + "wrong-value", + "md" + ], + "related_bugs": [], + "occurrences": 2, + "last_seen": "2026-06-16T23:38:53.585Z" + }, + { + "id": "bug-618", + "timestamp": "2026-06-16T23:40:05.565Z", + "error_message": "Significant refactor of ", + "file": ".kiro/specs/libxce-failover/tasks.md", + "root_cause": "3 lines replaced/restructured", + "fix": "Rewrote 3\u21923 lines (3 removed)", + "tags": [ + "auto-detected", + "refactor", + "md" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-16T23:40:05.565Z" + }, + { + "id": "bug-619", + "timestamp": "2026-06-16T23:40:22.003Z", + "error_message": "Wrong reference: address should be hint", + "file": ".kiro/specs/libxce-failover/tasks.md", + "root_cause": "Used \"address\" instead of \"hint\"", + "fix": "Changed address \u2192 hint", + "tags": [ + "auto-detected", + "wrong-reference", + "md" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-16T23:40:22.003Z" + }, + { + "id": "bug-620", + "timestamp": "2026-06-16T23:55:21.886Z", + "error_message": "Significant refactor of ", + "file": "src/lib/client.c", + "root_cause": "11 lines replaced/restructured", + "fix": "Rewrote 22\u219210 lines (11 removed)", + "tags": [ + "auto-detected", + "refactor", + "c" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-16T23:55:21.886Z" + }, + { + "id": "bug-621", + "timestamp": "2026-06-16T23:56:22.835Z", + "error_message": "Significant refactor of ", + "file": "src/lib/request.c", + "root_cause": "81 lines replaced/restructured", + "fix": "Rewrote 106\u219214 lines (81 removed)", + "tags": [ + "auto-detected", + "refactor", + "c" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-16T23:56:22.835Z" + }, + { + "id": "bug-622", + "timestamp": "2026-06-16T23:56:36.669Z", + "error_message": "Significant refactor of ", + "file": "src/include/request.h", + "root_cause": "22 lines replaced/restructured", + "fix": "Rewrote 40\u219217 lines (22 removed)", + "tags": [ + "auto-detected", + "refactor", + "h" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-16T23:56:36.669Z" + }, + { + "id": "bug-623", + "timestamp": "2026-06-17T00:04:51.775Z", + "error_message": "Wrong reference: XCE_MAXHOST_LEN should be XCE_MAXHOSTNAME_LEN", + "file": ".kiro/specs/libxce-failover/design.md", + "root_cause": "Used \"XCE_MAXHOST_LEN\" instead of \"XCE_MAXHOSTNAME_LEN\"", + "fix": "Changed XCE_MAXHOST_LEN \u2192 XCE_MAXHOSTNAME_LEN", + "tags": [ + "auto-detected", + "wrong-reference", + "md" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-17T00:04:51.775Z" + }, + { + "id": "bug-624", + "timestamp": "2026-06-17T02:35:29.695Z", + "error_message": "Incorrect value in code", + "file": "src/lib/meson.build", + "root_cause": "Had 'dataq.c'", + "fix": "Changed to 'client_failover.c'", + "tags": [ + "auto-detected", + "wrong-value", + "build" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-17T02:35:29.695Z" + }, + { + "id": "bug-625", + "timestamp": "2026-06-17T02:47:55.586Z", + "error_message": "Significant refactor of ", + "file": "src/lib/client_failover.c", + "root_cause": "4 lines replaced/restructured", + "fix": "Rewrote 13\u219223 lines (4 removed)", + "tags": [ + "auto-detected", + "refactor", + "c" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-17T02:47:55.586Z" + }, + { + "id": "bug-626", + "timestamp": "2026-06-17T02:48:27.547Z", + "error_message": "Missing guard clause", + "file": "src/lib/client_failover.c", + "root_cause": "No early return/throw for edge case: ac == NULL || host_out == NULL || host_sz == 0 || port_out =", + "fix": "Added guard clause: if (ac == NULL || host_out == NULL || host_s)", + "tags": [ + "auto-detected", + "guard-clause", + "c" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-17T02:48:27.547Z" + }, + { + "id": "bug-627", + "timestamp": "2026-06-17T02:54:16.472Z", + "error_message": "Significant refactor of ", + "file": "src/include/xce/client_failover.h", + "root_cause": "3 lines replaced/restructured", + "fix": "Rewrote 7\u21923 lines (3 removed) | Also: * EndpointRotator \u2014 choose the next connect target; *", + "tags": [ + "auto-detected", + "refactor", + "h" + ], + "related_bugs": [], + "occurrences": 2, + "last_seen": "2026-06-17T02:54:47.480Z" + }, + { + "id": "bug-628", + "timestamp": "2026-06-17T02:55:18.468Z", + "error_message": "Significant refactor of ", + "file": "src/lib/client_failover.c", + "root_cause": "2 lines replaced/restructured", + "fix": "Rewrote 3\u21924 lines (2 removed)", + "tags": [ + "auto-detected", + "refactor", + "c" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-17T02:55:18.468Z" + }, + { + "id": "bug-629", + "timestamp": "2026-06-17T03:18:22.365Z", + "error_message": "Missing guard clause", + "file": "src/lib/client_failover.c", + "root_cause": "No early return/throw for edge case: base >= (double", + "fix": "Added guard clause: if (base >= (double)", + "tags": [ + "auto-detected", + "guard-clause", + "c" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-17T03:18:22.365Z" + }, + { + "id": "bug-630", + "timestamp": "2026-06-17T03:18:49.743Z", + "error_message": "Significant refactor of ", + "file": "src/lib/client_failover.c", + "root_cause": "2 lines replaced/restructured", + "fix": "Rewrote 9\u219221 lines (2 removed)", + "tags": [ + "auto-detected", + "refactor", + "c" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-17T03:18:49.743Z" + }, + { + "id": "bug-631", + "timestamp": "2026-06-17T03:31:29.120Z", + "error_message": "Missing guard clause", + "file": "src/lib/client_failover.c", + "root_cause": "No early return/throw for edge case: ac == NULL", + "fix": "Added guard clause: if (ac == NULL)", + "tags": [ + "auto-detected", + "guard-clause", + "c" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-17T03:31:29.120Z" + }, + { + "id": "bug-632", + "timestamp": "2026-06-17T03:39:44.187Z", + "error_message": "Significant refactor of ", + "file": "src/lib/client_internal.h", + "root_cause": "6 lines replaced/restructured", + "fix": "Rewrote 10\u21927 lines (6 removed)", + "tags": [ + "auto-detected", + "refactor", + "h" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-17T03:39:44.187Z" + }, + { + "id": "bug-633", + "timestamp": "2026-06-17T03:40:10.853Z", + "error_message": "Missing guard clause", + "file": "src/lib/client_failover.c", + "root_cause": "No early return/throw for edge case: ac == NULL", + "fix": "Added guard clause: if (ac == NULL)", + "tags": [ + "auto-detected", + "guard-clause", + "c" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-17T03:40:10.853Z" + }, + { + "id": "bug-634", + "timestamp": "2026-06-17T03:40:45.973Z", + "error_message": "Significant refactor of ", + "file": "src/lib/client_failover.c", + "root_cause": "5 lines replaced/restructured", + "fix": "Rewrote 35\u219253 lines (5 removed)", + "tags": [ + "auto-detected", + "refactor", + "c" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-17T03:40:45.973Z" + }, + { + "id": "bug-635", + "timestamp": "2026-06-17T03:51:08.767Z", + "error_message": "Wrong return value", + "file": "src/lib/client.c", + "root_cause": "Was returning: xce_reslist(&(ctx->ctx), res);", + "fix": "Now returns: XCECLI_FAILOVER_CALL(ctx, /*is_write=*/false,", + "tags": [ + "auto-detected", + "return-value", + "c" + ], + "related_bugs": [], + "occurrences": 8, + "last_seen": "2026-06-17T03:51:29.554Z" + }, + { + "id": "bug-636", + "timestamp": "2026-06-17T03:51:27.648Z", + "error_message": "Significant refactor of ", + "file": "src/lib/client.c", + "root_cause": "2 lines replaced/restructured", + "fix": "Rewrote 5\u21927 lines (2 removed)", + "tags": [ + "auto-detected", + "refactor", + "c" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-17T03:51:27.648Z" + }, + { + "id": "bug-637", + "timestamp": "2026-06-17T03:57:33.232Z", + "error_message": "Incorrect value in code", + "file": ".kiro/specs/libxce-failover/design.md", + "root_cause": "Had `nop`", + "fix": "Changed to `jobinfo`", + "tags": [ + "auto-detected", + "wrong-value", + "md" + ], + "related_bugs": [], + "occurrences": 2, + "last_seen": "2026-06-17T03:57:40.348Z" + }, + { + "id": "bug-638", + "timestamp": "2026-06-17T04:02:05.572Z", + "error_message": "Missing guard clause", + "file": "src/lib/client_failover.c", + "root_cause": "No early return/throw for edge case: ac == NULL", + "fix": "Added guard clause: if (ac == NULL)", + "tags": [ + "auto-detected", + "guard-clause", + "c" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-17T04:02:05.572Z" + }, + { + "id": "bug-639", + "timestamp": "2026-06-17T04:11:59.759Z", + "error_message": "Incorrect value in code", + "file": "src/controller/meson.build", + "root_cause": "Had 'testsuite/test_p1_integration.c'", + "fix": "Changed to 'testsuite/test_join_redirect.c'", + "tags": [ + "auto-detected", + "wrong-value", + "build" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-17T04:11:59.759Z" + }, + { + "id": "bug-640", + "timestamp": "2026-06-17T04:12:01.203Z", + "error_message": "Wrong reference: test_p1_integration should be test_join_redirect", + "file": "src/controller/testsuite/controller_test.c", + "root_cause": "Used \"test_p1_integration\" instead of \"test_join_redirect\"", + "fix": "Changed test_p1_integration \u2192 test_join_redirect", + "tags": [ + "auto-detected", + "wrong-reference", + "c" + ], + "related_bugs": [], + "occurrences": 2, + "last_seen": "2026-06-17T04:12:02.464Z" + }, + { + "id": "bug-641", + "timestamp": "2026-06-17T04:19:58.697Z", + "error_message": "Wrong reference: test_recover_resets_err_state_to_run should be test_decide_jitter_staggers_many_clients", + "file": "src/lib/testsuite/test_libxce_retry.c", + "root_cause": "Used \"test_recover_resets_err_state_to_run\" instead of \"test_decide_jitter_staggers_many_clients\"", + "fix": "Changed test_recover_resets_err_state_to_run \u2192 test_decide_jitter_staggers_many_clients", + "tags": [ + "auto-detected", + "wrong-reference", + "c" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-17T04:19:58.697Z" + }, + { + "id": "bug-609", + "timestamp": "2026-06-17T04:45:00Z", + "error_message": "After primary killed, libxce client xce_joblist/jobsubmit returns 'Invalid argument' (-EINVAL) on every fresh call \u2014 transparent failover did not engage", + "file": "src/lib/client_failover.c", + "root_cause": "handle_disconnect leaves xce_ctx_t.state=XCE_CTX_ERR after the primary's connection drops; the per-op xceapi guard (xceapi.c:14) returns -EINVAL when state!=XCE_CTX_RUN, BEFORE the op reaches a primary. xcecli_retry_decide classified -EINVAL as TERMINAL (non-retriable), so the failover loop bailed without rotating/recovering. The ctx->state ERR->RUN recovery only ran AFTER a rotate, which a terminal classification never triggered. Surfaced by the live 3-node E2E (task 6.2); the unit tests fed status codes directly to decide() and bypassed the real guard, so they missed it.", + "fix": "In xcecli_retry_decide, before classifying *rc, detect ac->ctx.state==XCE_CTX_ERR (known-dead connection) and treat it as a prompt transport-style failover cause (XCE_FO_RETRY_NOW, no backoff) regardless of *rc, so the loop rotates+recovers. On budget exhaustion surface -E_XCE_NOT_READY (cluster-not-ready), never -EINVAL. Regression: test_decide_known_dead_ctx_retries_not_terminal.", + "tags": [ + "libxce", + "failover", + "EINVAL", + "XCE_CTX_ERR", + "reconnect", + "E2E" + ], + "related_bugs": [ + "bug-608" + ], + "occurrences": 1, + "last_seen": "2026-06-17T04:45:00Z" } ] } \ No newline at end of file diff --git a/.wolf/memory.md b/.wolf/memory.md index f13a17f..4b61211 100644 --- a/.wolf/memory.md +++ b/.wolf/memory.md @@ -1908,3 +1908,105 @@ Implemented sub-spec 3/4 of ha-consistency-invariants autonomously (per-task imp | 19:27 | Created src/lib/testsuite/test_libxce_retry.c | — | ~3468 | | 02:28 | task 1.2: built libxce-failover test harness (cursor/taxonomy/threads) + 7 GREEN self-tests | test_libxce_retry.c, failover_harness.h | Ok:1 Fail:0 | ~9k | | 19:32 | Edited .kiro/specs/libxce-failover/tasks.md | inline fix | ~17 | +| 19:34 | Created src/include/xce/client_failover.h | — | ~559 | +| 19:34 | Edited src/lib/testsuite/test_libxce_retry.c | 2→4 lines | ~29 | +| 19:34 | Edited src/lib/testsuite/test_libxce_retry.c | modified fo_expected_to_action() | ~1398 | +| 19:35 | Created src/lib/client_failover.c | — | ~560 | +| 19:35 | Edited src/lib/meson.build | 2→3 lines | ~24 | +| 19:39 | Edited .kiro/specs/libxce-failover/tasks.md | inline fix | ~11 | +| 19:46 | Edited src/lib/testsuite/test_libxce_retry.c | 6→9 lines | ~60 | +| 19:46 | Edited src/lib/testsuite/test_libxce_retry.c | modified fo_make_client_masters() | ~1809 | +| 19:46 | Edited src/lib/testsuite/test_libxce_retry.c | 3→7 lines | ~87 | +| 19:47 | Edited src/include/xce/client_failover.h | modified XCE_CLIENT_FAILOVER_H() | ~53 | +| 19:47 | Edited src/include/xce/client_failover.h | modified xce_failover_next_target() | ~439 | +| 19:47 | Edited src/lib/client_failover.c | expanded (+10 lines) | ~249 | +| 19:48 | Edited src/lib/client_failover.c | 3→4 lines | ~23 | +| 19:48 | Edited src/lib/client_failover.c | 4→3 lines | ~17 | +| 19:48 | Edited src/lib/client_failover.c | added 7 condition(s) | ~869 | +| 19:53 | Edited src/lib/client_internal.h | 3→4 lines | ~16 | +| 19:54 | Edited src/lib/client_internal.h | modified struct() | ~310 | +| 19:54 | Edited src/include/xce/client_failover.h | 7→3 lines | ~26 | +| 19:54 | Edited src/include/xce/client_failover.h | reduced (-25 lines) | ~90 | +| 19:55 | Edited src/lib/client_failover.c | 3→4 lines | ~64 | +| 19:56 | Edited .kiro/specs/libxce-failover/tasks.md | inline fix | ~16 | +| 20:02 | Edited src/lib/testsuite/test_libxce_retry.c | added 1 condition(s) | ~1824 | +| 20:03 | Edited src/include/xcectx.h | expanded (+15 lines) | ~242 | +| 20:03 | Edited src/lib/xcectx.c | added 2 condition(s) | ~337 | +| 20:11 | Edited .kiro/specs/libxce-failover/tasks.md | inline fix | ~23 | +| 20:15 | Edited src/lib/client_internal.h | 8→10 lines | ~58 | +| 20:16 | Edited src/lib/client_internal.h | modified struct() | ~802 | +| 20:17 | Edited src/lib/testsuite/test_libxce_retry.c | added 4 condition(s) | ~2707 | +| 20:17 | Edited src/lib/testsuite/test_libxce_retry.c | expanded (+7 lines) | ~129 | +| 20:17 | Edited src/lib/client_failover.c | 11→14 lines | ~122 | +| 20:18 | Edited src/lib/client_failover.c | added 15 condition(s) | ~1384 | +| 20:18 | Edited src/lib/client_failover.c | modified if() | ~259 | +| 03:22 | impl libxce-failover 3.1 decision loop (xcecli_retry_decide pure + xcecli_retry_next) | src/lib/client_failover.c, client_internal.h, testsuite/test_libxce_retry.c | GREEN lib Ok:1 Fail:0, 7 new tests pass | ~38k | +| 20:27 | Edited .kiro/specs/libxce-failover/tasks.md | inline fix | ~20 | +| 20:30 | Edited src/lib/testsuite/test_libxce_retry.c | modified test_recover_resets_err_state_to_run() | ~1105 | +| 20:30 | Edited src/lib/testsuite/test_libxce_retry.c | 2→7 lines | ~90 | +| 20:31 | Edited src/include/mylock.h | 3→4 lines | ~34 | +| 20:31 | Edited src/lib/client_internal.h | modified CI2() | ~340 | +| 20:31 | Edited src/lib/client_failover.c | added 2 condition(s) | ~415 | +| 20:31 | Edited src/lib/client_failover.c | modified if() | ~130 | +| 03:33 | task 3.2 reconnect recovery: xcecli_ctx_recover (CI2 ERR->RUN state reset + CI1 wbuf drain) wired into xcecli_retry_next | src/lib/client_failover.c, client_internal.h, mylock.h, testsuite/test_libxce_retry.c | GREEN Ok:1 Fail:0 (5 new tests) | ~9k | +| 20:36 | Edited .kiro/specs/libxce-failover/tasks.md | inline fix | ~22 | +| 20:38 | Edited src/lib/testsuite/test_libxce_retry.c | modified struct() | ~1616 | +| 20:38 | Edited src/lib/testsuite/test_libxce_retry.c | 3→8 lines | ~102 | +| 20:39 | Edited src/include/client.h | modified void() | ~191 | +| 20:39 | Edited src/include/client.h | 1→3 lines | ~54 | +| 20:39 | Edited src/lib/client_internal.h | 10→7 lines | ~64 | +| 20:39 | Edited src/lib/client_internal.h | expanded (+18 lines) | ~258 | +| 20:40 | Edited src/lib/client.c | added 1 condition(s) | ~194 | +| 20:40 | Edited src/lib/client_failover.c | added 2 condition(s) | ~223 | +| 20:40 | Edited src/lib/client_failover.c | added 1 condition(s) | ~678 | +| 03:42 | task 4.1 ReconnectRejoinHook: public typedef+setter in client.h, xcecli_set_reconnect_handler in client.c, xcecli_invoke_rejoin in client_failover.c wired into retry_next (recover→hook→resume; failed rejoin re-rotates) | client.h, client.c, client_failover.c, client_internal.h, test_libxce_retry.c | lib build Ok:1 Fail:0 | ~9k | +| 20:45 | Edited .kiro/specs/libxce-failover/tasks.md | inline fix | ~14 | +| 20:50 | Edited src/lib/testsuite/test_libxce_retry.c | added 1 condition(s) | ~1428 | +| 20:50 | Edited src/lib/testsuite/test_libxce_retry.c | 2→4 lines | ~46 | +| 20:50 | Edited src/lib/testsuite/test_libxce_retry.c | 2→3 lines | ~32 | +| 20:50 | Edited src/lib/client_internal.h | modified serialised() | ~607 | +| 20:51 | Edited src/lib/client.c | modified if() | ~46 | +| 20:51 | Edited src/lib/client.c | 2→3 lines | ~33 | +| 20:51 | Edited src/lib/client.c | 2→3 lines | ~34 | +| 20:51 | Edited src/lib/client.c | 2→3 lines | ~35 | +| 20:51 | Edited src/lib/client.c | 2→3 lines | ~34 | +| 20:51 | Edited src/lib/client.c | 2→3 lines | ~36 | +| 20:51 | Edited src/lib/client.c | 2→3 lines | ~33 | +| 20:51 | Edited src/lib/client.c | 2→3 lines | ~34 | +| 20:51 | Edited src/lib/client.c | 4→5 lines | ~40 | +| 20:51 | Edited src/lib/client.c | 5→7 lines | ~53 | +| 20:51 | Edited src/lib/client.c | 2→3 lines | ~36 | +| 20:51 | Edited src/lib/client.c | 4→5 lines | ~37 | +| 20:51 | Edited src/lib/client.c | 4→5 lines | ~38 | +| 20:51 | Edited src/lib/client.c | 3→4 lines | ~37 | +| 20:51 | Edited src/lib/client.c | 4→5 lines | ~36 | +| 20:51 | Edited src/lib/client.c | 3→4 lines | ~39 | +| 20:51 | Edited src/lib/client.c | 3→4 lines | ~38 | +| 20:51 | Edited src/lib/client.c | 3→4 lines | ~38 | +| 03:53 | task 5.1: wrap reply-bearing xcecli_* ops in XCECLI_FAILOVER_CALL failover loop under per-ctx lock | client.c, client_internal.h, test_libxce_retry.c | lib build Ok:1 Fail:0; +2 concurrency tests PASS | ~28k | +| 20:57 | Edited .kiro/specs/libxce-failover/tasks.md | inline fix | ~24 | +| 20:57 | Edited .kiro/specs/libxce-failover/design.md | inline fix | ~44 | +| 20:57 | Edited .kiro/specs/libxce-failover/design.md | inline fix | ~78 | +| 21:01 | Edited src/lib/testsuite/test_libxce_retry.c | modified struct() | ~1265 | +| 21:01 | Edited src/lib/testsuite/test_libxce_retry.c | 3→8 lines | ~98 | +| 21:01 | Edited src/lib/client_internal.h | added 1 condition(s) | ~897 | +| 21:02 | Edited src/lib/client_failover.c | added 1 condition(s) | ~230 | +| 21:02 | Edited src/lib/client.c | modified if() | ~83 | +| 21:02 | Edited src/lib/client.c | modified if() | ~90 | +| 21:02 | Edited src/lib/client.c | modified if() | ~116 | +| 21:04 | Edited src/lib/client.c | 5→6 lines | ~103 | +| 21:08 | Edited .kiro/specs/libxce-failover/tasks.md | inline fix | ~16 | +| 21:11 | Created src/controller/testsuite/test_join_redirect.c | — | ~937 | +| 21:11 | Edited src/controller/meson.build | 2→3 lines | ~29 | +| 21:12 | Edited src/controller/testsuite/controller_test.c | 2→3 lines | ~30 | +| 21:12 | Edited src/controller/testsuite/controller_test.c | 2→3 lines | ~24 | +| 21:12 | Edited src/controller/join.c | modified validate_host() | ~203 | +| 21:12 | Edited src/controller/join.c | 7→8 lines | ~75 | +| 04:13 | task 5.3: populate redirect addr on E_XCE_REDIRECT (fill_redirect_reply) + new test | join.c, test_join_redirect.c, meson.build, controller_test.c | controller tests PASS | ~8k | +| 21:16 | Edited .kiro/specs/libxce-failover/tasks.md | inline fix | ~20 | +| 21:19 | Edited src/lib/testsuite/test_libxce_retry.c | added 4 condition(s) | ~1344 | +| 21:19 | Edited src/lib/testsuite/test_libxce_retry.c | 2→3 lines | ~46 | +| 04:21 | task 6.1: gap-fill jitter-staggering test for many clients (Req 6.3) | src/lib/testsuite/test_libxce_retry.c | GREEN Ok:1 Fail:0, 46 tests, new test 48 distinct delays/64 clients | ~12k | +| 21:25 | Edited .kiro/specs/libxce-failover/tasks.md | inline fix | ~13 | +| 21:33 | Edited src/lib/client_failover.c | added 3 condition(s) | ~352 | +| 21:38 | Edited src/lib/testsuite/test_libxce_retry.c | modified test_decide_known_dead_ctx_retries_not_terminal() | ~466 |