docs(libxce-failover): log bug-609 (known-dead EINVAL) + record 6.2 live-E2E partial status
This commit is contained in:
parent
fa65e794ec
commit
16f4ba67c7
4 changed files with 770 additions and 5 deletions
|
|
@ -9812,6 +9812,640 @@
|
|||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-16T15:55:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-605",
|
||||
"timestamp": "2026-06-16T16:30:00Z",
|
||||
"error_message": "3-node fresh HA deploy never elects a primary: all nodes stay CTRL_STANDBY, config-holder (ctrl-1) loops PreVote LOST every ~11s forever; config-less peers idle as voter-only. REPRODUCED via /xce-deploy-ha + clean restart 2026-06-16.",
|
||||
"file": "src/controller/ha.c (ha_dispatch_gate); test: src/controller/testsuite/test_ha_role.c",
|
||||
"root_cause": "ha_dispatch_gate() admitted OPCode_requestVote in CTRL_INIT/CTRL_ELECTING/CTRL_STANDBY but NEVER OPCode_preVoteRequest (opcode 38). ha-prevote added the new opcode + handler (cluster.c:440) but missed wiring it into the role-admission gate. So a STANDBY voter rejected an incoming PreVote with -XCE_ERR_STANDBY (-10014) BEFORE the handler ran -> no preVoteReply sent -> candidate's failed/empty exchange tallied as a denial (pre_vote_reply_decode_and_dispatch counts ok=0 as non-grant; vote_ep_err_handler synthesizes a denied vote). In a cold cluster (config-holder + config-less voter-only peers) every PreVote 'lost' with NO voter ever making a liveness decision (zero 'PreVote DENIED' logs), so no primary was ever elected and the config-holder looped PreVote forever. Confirmed via gdb: myreq_handler ha_dispatch_gate rc=-10014 on opcode 38. Design invariant: a PreVote (strictly non-mutating) must be at least as admissible as a real requestVote in every role.",
|
||||
"fix": "Added OPCode_preVoteRequest alongside OPCode_requestVote in ha_dispatch_gate() for CTRL_INIT, CTRL_ELECTING, and the CTRL_STANDBY switch (src/controller/ha.c). PreVote now reaches the cluster.c handler and a proper preVoteReply is sent. Added TDD test test_dispatch_gate_allows_prevote_in_voter_roles (test_ha_role.c) asserting gate(preVoteRequest)==gate(requestVote)==0 across voter roles. VERIFIED: controller unit suite OK (new test PASS); fresh /xce-deploy-ha 3-node cold start now elects ctrl-1 PRIMARY (term=2), standbys establish primary channel + config committed, prevote-LOST=0, connections 96->8.",
|
||||
"tags": [
|
||||
"ha",
|
||||
"prevote",
|
||||
"ha-dispatch-gate",
|
||||
"opcode-38",
|
||||
"cold-start",
|
||||
"bootstrap-deadlock",
|
||||
"FIXED",
|
||||
"XCE_ERR_STANDBY"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-339"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-16T09:38:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-606",
|
||||
"timestamp": "2026-06-16T17:46:00Z",
|
||||
"error_message": "After killing the primary (ctrl-1), the two surviving standbys (ctrl-2,ctrl-3) livelock in synchronized elections: PreVote win \u2192 CTRL_ELECTING \u2192 become_follower (term bumped) \u2192 ~10s \u2192 repeat, term climbing unbounded (3\u21929\u2192\u221e), no new primary ever elected. Reproduce: deploy 3-node HA, elect primary, submit 3-5 jobs, kill ctrl-1.",
|
||||
"file": "src/controller/ha.c (ha_begin_failover, ha_election_stagger_ms, ha_failover_stagger_fn); src/controller/ha.h; test: src/controller/testsuite/test_ha_role.c",
|
||||
"root_cause": "Dueling-candidates split-vote livelock \u2014 Raft \u00a75.2 randomized election timeout is missing on the FAILOVER path. Both survivors lose the primary at the same instant; their liveness polls fire on the identical fixed heartbeat_timeout, so they start PreVote+real-election in perfect lock-step (ms-aligned timestamps). Both win PreVote (neither sees a primary, so each grants the other \u2014 PreVote cannot break this tie), then both campaign at the same term, each votes for itself and denies the other (quorum 2, only 2 live nodes) \u2192 split vote \u2192 election times out (fixed election_timeout_ms, raft.c:105, NO jitter) \u2192 become_follower, term+1 \u2192 repeat forever. The priority/jitter config (election_jitter_max_ms, election_priority_step_ms) is applied ONLY at startup (ha_wait_for_heartbeat_or_timeout); the failover re-election path (liveness_poll_fn \u2192 ha_on_primary_ctrl_fd_rdhup \u2192 PreVote \u2192 ha_start_election_async \u2192 raft_start_election) has no de-synchronization. raft_on_leader_lost DOES have a priority stagger (raft.c:421) but the failover loop does not go through it.",
|
||||
"fix": "Added ha_election_stagger_ms() (ha.c/ha.h): per-node failover delay = my_priority*election_priority_step_ms + random[0,election_jitter_max_ms], re-randomized each call (mirrors the startup stagger). ha_begin_failover() now schedules the PreVote start via delaymgr after that stagger (production only; c==NULL/no-delaymgr stays synchronous so existing tests are unaffected). The delaymgr callback ha_failover_stagger_fn guards with role==STANDBY && ha_heartbeat_is_fresh \u2192 skips PreVote if a primary connected during the stagger. Effect: two survivors no longer campaign in lock-step; the higher-priority survivor wins and its heartbeats reset the other. TDD: test_election_stagger_desyncs_by_priority (bounds + step>jitter ordering). VERIFIED on kill-primary repro: ctrl-2 (stagger 630ms) won \u2192 PRIMARY term=1; ctrl-3 (stagger 1072ms) received ctrl-2's ctrlhello first \u2192 STANDBY, then its stagger timer hit the guard and skipped PreVote ('a primary is alive \u2014 skipping PreVote'). No term inflation, no livelock. NOTE: optional raft.c:105 give-up-watchdog randomization NOT needed \u2014 the failover-start stagger alone resolves it; left as future defense-in-depth.",
|
||||
"tags": [
|
||||
"ha",
|
||||
"raft",
|
||||
"election",
|
||||
"split-vote",
|
||||
"dueling-candidates",
|
||||
"livelock",
|
||||
"randomized-timeout",
|
||||
"failover",
|
||||
"stagger",
|
||||
"jitter",
|
||||
"FIXED"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-605"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-16T10:04:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-607",
|
||||
"timestamp": "2026-06-16T10:37:37.021Z",
|
||||
"error_message": "Significant refactor of ",
|
||||
"file": ".kiro/specs/libxce-failover/requirements.md",
|
||||
"root_cause": "5 lines replaced/restructured",
|
||||
"fix": "Rewrote 5\u21928 lines (5 removed) | Also: 1. If a controller returns a not-ready (no current; shall retry the request with backoff.",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"refactor",
|
||||
"md"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 2,
|
||||
"last_seen": "2026-06-16T10:38:03.618Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-608",
|
||||
"timestamp": "2026-06-16T10:36:00Z",
|
||||
"error_message": "libxce client never fails over after primary killed; xce_joblist returns 'Invalid argument' (EINVAL) indefinitely",
|
||||
"file": "src/lib/request.c",
|
||||
"root_cause": "Two gaps: (1) xce_request_with_retry() (request.c:1956) is implemented+unit-tested but NO production path calls it - every xce_*_req wrapper calls plain xce_request() (request.c:1385 etc). (2) The retry primitive keys on E_XCE_STANDBY + reply.redirect_host/port (request.c:2010), but the real controller emits a dedicated E_XCE_REDIRECT status with data_which=nodata (join.c:264-266) and never populates those trailer fields. Controller non-primary taxonomy is E_XCE_REDIRECT / E_XCE_NOMASTER / E_XCE_NOT_READY / E_XCE_STANDBY (errnox.h:22-25). So the client both bypasses the wrapper AND, even if wired, would not detect the controller's actual redirect signal.",
|
||||
"fix": "Spec libxce-failover: wire all production request paths through transparent failover; classify on actual controller statuses (REDIRECT=fast-rotate to find existing primary, address-if-present else rotate master list; NOMASTER/NOT_READY=backoff then cluster-not-ready; STANDBY=rotate); realign the existing retry unit tests to the emitted statuses. Implementation pending (requirements phase).",
|
||||
"tags": [
|
||||
"libxce",
|
||||
"failover",
|
||||
"ha",
|
||||
"E_XCE_REDIRECT",
|
||||
"retry",
|
||||
"client",
|
||||
"protocol-mismatch"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-16T10:36:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-609",
|
||||
"timestamp": "2026-06-16T10:52:01.800Z",
|
||||
"error_message": "Significant refactor of ",
|
||||
"file": ".kiro/specs/libxce-failover/requirements.md",
|
||||
"root_cause": "2 lines replaced/restructured",
|
||||
"fix": "Rewrote 6\u21929 lines (2 removed)",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"refactor",
|
||||
"md"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-16T10:52:01.800Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-610",
|
||||
"timestamp": "2026-06-16T10:52:36.535Z",
|
||||
"error_message": "Function not marked async",
|
||||
"file": ".kiro/steering/roadmap.md",
|
||||
"root_cause": "Function uses await but wasn't declared async",
|
||||
"fix": "Added async modifier",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"async-fix",
|
||||
"md"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-16T10:52:36.535Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-611",
|
||||
"timestamp": "2026-06-16T13:02:01.004Z",
|
||||
"error_message": "Significant refactor of ",
|
||||
"file": ".kiro/specs/libxce-failover/design.md",
|
||||
"root_cause": "3 lines replaced/restructured",
|
||||
"fix": "Rewrote 20\u219227 lines (3 removed) | Also: - On every reply, copy `reply.status` and **deep-c; with the reply \u2014 see request.c:243-250 hazard) + ` | Also: - Preconditions: `ctx` initialized; `conf->nmaster; exhaustion returns `-E_XCE_NOT_READY`; ctx left us",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"refactor",
|
||||
"md"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 3,
|
||||
"last_seen": "2026-06-16T13:02:46.348Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-612",
|
||||
"timestamp": "2026-06-16T13:09:39.339Z",
|
||||
"error_message": "Significant refactor of ",
|
||||
"file": ".kiro/specs/libxce-failover/design.md",
|
||||
"root_cause": "4 lines replaced/restructured",
|
||||
"fix": "Rewrote 6\u219210 lines (4 removed)",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"refactor",
|
||||
"md"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-16T13:09:39.339Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-613",
|
||||
"timestamp": "2026-06-16T13:15:11.048Z",
|
||||
"error_message": "Function not marked async",
|
||||
"file": ".kiro/specs/libxce-failover/design.md",
|
||||
"root_cause": "Function uses await but wasn't declared async",
|
||||
"fix": "Added async modifier",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"async-fix",
|
||||
"md"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-16T13:15:11.048Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-614",
|
||||
"timestamp": "2026-06-16T13:18:29.113Z",
|
||||
"error_message": "Incorrect value in code",
|
||||
"file": ".kiro/specs/libxce-failover/design.md",
|
||||
"root_cause": "Had `xce_ctx_call`",
|
||||
"fix": "Changed to `xcecli_retry_next`",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"wrong-value",
|
||||
"md"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 2,
|
||||
"last_seen": "2026-06-16T13:18:35.874Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-615",
|
||||
"timestamp": "2026-06-16T13:18:59.337Z",
|
||||
"error_message": "Significant refactor of ",
|
||||
"file": ".kiro/specs/libxce-failover/research.md",
|
||||
"root_cause": "4 lines replaced/restructured",
|
||||
"fix": "Rewrote 4\u219210 lines (4 removed)",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"refactor",
|
||||
"md"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-16T13:18:59.337Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-616",
|
||||
"timestamp": "2026-06-16T13:36:08.340Z",
|
||||
"error_message": "Significant refactor of ",
|
||||
"file": ".kiro/specs/libxce-failover/design.md",
|
||||
"root_cause": "2 lines replaced/restructured",
|
||||
"fix": "Rewrote 12\u219220 lines (2 removed) | Also: F->>E: reconnect to address; F->>E: reconnect to next master | Also: | 5.1, 5.3 | Reads+writes; no silent drop | Xcecli; | XcecliRetryLoop | client (`client.c`) | `do/whil | Also: - `src/lib/client.c` \u2014 wrap each `xcecli_*` op bod; (xcecli_retry_next(ac,&rc,is_write,&st));` loop; a",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"refactor",
|
||||
"md"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 4,
|
||||
"last_seen": "2026-06-16T13:39:20.214Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-617",
|
||||
"timestamp": "2026-06-16T23:38:24.000Z",
|
||||
"error_message": "Incorrect value in code",
|
||||
"file": ".kiro/specs/libxce-failover/design.md",
|
||||
"root_cause": "Had `redirect_host/port`",
|
||||
"fix": "Changed to `host/port`",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"wrong-value",
|
||||
"md"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 2,
|
||||
"last_seen": "2026-06-16T23:38:53.585Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-618",
|
||||
"timestamp": "2026-06-16T23:40:05.565Z",
|
||||
"error_message": "Significant refactor of ",
|
||||
"file": ".kiro/specs/libxce-failover/tasks.md",
|
||||
"root_cause": "3 lines replaced/restructured",
|
||||
"fix": "Rewrote 3\u21923 lines (3 removed)",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"refactor",
|
||||
"md"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-16T23:40:05.565Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-619",
|
||||
"timestamp": "2026-06-16T23:40:22.003Z",
|
||||
"error_message": "Wrong reference: address should be hint",
|
||||
"file": ".kiro/specs/libxce-failover/tasks.md",
|
||||
"root_cause": "Used \"address\" instead of \"hint\"",
|
||||
"fix": "Changed address \u2192 hint",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"wrong-reference",
|
||||
"md"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-16T23:40:22.003Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-620",
|
||||
"timestamp": "2026-06-16T23:55:21.886Z",
|
||||
"error_message": "Significant refactor of ",
|
||||
"file": "src/lib/client.c",
|
||||
"root_cause": "11 lines replaced/restructured",
|
||||
"fix": "Rewrote 22\u219210 lines (11 removed)",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"refactor",
|
||||
"c"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-16T23:55:21.886Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-621",
|
||||
"timestamp": "2026-06-16T23:56:22.835Z",
|
||||
"error_message": "Significant refactor of ",
|
||||
"file": "src/lib/request.c",
|
||||
"root_cause": "81 lines replaced/restructured",
|
||||
"fix": "Rewrote 106\u219214 lines (81 removed)",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"refactor",
|
||||
"c"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-16T23:56:22.835Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-622",
|
||||
"timestamp": "2026-06-16T23:56:36.669Z",
|
||||
"error_message": "Significant refactor of ",
|
||||
"file": "src/include/request.h",
|
||||
"root_cause": "22 lines replaced/restructured",
|
||||
"fix": "Rewrote 40\u219217 lines (22 removed)",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"refactor",
|
||||
"h"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-16T23:56:36.669Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-623",
|
||||
"timestamp": "2026-06-17T00:04:51.775Z",
|
||||
"error_message": "Wrong reference: XCE_MAXHOST_LEN should be XCE_MAXHOSTNAME_LEN",
|
||||
"file": ".kiro/specs/libxce-failover/design.md",
|
||||
"root_cause": "Used \"XCE_MAXHOST_LEN\" instead of \"XCE_MAXHOSTNAME_LEN\"",
|
||||
"fix": "Changed XCE_MAXHOST_LEN \u2192 XCE_MAXHOSTNAME_LEN",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"wrong-reference",
|
||||
"md"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-17T00:04:51.775Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-624",
|
||||
"timestamp": "2026-06-17T02:35:29.695Z",
|
||||
"error_message": "Incorrect value in code",
|
||||
"file": "src/lib/meson.build",
|
||||
"root_cause": "Had 'dataq.c'",
|
||||
"fix": "Changed to 'client_failover.c'",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"wrong-value",
|
||||
"build"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-17T02:35:29.695Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-625",
|
||||
"timestamp": "2026-06-17T02:47:55.586Z",
|
||||
"error_message": "Significant refactor of ",
|
||||
"file": "src/lib/client_failover.c",
|
||||
"root_cause": "4 lines replaced/restructured",
|
||||
"fix": "Rewrote 13\u219223 lines (4 removed)",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"refactor",
|
||||
"c"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-17T02:47:55.586Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-626",
|
||||
"timestamp": "2026-06-17T02:48:27.547Z",
|
||||
"error_message": "Missing guard clause",
|
||||
"file": "src/lib/client_failover.c",
|
||||
"root_cause": "No early return/throw for edge case: ac == NULL || host_out == NULL || host_sz == 0 || port_out =",
|
||||
"fix": "Added guard clause: if (ac == NULL || host_out == NULL || host_s)",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"guard-clause",
|
||||
"c"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-17T02:48:27.547Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-627",
|
||||
"timestamp": "2026-06-17T02:54:16.472Z",
|
||||
"error_message": "Significant refactor of ",
|
||||
"file": "src/include/xce/client_failover.h",
|
||||
"root_cause": "3 lines replaced/restructured",
|
||||
"fix": "Rewrote 7\u21923 lines (3 removed) | Also: * EndpointRotator \u2014 choose the next connect target; *",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"refactor",
|
||||
"h"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 2,
|
||||
"last_seen": "2026-06-17T02:54:47.480Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-628",
|
||||
"timestamp": "2026-06-17T02:55:18.468Z",
|
||||
"error_message": "Significant refactor of ",
|
||||
"file": "src/lib/client_failover.c",
|
||||
"root_cause": "2 lines replaced/restructured",
|
||||
"fix": "Rewrote 3\u21924 lines (2 removed)",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"refactor",
|
||||
"c"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-17T02:55:18.468Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-629",
|
||||
"timestamp": "2026-06-17T03:18:22.365Z",
|
||||
"error_message": "Missing guard clause",
|
||||
"file": "src/lib/client_failover.c",
|
||||
"root_cause": "No early return/throw for edge case: base >= (double",
|
||||
"fix": "Added guard clause: if (base >= (double)",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"guard-clause",
|
||||
"c"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-17T03:18:22.365Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-630",
|
||||
"timestamp": "2026-06-17T03:18:49.743Z",
|
||||
"error_message": "Significant refactor of ",
|
||||
"file": "src/lib/client_failover.c",
|
||||
"root_cause": "2 lines replaced/restructured",
|
||||
"fix": "Rewrote 9\u219221 lines (2 removed)",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"refactor",
|
||||
"c"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-17T03:18:49.743Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-631",
|
||||
"timestamp": "2026-06-17T03:31:29.120Z",
|
||||
"error_message": "Missing guard clause",
|
||||
"file": "src/lib/client_failover.c",
|
||||
"root_cause": "No early return/throw for edge case: ac == NULL",
|
||||
"fix": "Added guard clause: if (ac == NULL)",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"guard-clause",
|
||||
"c"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-17T03:31:29.120Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-632",
|
||||
"timestamp": "2026-06-17T03:39:44.187Z",
|
||||
"error_message": "Significant refactor of ",
|
||||
"file": "src/lib/client_internal.h",
|
||||
"root_cause": "6 lines replaced/restructured",
|
||||
"fix": "Rewrote 10\u21927 lines (6 removed)",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"refactor",
|
||||
"h"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-17T03:39:44.187Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-633",
|
||||
"timestamp": "2026-06-17T03:40:10.853Z",
|
||||
"error_message": "Missing guard clause",
|
||||
"file": "src/lib/client_failover.c",
|
||||
"root_cause": "No early return/throw for edge case: ac == NULL",
|
||||
"fix": "Added guard clause: if (ac == NULL)",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"guard-clause",
|
||||
"c"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-17T03:40:10.853Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-634",
|
||||
"timestamp": "2026-06-17T03:40:45.973Z",
|
||||
"error_message": "Significant refactor of ",
|
||||
"file": "src/lib/client_failover.c",
|
||||
"root_cause": "5 lines replaced/restructured",
|
||||
"fix": "Rewrote 35\u219253 lines (5 removed)",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"refactor",
|
||||
"c"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-17T03:40:45.973Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-635",
|
||||
"timestamp": "2026-06-17T03:51:08.767Z",
|
||||
"error_message": "Wrong return value",
|
||||
"file": "src/lib/client.c",
|
||||
"root_cause": "Was returning: xce_reslist(&(ctx->ctx), res);",
|
||||
"fix": "Now returns: XCECLI_FAILOVER_CALL(ctx, /*is_write=*/false,",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"return-value",
|
||||
"c"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 8,
|
||||
"last_seen": "2026-06-17T03:51:29.554Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-636",
|
||||
"timestamp": "2026-06-17T03:51:27.648Z",
|
||||
"error_message": "Significant refactor of ",
|
||||
"file": "src/lib/client.c",
|
||||
"root_cause": "2 lines replaced/restructured",
|
||||
"fix": "Rewrote 5\u21927 lines (2 removed)",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"refactor",
|
||||
"c"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-17T03:51:27.648Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-637",
|
||||
"timestamp": "2026-06-17T03:57:33.232Z",
|
||||
"error_message": "Incorrect value in code",
|
||||
"file": ".kiro/specs/libxce-failover/design.md",
|
||||
"root_cause": "Had `nop`",
|
||||
"fix": "Changed to `jobinfo`",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"wrong-value",
|
||||
"md"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 2,
|
||||
"last_seen": "2026-06-17T03:57:40.348Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-638",
|
||||
"timestamp": "2026-06-17T04:02:05.572Z",
|
||||
"error_message": "Missing guard clause",
|
||||
"file": "src/lib/client_failover.c",
|
||||
"root_cause": "No early return/throw for edge case: ac == NULL",
|
||||
"fix": "Added guard clause: if (ac == NULL)",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"guard-clause",
|
||||
"c"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-17T04:02:05.572Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-639",
|
||||
"timestamp": "2026-06-17T04:11:59.759Z",
|
||||
"error_message": "Incorrect value in code",
|
||||
"file": "src/controller/meson.build",
|
||||
"root_cause": "Had 'testsuite/test_p1_integration.c'",
|
||||
"fix": "Changed to 'testsuite/test_join_redirect.c'",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"wrong-value",
|
||||
"build"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-17T04:11:59.759Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-640",
|
||||
"timestamp": "2026-06-17T04:12:01.203Z",
|
||||
"error_message": "Wrong reference: test_p1_integration should be test_join_redirect",
|
||||
"file": "src/controller/testsuite/controller_test.c",
|
||||
"root_cause": "Used \"test_p1_integration\" instead of \"test_join_redirect\"",
|
||||
"fix": "Changed test_p1_integration \u2192 test_join_redirect",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"wrong-reference",
|
||||
"c"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 2,
|
||||
"last_seen": "2026-06-17T04:12:02.464Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-641",
|
||||
"timestamp": "2026-06-17T04:19:58.697Z",
|
||||
"error_message": "Wrong reference: test_recover_resets_err_state_to_run should be test_decide_jitter_staggers_many_clients",
|
||||
"file": "src/lib/testsuite/test_libxce_retry.c",
|
||||
"root_cause": "Used \"test_recover_resets_err_state_to_run\" instead of \"test_decide_jitter_staggers_many_clients\"",
|
||||
"fix": "Changed test_recover_resets_err_state_to_run \u2192 test_decide_jitter_staggers_many_clients",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"wrong-reference",
|
||||
"c"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-17T04:19:58.697Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-609",
|
||||
"timestamp": "2026-06-17T04:45:00Z",
|
||||
"error_message": "After primary killed, libxce client xce_joblist/jobsubmit returns 'Invalid argument' (-EINVAL) on every fresh call \u2014 transparent failover did not engage",
|
||||
"file": "src/lib/client_failover.c",
|
||||
"root_cause": "handle_disconnect leaves xce_ctx_t.state=XCE_CTX_ERR after the primary's connection drops; the per-op xceapi guard (xceapi.c:14) returns -EINVAL when state!=XCE_CTX_RUN, BEFORE the op reaches a primary. xcecli_retry_decide classified -EINVAL as TERMINAL (non-retriable), so the failover loop bailed without rotating/recovering. The ctx->state ERR->RUN recovery only ran AFTER a rotate, which a terminal classification never triggered. Surfaced by the live 3-node E2E (task 6.2); the unit tests fed status codes directly to decide() and bypassed the real guard, so they missed it.",
|
||||
"fix": "In xcecli_retry_decide, before classifying *rc, detect ac->ctx.state==XCE_CTX_ERR (known-dead connection) and treat it as a prompt transport-style failover cause (XCE_FO_RETRY_NOW, no backoff) regardless of *rc, so the loop rotates+recovers. On budget exhaustion surface -E_XCE_NOT_READY (cluster-not-ready), never -EINVAL. Regression: test_decide_known_dead_ctx_retries_not_terminal.",
|
||||
"tags": [
|
||||
"libxce",
|
||||
"failover",
|
||||
"EINVAL",
|
||||
"XCE_CTX_ERR",
|
||||
"reconnect",
|
||||
"E2E"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-608"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-17T04:45:00Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue