wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

docs(libxce-failover): log bug-609 (known-dead EINVAL) + record 6.2 live-E2E partial status

Browse source
This commit is contained in:
Rongsong Shen 2026-06-17 04:46:45 +00:00
parent fa65e794ec
commit 16f4ba67c7
4 changed files with 770 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

1
.kiro/specs/libxce-failover/tasks.md
View file

@ -111,3 +111,4 @@
- 5.2: `XCECLI_FF_CALL` macro (single-shot, NO retry loop/backoff/budget): lock → if known-dead (xcecli_ff_needs_reconnect = ctx.state==XCE_CTX_ERR) one rotate+recover → one send → unlock. Applied at the 3 wire chokepoints xcecli_notify/heartbeat/nop (notify_* delegate to xcecli_notify). wakeup/try_wakeup are LOCAL-only (cbuf_tryput own wbuf, no wire send) so correctly NOT wrapped.
- 5.3: `fill_redirect_reply(xce_reply_t*, const ha_state_t*)` (non-static, join.c) sets redirect_host=ha.primary_host + redirect_port=(uint16_t)ha.primary_port; called ONLY on the E_XCE_REDIRECT branch before xce_reply. Audit: join.c is the SOLE E_XCE_REDIRECT emitter. NOMASTER path unchanged (8.2). host alias (no copy) safe: ha lives cluster-lifetime, xce_reply serializes synchronously. New test_join_redirect.c registered in controller_test.c + meson.build.
- 6.1: full lib suite green (161 tests). All 14 reqs (2.4,2.5,3.2,3.3,4.1-4.5,5.2,6.2,6.3,8.3,8.4) map to genuine passing tests. Gap-filled 6.3 with `test_decide_jitter_staggers_many_clients` (64 distinct-seed carriers → staggered delays, anti-thundering-herd). No production bugs found.
- 6.2 (LIVE E2E — partial): Live 3-node cluster (deploy-ha.sh) was brought up and DID elect a primary; killing the primary triggered server-side failover (ctrl-2 self-elected term=2). The run REPRODUCED the original `Invalid argument` client bug → root-caused + FIXED (bug-609, known-dead-ctx override) + regression unit test added. The clean within-session client transparent-recovery could NOT be visually demonstrated in this sandbox (background controller processes are reaped between tool calls; nix runtime env needed for backgrounded procs; lib build cleans the controller binary). RECOMMEND final visual confirmation in a stable env: build all, `/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993`, run xce_test (set masters / client / jobsub / joblist), kill the primary, confirm joblist reports cluster-not-ready then transparently recovers (no EINVAL).

38
.wolf/anatomy.md
View file

@ -1,7 +1,7 @@
# anatomy.md
> Auto-maintained by OpenWolf. Last scanned: 2026-06-17T02:32:09.045Z
> Files: 510 tracked | Anatomy hits: 0 | Misses: 0
> Auto-maintained by OpenWolf. Last scanned: 2026-06-17T04:38:41.384Z
> Files: 523 tracked | Anatomy hits: 0 | Misses: 0
## ./
@ -365,11 +365,11 @@
## .kiro/specs/libxce-failover/
- `brief.md` — Brief: libxce-failover (~2238 tok)
- `design.md` — Design Document — libxce-failover (~8277 tok)
- `design.md` — Design Document — libxce-failover (~8279 tok)
- `requirements.md` — Requirements Document (~2933 tok)
- `research.md` — Research Log — libxce-failover (~2344 tok)
- `spec.json` (~128 tok)
- `tasks.md` — Implementation Plan (~2305 tok)
- `tasks.md` — Implementation Plan (~3305 tok)
## .kiro/specs/raft-library/
@ -723,8 +723,36 @@
- `6180aec3d86b1cb2980831e6c75a8ac6216fd14493f52a1539ad73a750bd10b7.json` (~5053 tok)
- `61f1333461274a206f965221fa31fd7ce681795ce2c4271c9455759288e83655.json` (~4360 tok)
## src/controller/
- `join.c` — include "config.h" (~2783 tok)
- `meson.build` (~1216 tok)
## src/controller/testsuite/
- `controller_test.c` — include <argp.h> (~1487 tok)
- `test_join_redirect.c` — include <stdint.h> (~937 tok)
## src/include/
- `client.h` — if !defined(_CLIENT_H_) (~1164 tok)
- `mylock.h` — if !defined(_MYLOCK_H_) (~273 tok)
- `xcectx.h` — if !defined(_XCECTX_H_) (~678 tok)
## src/include/xce/
- `client_failover.h` — ifndef XCE_CLIENT_FAILOVER_H (~618 tok)
## src/lib/
- `client_failover.c` — define _GNU_SOURCE (~4435 tok)
- `client_internal.h` — if !defined(_CLIENT_INTERNAL_H_) (~3500 tok)
- `client.c` — include <errno.h> (~4370 tok)
- `meson.build` (~1535 tok)
- `xcectx.c` — include <errno.h> (~2223 tok)
## src/lib/testsuite/
- `failover_harness.h` — Declares xce_cluster_config_t (~1393 tok)
- `failover_harness.h` — test-only header for the libxce-failover harness: fo_master_cursor_t, FO_STATUS_TABLE rows, fo_run_threads(); no production failover symbols (~900 tok)
- `test_libxce_retry.c` — libxce-failover reusable harness (fake-master cursor, status taxonomy table, N-thread runner) + GREEN self-tests; consumed by later failover tasks 2.x/3.x/5.x (~2600 tok)
- `test_libxce_retry.c` — Declares struct (~18140 tok)

634
.wolf/buglog.json
View file

@ -9812,6 +9812,640 @@
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-16T15:55:00Z"
},
{
"id": "bug-605",
"timestamp": "2026-06-16T16:30:00Z",
"error_message": "3-node fresh HA deploy never elects a primary: all nodes stay CTRL_STANDBY, config-holder (ctrl-1) loops PreVote LOST every ~11s forever; config-less peers idle as voter-only. REPRODUCED via /xce-deploy-ha + clean restart 2026-06-16.",
"file": "src/controller/ha.c (ha_dispatch_gate); test: src/controller/testsuite/test_ha_role.c",
"root_cause": "ha_dispatch_gate() admitted OPCode_requestVote in CTRL_INIT/CTRL_ELECTING/CTRL_STANDBY but NEVER OPCode_preVoteRequest (opcode 38). ha-prevote added the new opcode + handler (cluster.c:440) but missed wiring it into the role-admission gate. So a STANDBY voter rejected an incoming PreVote with -XCE_ERR_STANDBY (-10014) BEFORE the handler ran -> no preVoteReply sent -> candidate's failed/empty exchange tallied as a denial (pre_vote_reply_decode_and_dispatch counts ok=0 as non-grant; vote_ep_err_handler synthesizes a denied vote). In a cold cluster (config-holder + config-less voter-only peers) every PreVote 'lost' with NO voter ever making a liveness decision (zero 'PreVote DENIED' logs), so no primary was ever elected and the config-holder looped PreVote forever. Confirmed via gdb: myreq_handler ha_dispatch_gate rc=-10014 on opcode 38. Design invariant: a PreVote (strictly non-mutating) must be at least as admissible as a real requestVote in every role.",
"fix": "Added OPCode_preVoteRequest alongside OPCode_requestVote in ha_dispatch_gate() for CTRL_INIT, CTRL_ELECTING, and the CTRL_STANDBY switch (src/controller/ha.c). PreVote now reaches the cluster.c handler and a proper preVoteReply is sent. Added TDD test test_dispatch_gate_allows_prevote_in_voter_roles (test_ha_role.c) asserting gate(preVoteRequest)==gate(requestVote)==0 across voter roles. VERIFIED: controller unit suite OK (new test PASS); fresh /xce-deploy-ha 3-node cold start now elects ctrl-1 PRIMARY (term=2), standbys establish primary channel + config committed, prevote-LOST=0, connections 96->8.",
"tags": [
"ha",
"prevote",
"ha-dispatch-gate",
"opcode-38",
"cold-start",
"bootstrap-deadlock",
"FIXED",
"XCE_ERR_STANDBY"
],
"related_bugs": [
"bug-339"
],
"occurrences": 1,
"last_seen": "2026-06-16T09:38:00Z"
},
{
"id": "bug-606",
"timestamp": "2026-06-16T17:46:00Z",
"error_message": "After killing the primary (ctrl-1), the two surviving standbys (ctrl-2,ctrl-3) livelock in synchronized elections: PreVote win \u2192 CTRL_ELECTING \u2192 become_follower (term bumped) \u2192 ~10s \u2192 repeat, term climbing unbounded (3\u21929\u2192\u221e), no new primary ever elected. Reproduce: deploy 3-node HA, elect primary, submit 3-5 jobs, kill ctrl-1.",
"file": "src/controller/ha.c (ha_begin_failover, ha_election_stagger_ms, ha_failover_stagger_fn); src/controller/ha.h; test: src/controller/testsuite/test_ha_role.c",
"root_cause": "Dueling-candidates split-vote livelock \u2014 Raft \u00a75.2 randomized election timeout is missing on the FAILOVER path. Both survivors lose the primary at the same instant; their liveness polls fire on the identical fixed heartbeat_timeout, so they start PreVote+real-election in perfect lock-step (ms-aligned timestamps). Both win PreVote (neither sees a primary, so each grants the other \u2014 PreVote cannot break this tie), then both campaign at the same term, each votes for itself and denies the other (quorum 2, only 2 live nodes) \u2192 split vote \u2192 election times out (fixed election_timeout_ms, raft.c:105, NO jitter) \u2192 become_follower, term+1 \u2192 repeat forever. The priority/jitter config (election_jitter_max_ms, election_priority_step_ms) is applied ONLY at startup (ha_wait_for_heartbeat_or_timeout); the failover re-election path (liveness_poll_fn \u2192 ha_on_primary_ctrl_fd_rdhup \u2192 PreVote \u2192 ha_start_election_async \u2192 raft_start_election) has no de-synchronization. raft_on_leader_lost DOES have a priority stagger (raft.c:421) but the failover loop does not go through it.",
"fix": "Added ha_election_stagger_ms() (ha.c/ha.h): per-node failover delay = my_priority*election_priority_step_ms + random[0,election_jitter_max_ms], re-randomized each call (mirrors the startup stagger). ha_begin_failover() now schedules the PreVote start via delaymgr after that stagger (production only; c==NULL/no-delaymgr stays synchronous so existing tests are unaffected). The delaymgr callback ha_failover_stagger_fn guards with role==STANDBY && ha_heartbeat_is_fresh \u2192 skips PreVote if a primary connected during the stagger. Effect: two survivors no longer campaign in lock-step; the higher-priority survivor wins and its heartbeats reset the other. TDD: test_election_stagger_desyncs_by_priority (bounds + step>jitter ordering). VERIFIED on kill-primary repro: ctrl-2 (stagger 630ms) won \u2192 PRIMARY term=1; ctrl-3 (stagger 1072ms) received ctrl-2's ctrlhello first \u2192 STANDBY, then its stagger timer hit the guard and skipped PreVote ('a primary is alive \u2014 skipping PreVote'). No term inflation, no livelock. NOTE: optional raft.c:105 give-up-watchdog randomization NOT needed \u2014 the failover-start stagger alone resolves it; left as future defense-in-depth.",
"tags": [
"ha",
"raft",
"election",
"split-vote",
"dueling-candidates",
"livelock",
"randomized-timeout",
"failover",
"stagger",
"jitter",
"FIXED"
],
"related_bugs": [
"bug-605"
],
"occurrences": 1,
"last_seen": "2026-06-16T10:04:00Z"
},
{
"id": "bug-607",
"timestamp": "2026-06-16T10:37:37.021Z",
"error_message": "Significant refactor of ",
"file": ".kiro/specs/libxce-failover/requirements.md",
"root_cause": "5 lines replaced/restructured",
"fix": "Rewrote 5\u21928 lines (5 removed) | Also: 1. If a controller returns a not-ready (no current; shall retry the request with backoff.",
"tags": [
"auto-detected",
"refactor",
"md"
],
"related_bugs": [],
"occurrences": 2,
"last_seen": "2026-06-16T10:38:03.618Z"
},
{
"id": "bug-608",
"timestamp": "2026-06-16T10:36:00Z",
"error_message": "libxce client never fails over after primary killed; xce_joblist returns 'Invalid argument' (EINVAL) indefinitely",
"file": "src/lib/request.c",
"root_cause": "Two gaps: (1) xce_request_with_retry() (request.c:1956) is implemented+unit-tested but NO production path calls it - every xce_*_req wrapper calls plain xce_request() (request.c:1385 etc). (2) The retry primitive keys on E_XCE_STANDBY + reply.redirect_host/port (request.c:2010), but the real controller emits a dedicated E_XCE_REDIRECT status with data_which=nodata (join.c:264-266) and never populates those trailer fields. Controller non-primary taxonomy is E_XCE_REDIRECT / E_XCE_NOMASTER / E_XCE_NOT_READY / E_XCE_STANDBY (errnox.h:22-25). So the client both bypasses the wrapper AND, even if wired, would not detect the controller's actual redirect signal.",
"fix": "Spec libxce-failover: wire all production request paths through transparent failover; classify on actual controller statuses (REDIRECT=fast-rotate to find existing primary, address-if-present else rotate master list; NOMASTER/NOT_READY=backoff then cluster-not-ready; STANDBY=rotate); realign the existing retry unit tests to the emitted statuses. Implementation pending (requirements phase).",
"tags": [
"libxce",
"failover",
"ha",
"E_XCE_REDIRECT",
"retry",
"client",
"protocol-mismatch"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-16T10:36:00Z"
},
{
"id": "bug-609",
"timestamp": "2026-06-16T10:52:01.800Z",
"error_message": "Significant refactor of ",
"file": ".kiro/specs/libxce-failover/requirements.md",
"root_cause": "2 lines replaced/restructured",
"fix": "Rewrote 6\u21929 lines (2 removed)",
"tags": [
"auto-detected",
"refactor",
"md"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-16T10:52:01.800Z"
},
{
"id": "bug-610",
"timestamp": "2026-06-16T10:52:36.535Z",
"error_message": "Function not marked async",
"file": ".kiro/steering/roadmap.md",
"root_cause": "Function uses await but wasn't declared async",
"fix": "Added async modifier",
"tags": [
"auto-detected",
"async-fix",
"md"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-16T10:52:36.535Z"
},
{
"id": "bug-611",
"timestamp": "2026-06-16T13:02:01.004Z",
"error_message": "Significant refactor of ",
"file": ".kiro/specs/libxce-failover/design.md",
"root_cause": "3 lines replaced/restructured",
"fix": "Rewrote 20\u219227 lines (3 removed) | Also: - On every reply, copy `reply.status` and **deep-c; with the reply \u2014 see request.c:243-250 hazard) + ` | Also: - Preconditions: `ctx` initialized; `conf->nmaster; exhaustion returns `-E_XCE_NOT_READY`; ctx left us",
"tags": [
"auto-detected",
"refactor",
"md"
],
"related_bugs": [],
"occurrences": 3,
"last_seen": "2026-06-16T13:02:46.348Z"
},
{
"id": "bug-612",
"timestamp": "2026-06-16T13:09:39.339Z",
"error_message": "Significant refactor of ",
"file": ".kiro/specs/libxce-failover/design.md",
"root_cause": "4 lines replaced/restructured",
"fix": "Rewrote 6\u219210 lines (4 removed)",
"tags": [
"auto-detected",
"refactor",
"md"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-16T13:09:39.339Z"
},
{
"id": "bug-613",
"timestamp": "2026-06-16T13:15:11.048Z",
"error_message": "Function not marked async",
"file": ".kiro/specs/libxce-failover/design.md",
"root_cause": "Function uses await but wasn't declared async",
"fix": "Added async modifier",
"tags": [
"auto-detected",
"async-fix",
"md"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-16T13:15:11.048Z"
},
{
"id": "bug-614",
"timestamp": "2026-06-16T13:18:29.113Z",
"error_message": "Incorrect value in code",
"file": ".kiro/specs/libxce-failover/design.md",
"root_cause": "Had `xce_ctx_call`",
"fix": "Changed to `xcecli_retry_next`",
"tags": [
"auto-detected",
"wrong-value",
"md"
],
"related_bugs": [],
"occurrences": 2,
"last_seen": "2026-06-16T13:18:35.874Z"
},
{
"id": "bug-615",
"timestamp": "2026-06-16T13:18:59.337Z",
"error_message": "Significant refactor of ",
"file": ".kiro/specs/libxce-failover/research.md",
"root_cause": "4 lines replaced/restructured",
"fix": "Rewrote 4\u219210 lines (4 removed)",
"tags": [
"auto-detected",
"refactor",
"md"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-16T13:18:59.337Z"
},
{
"id": "bug-616",
"timestamp": "2026-06-16T13:36:08.340Z",
"error_message": "Significant refactor of ",
"file": ".kiro/specs/libxce-failover/design.md",
"root_cause": "2 lines replaced/restructured",
"fix": "Rewrote 12\u219220 lines (2 removed) | Also: F->>E: reconnect to address; F->>E: reconnect to next master | Also: | 5.1, 5.3 | Reads+writes; no silent drop | Xcecli; | XcecliRetryLoop | client (`client.c`) | `do/whil | Also: - `src/lib/client.c` \u2014 wrap each `xcecli_*` op bod; (xcecli_retry_next(ac,&rc,is_write,&st));` loop; a",
"tags": [
"auto-detected",
"refactor",
"md"
],
"related_bugs": [],
"occurrences": 4,
"last_seen": "2026-06-16T13:39:20.214Z"
},
{
"id": "bug-617",
"timestamp": "2026-06-16T23:38:24.000Z",
"error_message": "Incorrect value in code",
"file": ".kiro/specs/libxce-failover/design.md",
"root_cause": "Had `redirect_host/port`",
"fix": "Changed to `host/port`",
"tags": [
"auto-detected",
"wrong-value",
"md"
],
"related_bugs": [],
"occurrences": 2,
"last_seen": "2026-06-16T23:38:53.585Z"
},
{
"id": "bug-618",
"timestamp": "2026-06-16T23:40:05.565Z",
"error_message": "Significant refactor of ",
"file": ".kiro/specs/libxce-failover/tasks.md",
"root_cause": "3 lines replaced/restructured",
"fix": "Rewrote 3\u21923 lines (3 removed)",
"tags": [
"auto-detected",
"refactor",
"md"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-16T23:40:05.565Z"
},
{
"id": "bug-619",
"timestamp": "2026-06-16T23:40:22.003Z",
"error_message": "Wrong reference: address should be hint",
"file": ".kiro/specs/libxce-failover/tasks.md",
"root_cause": "Used \"address\" instead of \"hint\"",
"fix": "Changed address \u2192 hint",
"tags": [
"auto-detected",
"wrong-reference",
"md"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-16T23:40:22.003Z"
},
{
"id": "bug-620",
"timestamp": "2026-06-16T23:55:21.886Z",
"error_message": "Significant refactor of ",
"file": "src/lib/client.c",
"root_cause": "11 lines replaced/restructured",
"fix": "Rewrote 22\u219210 lines (11 removed)",
"tags": [
"auto-detected",
"refactor",
"c"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-16T23:55:21.886Z"
},
{
"id": "bug-621",
"timestamp": "2026-06-16T23:56:22.835Z",
"error_message": "Significant refactor of ",
"file": "src/lib/request.c",
"root_cause": "81 lines replaced/restructured",
"fix": "Rewrote 106\u219214 lines (81 removed)",
"tags": [
"auto-detected",
"refactor",
"c"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-16T23:56:22.835Z"
},
{
"id": "bug-622",
"timestamp": "2026-06-16T23:56:36.669Z",
"error_message": "Significant refactor of ",
"file": "src/include/request.h",
"root_cause": "22 lines replaced/restructured",
"fix": "Rewrote 40\u219217 lines (22 removed)",
"tags": [
"auto-detected",
"refactor",
"h"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-16T23:56:36.669Z"
},
{
"id": "bug-623",
"timestamp": "2026-06-17T00:04:51.775Z",
"error_message": "Wrong reference: XCE_MAXHOST_LEN should be XCE_MAXHOSTNAME_LEN",
"file": ".kiro/specs/libxce-failover/design.md",
"root_cause": "Used \"XCE_MAXHOST_LEN\" instead of \"XCE_MAXHOSTNAME_LEN\"",
"fix": "Changed XCE_MAXHOST_LEN \u2192 XCE_MAXHOSTNAME_LEN",
"tags": [
"auto-detected",
"wrong-reference",
"md"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-17T00:04:51.775Z"
},
{
"id": "bug-624",
"timestamp": "2026-06-17T02:35:29.695Z",
"error_message": "Incorrect value in code",
"file": "src/lib/meson.build",
"root_cause": "Had 'dataq.c'",
"fix": "Changed to 'client_failover.c'",
"tags": [
"auto-detected",
"wrong-value",
"build"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-17T02:35:29.695Z"
},
{
"id": "bug-625",
"timestamp": "2026-06-17T02:47:55.586Z",
"error_message": "Significant refactor of ",
"file": "src/lib/client_failover.c",
"root_cause": "4 lines replaced/restructured",
"fix": "Rewrote 13\u219223 lines (4 removed)",
"tags": [
"auto-detected",
"refactor",
"c"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-17T02:47:55.586Z"
},
{
"id": "bug-626",
"timestamp": "2026-06-17T02:48:27.547Z",
"error_message": "Missing guard clause",
"file": "src/lib/client_failover.c",
"root_cause": "No early return/throw for edge case: ac == NULL || host_out == NULL || host_sz == 0 || port_out =",
"fix": "Added guard clause: if (ac == NULL || host_out == NULL || host_s)",
"tags": [
"auto-detected",
"guard-clause",
"c"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-17T02:48:27.547Z"
},
{
"id": "bug-627",
"timestamp": "2026-06-17T02:54:16.472Z",
"error_message": "Significant refactor of ",
"file": "src/include/xce/client_failover.h",
"root_cause": "3 lines replaced/restructured",
"fix": "Rewrote 7\u21923 lines (3 removed) | Also: * EndpointRotator \u2014 choose the next connect target; *",
"tags": [
"auto-detected",
"refactor",
"h"
],
"related_bugs": [],
"occurrences": 2,
"last_seen": "2026-06-17T02:54:47.480Z"
},
{
"id": "bug-628",
"timestamp": "2026-06-17T02:55:18.468Z",
"error_message": "Significant refactor of ",
"file": "src/lib/client_failover.c",
"root_cause": "2 lines replaced/restructured",
"fix": "Rewrote 3\u21924 lines (2 removed)",
"tags": [
"auto-detected",
"refactor",
"c"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-17T02:55:18.468Z"
},
{
"id": "bug-629",
"timestamp": "2026-06-17T03:18:22.365Z",
"error_message": "Missing guard clause",
"file": "src/lib/client_failover.c",
"root_cause": "No early return/throw for edge case: base >= (double",
"fix": "Added guard clause: if (base >= (double)",
"tags": [
"auto-detected",
"guard-clause",
"c"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-17T03:18:22.365Z"
},
{
"id": "bug-630",
"timestamp": "2026-06-17T03:18:49.743Z",
"error_message": "Significant refactor of ",
"file": "src/lib/client_failover.c",
"root_cause": "2 lines replaced/restructured",
"fix": "Rewrote 9\u219221 lines (2 removed)",
"tags": [
"auto-detected",
"refactor",
"c"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-17T03:18:49.743Z"
},
{
"id": "bug-631",
"timestamp": "2026-06-17T03:31:29.120Z",
"error_message": "Missing guard clause",
"file": "src/lib/client_failover.c",
"root_cause": "No early return/throw for edge case: ac == NULL",
"fix": "Added guard clause: if (ac == NULL)",
"tags": [
"auto-detected",
"guard-clause",
"c"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-17T03:31:29.120Z"
},
{
"id": "bug-632",
"timestamp": "2026-06-17T03:39:44.187Z",
"error_message": "Significant refactor of ",
"file": "src/lib/client_internal.h",
"root_cause": "6 lines replaced/restructured",
"fix": "Rewrote 10\u21927 lines (6 removed)",
"tags": [
"auto-detected",
"refactor",
"h"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-17T03:39:44.187Z"
},
{
"id": "bug-633",
"timestamp": "2026-06-17T03:40:10.853Z",
"error_message": "Missing guard clause",
"file": "src/lib/client_failover.c",
"root_cause": "No early return/throw for edge case: ac == NULL",
"fix": "Added guard clause: if (ac == NULL)",
"tags": [
"auto-detected",
"guard-clause",
"c"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-17T03:40:10.853Z"
},
{
"id": "bug-634",
"timestamp": "2026-06-17T03:40:45.973Z",
"error_message": "Significant refactor of ",
"file": "src/lib/client_failover.c",
"root_cause": "5 lines replaced/restructured",
"fix": "Rewrote 35\u219253 lines (5 removed)",
"tags": [
"auto-detected",
"refactor",
"c"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-17T03:40:45.973Z"
},
{
"id": "bug-635",
"timestamp": "2026-06-17T03:51:08.767Z",
"error_message": "Wrong return value",
"file": "src/lib/client.c",
"root_cause": "Was returning: xce_reslist(&(ctx->ctx), res);",
"fix": "Now returns: XCECLI_FAILOVER_CALL(ctx, /*is_write=*/false,",
"tags": [
"auto-detected",
"return-value",
"c"
],
"related_bugs": [],
"occurrences": 8,
"last_seen": "2026-06-17T03:51:29.554Z"
},
{
"id": "bug-636",
"timestamp": "2026-06-17T03:51:27.648Z",
"error_message": "Significant refactor of ",
"file": "src/lib/client.c",
"root_cause": "2 lines replaced/restructured",
"fix": "Rewrote 5\u21927 lines (2 removed)",
"tags": [
"auto-detected",
"refactor",
"c"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-17T03:51:27.648Z"
},
{
"id": "bug-637",
"timestamp": "2026-06-17T03:57:33.232Z",
"error_message": "Incorrect value in code",
"file": ".kiro/specs/libxce-failover/design.md",
"root_cause": "Had `nop`",
"fix": "Changed to `jobinfo`",
"tags": [
"auto-detected",
"wrong-value",
"md"
],
"related_bugs": [],
"occurrences": 2,
"last_seen": "2026-06-17T03:57:40.348Z"
},
{
"id": "bug-638",
"timestamp": "2026-06-17T04:02:05.572Z",
"error_message": "Missing guard clause",
"file": "src/lib/client_failover.c",
"root_cause": "No early return/throw for edge case: ac == NULL",
"fix": "Added guard clause: if (ac == NULL)",
"tags": [
"auto-detected",
"guard-clause",
"c"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-17T04:02:05.572Z"
},
{
"id": "bug-639",
"timestamp": "2026-06-17T04:11:59.759Z",
"error_message": "Incorrect value in code",
"file": "src/controller/meson.build",
"root_cause": "Had 'testsuite/test_p1_integration.c'",
"fix": "Changed to 'testsuite/test_join_redirect.c'",
"tags": [
"auto-detected",
"wrong-value",
"build"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-17T04:11:59.759Z"
},
{
"id": "bug-640",
"timestamp": "2026-06-17T04:12:01.203Z",
"error_message": "Wrong reference: test_p1_integration should be test_join_redirect",
"file": "src/controller/testsuite/controller_test.c",
"root_cause": "Used \"test_p1_integration\" instead of \"test_join_redirect\"",
"fix": "Changed test_p1_integration \u2192 test_join_redirect",
"tags": [
"auto-detected",
"wrong-reference",
"c"
],
"related_bugs": [],
"occurrences": 2,
"last_seen": "2026-06-17T04:12:02.464Z"
},
{
"id": "bug-641",
"timestamp": "2026-06-17T04:19:58.697Z",
"error_message": "Wrong reference: test_recover_resets_err_state_to_run should be test_decide_jitter_staggers_many_clients",
"file": "src/lib/testsuite/test_libxce_retry.c",
"root_cause": "Used \"test_recover_resets_err_state_to_run\" instead of \"test_decide_jitter_staggers_many_clients\"",
"fix": "Changed test_recover_resets_err_state_to_run \u2192 test_decide_jitter_staggers_many_clients",
"tags": [
"auto-detected",
"wrong-reference",
"c"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-17T04:19:58.697Z"
},
{
"id": "bug-609",
"timestamp": "2026-06-17T04:45:00Z",
"error_message": "After primary killed, libxce client xce_joblist/jobsubmit returns 'Invalid argument' (-EINVAL) on every fresh call \u2014 transparent failover did not engage",
"file": "src/lib/client_failover.c",
"root_cause": "handle_disconnect leaves xce_ctx_t.state=XCE_CTX_ERR after the primary's connection drops; the per-op xceapi guard (xceapi.c:14) returns -EINVAL when state!=XCE_CTX_RUN, BEFORE the op reaches a primary. xcecli_retry_decide classified -EINVAL as TERMINAL (non-retriable), so the failover loop bailed without rotating/recovering. The ctx->state ERR->RUN recovery only ran AFTER a rotate, which a terminal classification never triggered. Surfaced by the live 3-node E2E (task 6.2); the unit tests fed status codes directly to decide() and bypassed the real guard, so they missed it.",
"fix": "In xcecli_retry_decide, before classifying *rc, detect ac->ctx.state==XCE_CTX_ERR (known-dead connection) and treat it as a prompt transport-style failover cause (XCE_FO_RETRY_NOW, no backoff) regardless of *rc, so the loop rotates+recovers. On budget exhaustion surface -E_XCE_NOT_READY (cluster-not-ready), never -EINVAL. Regression: test_decide_known_dead_ctx_retries_not_terminal.",
"tags": [
"libxce",
"failover",
"EINVAL",
"XCE_CTX_ERR",
"reconnect",
"E2E"
],
"related_bugs": [
"bug-608"
],
"occurrences": 1,
"last_seen": "2026-06-17T04:45:00Z"
}
]
}

102
.wolf/memory.md
View file

@ -1908,3 +1908,105 @@ Implemented sub-spec 3/4 of ha-consistency-invariants autonomously (per-task imp
| 19:27 | Created src/lib/testsuite/test_libxce_retry.c | — | ~3468 |
| 02:28 | task 1.2: built libxce-failover test harness (cursor/taxonomy/threads) + 7 GREEN self-tests | test_libxce_retry.c, failover_harness.h | Ok:1 Fail:0 | ~9k |
| 19:32 | Edited .kiro/specs/libxce-failover/tasks.md | inline fix | ~17 |
| 19:34 | Created src/include/xce/client_failover.h | — | ~559 |
| 19:34 | Edited src/lib/testsuite/test_libxce_retry.c | 2→4 lines | ~29 |
| 19:34 | Edited src/lib/testsuite/test_libxce_retry.c | modified fo_expected_to_action() | ~1398 |
| 19:35 | Created src/lib/client_failover.c | — | ~560 |
| 19:35 | Edited src/lib/meson.build | 2→3 lines | ~24 |
| 19:39 | Edited .kiro/specs/libxce-failover/tasks.md | inline fix | ~11 |
| 19:46 | Edited src/lib/testsuite/test_libxce_retry.c | 6→9 lines | ~60 |
| 19:46 | Edited src/lib/testsuite/test_libxce_retry.c | modified fo_make_client_masters() | ~1809 |
| 19:46 | Edited src/lib/testsuite/test_libxce_retry.c | 3→7 lines | ~87 |
| 19:47 | Edited src/include/xce/client_failover.h | modified XCE_CLIENT_FAILOVER_H() | ~53 |
| 19:47 | Edited src/include/xce/client_failover.h | modified xce_failover_next_target() | ~439 |
| 19:47 | Edited src/lib/client_failover.c | expanded (+10 lines) | ~249 |
| 19:48 | Edited src/lib/client_failover.c | 3→4 lines | ~23 |
| 19:48 | Edited src/lib/client_failover.c | 4→3 lines | ~17 |
| 19:48 | Edited src/lib/client_failover.c | added 7 condition(s) | ~869 |
| 19:53 | Edited src/lib/client_internal.h | 3→4 lines | ~16 |
| 19:54 | Edited src/lib/client_internal.h | modified struct() | ~310 |
| 19:54 | Edited src/include/xce/client_failover.h | 7→3 lines | ~26 |
| 19:54 | Edited src/include/xce/client_failover.h | reduced (-25 lines) | ~90 |
| 19:55 | Edited src/lib/client_failover.c | 3→4 lines | ~64 |
| 19:56 | Edited .kiro/specs/libxce-failover/tasks.md | inline fix | ~16 |
| 20:02 | Edited src/lib/testsuite/test_libxce_retry.c | added 1 condition(s) | ~1824 |
| 20:03 | Edited src/include/xcectx.h | expanded (+15 lines) | ~242 |
| 20:03 | Edited src/lib/xcectx.c | added 2 condition(s) | ~337 |
| 20:11 | Edited .kiro/specs/libxce-failover/tasks.md | inline fix | ~23 |
| 20:15 | Edited src/lib/client_internal.h | 8→10 lines | ~58 |
| 20:16 | Edited src/lib/client_internal.h | modified struct() | ~802 |
| 20:17 | Edited src/lib/testsuite/test_libxce_retry.c | added 4 condition(s) | ~2707 |
| 20:17 | Edited src/lib/testsuite/test_libxce_retry.c | expanded (+7 lines) | ~129 |
| 20:17 | Edited src/lib/client_failover.c | 11→14 lines | ~122 |
| 20:18 | Edited src/lib/client_failover.c | added 15 condition(s) | ~1384 |
| 20:18 | Edited src/lib/client_failover.c | modified if() | ~259 |
| 03:22 | impl libxce-failover 3.1 decision loop (xcecli_retry_decide pure + xcecli_retry_next) | src/lib/client_failover.c, client_internal.h, testsuite/test_libxce_retry.c | GREEN lib Ok:1 Fail:0, 7 new tests pass | ~38k |
| 20:27 | Edited .kiro/specs/libxce-failover/tasks.md | inline fix | ~20 |
| 20:30 | Edited src/lib/testsuite/test_libxce_retry.c | modified test_recover_resets_err_state_to_run() | ~1105 |
| 20:30 | Edited src/lib/testsuite/test_libxce_retry.c | 2→7 lines | ~90 |
| 20:31 | Edited src/include/mylock.h | 3→4 lines | ~34 |
| 20:31 | Edited src/lib/client_internal.h | modified CI2() | ~340 |
| 20:31 | Edited src/lib/client_failover.c | added 2 condition(s) | ~415 |
| 20:31 | Edited src/lib/client_failover.c | modified if() | ~130 |
| 03:33 | task 3.2 reconnect recovery: xcecli_ctx_recover (CI2 ERR->RUN state reset + CI1 wbuf drain) wired into xcecli_retry_next | src/lib/client_failover.c, client_internal.h, mylock.h, testsuite/test_libxce_retry.c | GREEN Ok:1 Fail:0 (5 new tests) | ~9k |
| 20:36 | Edited .kiro/specs/libxce-failover/tasks.md | inline fix | ~22 |
| 20:38 | Edited src/lib/testsuite/test_libxce_retry.c | modified struct() | ~1616 |
| 20:38 | Edited src/lib/testsuite/test_libxce_retry.c | 3→8 lines | ~102 |
| 20:39 | Edited src/include/client.h | modified void() | ~191 |
| 20:39 | Edited src/include/client.h | 1→3 lines | ~54 |
| 20:39 | Edited src/lib/client_internal.h | 10→7 lines | ~64 |
| 20:39 | Edited src/lib/client_internal.h | expanded (+18 lines) | ~258 |
| 20:40 | Edited src/lib/client.c | added 1 condition(s) | ~194 |
| 20:40 | Edited src/lib/client_failover.c | added 2 condition(s) | ~223 |
| 20:40 | Edited src/lib/client_failover.c | added 1 condition(s) | ~678 |
| 03:42 | task 4.1 ReconnectRejoinHook: public typedef+setter in client.h, xcecli_set_reconnect_handler in client.c, xcecli_invoke_rejoin in client_failover.c wired into retry_next (recover→hook→resume; failed rejoin re-rotates) | client.h, client.c, client_failover.c, client_internal.h, test_libxce_retry.c | lib build Ok:1 Fail:0 | ~9k |
| 20:45 | Edited .kiro/specs/libxce-failover/tasks.md | inline fix | ~14 |
| 20:50 | Edited src/lib/testsuite/test_libxce_retry.c | added 1 condition(s) | ~1428 |
| 20:50 | Edited src/lib/testsuite/test_libxce_retry.c | 2→4 lines | ~46 |
| 20:50 | Edited src/lib/testsuite/test_libxce_retry.c | 2→3 lines | ~32 |
| 20:50 | Edited src/lib/client_internal.h | modified serialised() | ~607 |
| 20:51 | Edited src/lib/client.c | modified if() | ~46 |
| 20:51 | Edited src/lib/client.c | 2→3 lines | ~33 |
| 20:51 | Edited src/lib/client.c | 2→3 lines | ~34 |
| 20:51 | Edited src/lib/client.c | 2→3 lines | ~35 |
| 20:51 | Edited src/lib/client.c | 2→3 lines | ~34 |
| 20:51 | Edited src/lib/client.c | 2→3 lines | ~36 |
| 20:51 | Edited src/lib/client.c | 2→3 lines | ~33 |
| 20:51 | Edited src/lib/client.c | 2→3 lines | ~34 |
| 20:51 | Edited src/lib/client.c | 4→5 lines | ~40 |
| 20:51 | Edited src/lib/client.c | 5→7 lines | ~53 |
| 20:51 | Edited src/lib/client.c | 2→3 lines | ~36 |
| 20:51 | Edited src/lib/client.c | 4→5 lines | ~37 |
| 20:51 | Edited src/lib/client.c | 4→5 lines | ~38 |
| 20:51 | Edited src/lib/client.c | 3→4 lines | ~37 |
| 20:51 | Edited src/lib/client.c | 4→5 lines | ~36 |
| 20:51 | Edited src/lib/client.c | 3→4 lines | ~39 |
| 20:51 | Edited src/lib/client.c | 3→4 lines | ~38 |
| 20:51 | Edited src/lib/client.c | 3→4 lines | ~38 |
| 03:53 | task 5.1: wrap reply-bearing xcecli_* ops in XCECLI_FAILOVER_CALL failover loop under per-ctx lock | client.c, client_internal.h, test_libxce_retry.c | lib build Ok:1 Fail:0; +2 concurrency tests PASS | ~28k |
| 20:57 | Edited .kiro/specs/libxce-failover/tasks.md | inline fix | ~24 |
| 20:57 | Edited .kiro/specs/libxce-failover/design.md | inline fix | ~44 |
| 20:57 | Edited .kiro/specs/libxce-failover/design.md | inline fix | ~78 |
| 21:01 | Edited src/lib/testsuite/test_libxce_retry.c | modified struct() | ~1265 |
| 21:01 | Edited src/lib/testsuite/test_libxce_retry.c | 3→8 lines | ~98 |
| 21:01 | Edited src/lib/client_internal.h | added 1 condition(s) | ~897 |
| 21:02 | Edited src/lib/client_failover.c | added 1 condition(s) | ~230 |
| 21:02 | Edited src/lib/client.c | modified if() | ~83 |
| 21:02 | Edited src/lib/client.c | modified if() | ~90 |
| 21:02 | Edited src/lib/client.c | modified if() | ~116 |
| 21:04 | Edited src/lib/client.c | 5→6 lines | ~103 |
| 21:08 | Edited .kiro/specs/libxce-failover/tasks.md | inline fix | ~16 |
| 21:11 | Created src/controller/testsuite/test_join_redirect.c | — | ~937 |
| 21:11 | Edited src/controller/meson.build | 2→3 lines | ~29 |
| 21:12 | Edited src/controller/testsuite/controller_test.c | 2→3 lines | ~30 |
| 21:12 | Edited src/controller/testsuite/controller_test.c | 2→3 lines | ~24 |
| 21:12 | Edited src/controller/join.c | modified validate_host() | ~203 |
| 21:12 | Edited src/controller/join.c | 7→8 lines | ~75 |
| 04:13 | task 5.3: populate redirect addr on E_XCE_REDIRECT (fill_redirect_reply) + new test | join.c, test_join_redirect.c, meson.build, controller_test.c | controller tests PASS | ~8k |
| 21:16 | Edited .kiro/specs/libxce-failover/tasks.md | inline fix | ~20 |
| 21:19 | Edited src/lib/testsuite/test_libxce_retry.c | added 4 condition(s) | ~1344 |
| 21:19 | Edited src/lib/testsuite/test_libxce_retry.c | 2→3 lines | ~46 |
| 04:21 | task 6.1: gap-fill jitter-staggering test for many clients (Req 6.3) | src/lib/testsuite/test_libxce_retry.c | GREEN Ok:1 Fail:0, 46 tests, new test 48 distinct delays/64 clients | ~12k |
| 21:25 | Edited .kiro/specs/libxce-failover/tasks.md | inline fix | ~13 |
| 21:33 | Edited src/lib/client_failover.c | added 3 condition(s) | ~352 |
| 21:38 | Edited src/lib/testsuite/test_libxce_retry.c | modified test_decide_known_dead_ctx_retries_not_terminal() | ~466 |