From 223e1b857821ac3c88118deb5cf51365cad50bb1 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Wed, 17 Jun 2026 06:54:28 +0000 Subject: [PATCH] docs(libxce-failover): record live-E2E round-2 fixes (bug-609 EINVAL, bug-610 reconnect SEGV) --- .kiro/specs/libxce-failover/tasks.md | 1 + .wolf/memory.md | 10 ++++++++++ 2 files changed, 11 insertions(+) diff --git a/.kiro/specs/libxce-failover/tasks.md b/.kiro/specs/libxce-failover/tasks.md index 7961fb1..d712207 100644 --- a/.kiro/specs/libxce-failover/tasks.md +++ b/.kiro/specs/libxce-failover/tasks.md @@ -112,3 +112,4 @@ - 5.3: `fill_redirect_reply(xce_reply_t*, const ha_state_t*)` (non-static, join.c) sets redirect_host=ha.primary_host + redirect_port=(uint16_t)ha.primary_port; called ONLY on the E_XCE_REDIRECT branch before xce_reply. Audit: join.c is the SOLE E_XCE_REDIRECT emitter. NOMASTER path unchanged (8.2). host alias (no copy) safe: ha lives cluster-lifetime, xce_reply serializes synchronously. New test_join_redirect.c registered in controller_test.c + meson.build. - 6.1: full lib suite green (161 tests). All 14 reqs (2.4,2.5,3.2,3.3,4.1-4.5,5.2,6.2,6.3,8.3,8.4) map to genuine passing tests. Gap-filled 6.3 with `test_decide_jitter_staggers_many_clients` (64 distinct-seed carriers → staggered delays, anti-thundering-herd). No production bugs found. - 6.2 (LIVE E2E — partial): Live 3-node cluster (deploy-ha.sh) was brought up and DID elect a primary; killing the primary triggered server-side failover (ctrl-2 self-elected term=2). The run REPRODUCED the original `Invalid argument` client bug → root-caused + FIXED (bug-609, known-dead-ctx override) + regression unit test added. The clean within-session client transparent-recovery could NOT be visually demonstrated in this sandbox (background controller processes are reaped between tool calls; nix runtime env needed for backgrounded procs; lib build cleans the controller binary). RECOMMEND final visual confirmation in a stable env: build all, `/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993`, run xce_test (set masters / client / jobsub / joblist), kill the primary, confirm joblist reports cluster-not-ready then transparently recovers (no EINVAL). +- 6.2 (LIVE, round 2 — your env): failover now ENGAGES (xcecli_joblist→xcecli_retry_next→xce_failover_rotate). Two live-only bugs found+fixed: bug-609 (fresh-call-after-disconnect -EINVAL classified terminal → known-dead-ctx override in decide) and bug-610 (xce_failover_rotate SEGV in endpoint_connect — reconnect never re-bound ep->iomgr nor re-armed owner/msg/err handlers + iomgr_monitor; fixed via new `xce_ctx_reconnect()` in xcectx.c that mirrors xcecli_start establishment without restarting iomgr/vtaskmgr or touching ctx->state). REBUILD controller before retest: `cd src && nix develop --command bash tools/xce-build.sh controller` (relinks xce_test). NOTE: building `lib` removes the controller binary from the shared build dir — always rebuild controller before deploy. diff --git a/.wolf/memory.md b/.wolf/memory.md index 4b61211..ef84797 100644 --- a/.wolf/memory.md +++ b/.wolf/memory.md @@ -2010,3 +2010,13 @@ Implemented sub-spec 3/4 of ha-consistency-invariants autonomously (per-task imp | 21:25 | Edited .kiro/specs/libxce-failover/tasks.md | inline fix | ~13 | | 21:33 | Edited src/lib/client_failover.c | added 3 condition(s) | ~352 | | 21:38 | Edited src/lib/testsuite/test_libxce_retry.c | modified test_decide_known_dead_ctx_retries_not_terminal() | ~466 | +| 21:48 | Session end: 198 writes across 23 files (brief.md, tasks.md, spec.json, requirements.md, roadmap.md) | 47 reads | ~141658 tok | + +## Session: 2026-06-17 23:51 + +| Time | Action | File(s) | Outcome | ~Tokens | +|------|--------|---------|---------|--------| +| 23:51 | Edited src/lib/xcectx.c | added 4 condition(s) | ~615 | +| 23:51 | Edited src/include/xcectx.h | 1→2 lines | ~31 | +| 23:52 | Edited src/lib/client_failover.c | reduced (-7 lines) | ~251 | +| 23:53 | Edited .kiro/steering/roadmap.md | 1→2 lines | ~375 |