forked from lei.chen/myxce
fix(libxce): bug-1077 ctrlstatus crash from shared reply-union type confusion
ctx->reply.u is a single untagged union shared across opcodes (debugmsg /
get(dataset_t*) / jobid / clusterconf / ctrlstatus alias the same 8 bytes).
Commit 61e1391 added speculative free-guards
`if (ctx->reply.u.ctrlstatus) free_CtrlStatus_ptr(...)` (and the clusterconf
twin). A non-NULL slot does NOT imply a CtrlStatus*: a failover-aborted
reslist/joblist leaves ctx->reply.u.get == &getreply (a dead STACK pointer)
because xce_get_with_condition/xce_resval only cleared the slot on the success
path, not the xce_get_req<0 / cbuf_get<0 early returns. The next ctrlstatus
reply (worker thread) freed that foreign pointer as a CtrlStatus -> ASan
SEGV / stack-use-after-return in free_CtrlStatus (controller.capnp.c:11638).
Fix (root cause, both layers):
- Revert the unsafe free-guards in xce_handle_reply: MOVE_PTR overwrite only,
never free/deref an untagged union slot by type.
- Clear ctx->reply.u.get = NULL on ALL exit paths of xce_get_with_condition
and xce_resval via a single `goto done` cleanup (forget-proof; the bug was
one forgotten failover path). Also drops the now-unused controller.capnp.h.
- Defense-in-depth: handle_get_reply drops a reply when the slot is NULL.
Regression: expose xce_handle_reply and add
test_handle_reply_ctrlstatus_does_not_free_foreign_slot, which poisons the
shared slot with a foreign heap value and proves the ctrlstatus reply handler
does not free it (RED reproduced the exact free_CtrlStatus_ptr double-free;
GREEN after fix). xcelib_test 244/0, controller_unit_test 166/0, all binaries
build clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSxwb2Xt5F84ATgiAZnH8V
This commit is contained in:
parent
61e1391cd0
commit
8da8fd03dd
7 changed files with 943 additions and 767 deletions
1555
.wolf/buglog.json
1555
.wolf/buglog.json
File diff suppressed because it is too large
Load diff
|
|
@ -297,3 +297,11 @@ Pure library core (no I/O/threads) + controller callback adapters keeps it simul
|
|||
- WHY ctrl-3 never led: HA election timeout is PRIORITY-STAGGERED. my_priority = index in --peers list (ctrl-1=0, ctrl-2=1, ctrl-3=2; lower = higher priority), and the election delay = my_priority * election_priority_step_ms (ha.h:138, default 500ms → p0 immediate / p1 +500 / p2 +1000). The soak only ever kills ONE node (single-failover), so the surviving pair always contains a lower-index node that out-races ctrl-3. Confirmed: ctrl-3 had 0 CTRL_PRIMARY announcements but DID start 6 elections (it tries, always loses the race). No leadership-transfer/stepdown command exists in xce_test or ha.c, so leadership can't be handed over directly.
|
||||
- HOW to force a specific (lowest-priority) node T to lead, DETERMINISTICALLY and loss-free (run-ha-soak.sh force_elect_node): exploit the RequestVote freshness gate (a behind node cannot win). P=current primary, S=third node. (1) kill S so it stops receiving entries; (2) submit a batch → commits on {P,T}, S now provably behind; (3) kill P → only T alive briefly (T holds ALL commits — they were on {P,T} — so nothing is lost); (4) restart the STILL-BEHIND S → quorum {T fresh, S stale}; S cannot catch up while there is no primary, so it stays freshness-fenced for the whole election → T wins deterministically (NOT by timing); (5) verify no loss, restart P, reconverge. KEY INSIGHT: a behind node can't catch up without a primary, and there's no primary during the election, so the spoiler is guaranteed fenced — this removes all timing-race flakiness.
|
||||
- The soak now tracks SEEN_PRIMARY per node and the verdict ASSERTS all 3 led (PASS line: "all 3 nodes led (ctrl-1,ctrl-2,ctrl-3)"); a node that never leads is a FAIL. force_elect_node is generic (works for any never-led node), run post-loop for each missing node after restoring 3-of-3.
|
||||
|
||||
## Key Learning (2026-06-24, bug-1077) — ctx->reply.u is a SHARED, UNTAGGED union
|
||||
`xce_ctx_t.reply.u` (src/include/xcectx.h) overlays debugmsg/get(dataset_t*)/jobid/clusterconf/ctrlstatus in the SAME 8 bytes. A non-NULL slot does NOT tell you its type. NEVER free or dereference it by type in `xce_handle_reply` (xcectx.c) — `MOVE_PTR` overwrite only (it never touches the old value). The reply slot is owned by the *consumer* (xceapi.c), which MUST clear `u.get = NULL` on EVERY exit path (the get/resval consumers leaked the caller's dead stack `&getreply` on the failover error paths → a later ctrlstatus reply freed it as a CtrlStatus* → stack-use-after-return).
|
||||
|
||||
## Do-Not-Repeat (2026-06-24, bug-1077)
|
||||
- Do NOT add "free the prior occupant" guards on an untagged union slot (this is what commit 61e1391 did incidentally, bundled into an unrelated "fix bug-1060 bug-1067" — it caused a CRITICAL ASan crash). If you want leak-safety on such a slot you need a type tag, not a blind free.
|
||||
- When a request consumer stashes a STACK pointer into a shared ctx field (e.g. `ctx->reply.u.get = &local`), it MUST clear it on ALL exit paths, not just success. Use a single `goto done` cleanup so a new error path can't reintroduce the leak.
|
||||
- Keep speculative "while I'm here" hardening OUT of bugfix commits — it hid a memory-safety regression behind an unrelated commit title.
|
||||
|
|
|
|||
|
|
@ -2797,3 +2797,29 @@
|
|||
|
||||
| Time | Action | File(s) | Outcome | ~Tokens |
|
||||
|------|--------|---------|---------|--------|
|
||||
| 02:18 | Edited src/lib/xcectx.c | modified xce_handle_reply() | ~77 |
|
||||
| 02:18 | Edited src/lib/xcectx.c | inline fix | ~18 |
|
||||
| 02:19 | Edited src/include/xcectx.h | expanded (+6 lines) | ~111 |
|
||||
| 02:19 | Edited src/lib/testsuite/test_ctrlstatus.c | modified test_handle_reply_ctrlstatus_does_not_free_foreign_slot() | ~727 |
|
||||
| 02:19 | Edited src/lib/testsuite/test_ctrlstatus.c | 4→5 lines | ~50 |
|
||||
| 02:22 | Edited src/lib/xcectx.c | modified if() | ~333 |
|
||||
| 02:23 | Edited src/lib/xcectx.c | added 1 condition(s) | ~149 |
|
||||
| 02:23 | Edited src/lib/xceapi.c | modified if() | ~214 |
|
||||
| 02:23 | Edited src/lib/xceapi.c | modified if() | ~150 |
|
||||
|
||||
## Session 2026-06-24 (bug-1077 — ctrlstatus ASan crash) — summary
|
||||
| HH:MM | description | file(s) | outcome | ~tokens |
|
||||
| 17:30 | Root-caused ASan SEGV/use-after-return in free_CtrlStatus via handle_reply on `ctrlstatus` after failover | src/lib/xcectx.c, xceapi.c | union type-confusion: ctx->reply.u shared/untagged; free-guards added in 61e1391 freed a foreign (dead-stack) value | ~4k |
|
||||
| 17:35 | Logged bug-1077 to buglog | .wolf/buglog.json | appended | ~1k |
|
||||
| 17:45 | TDD: exposed xce_handle_reply, wrote deterministic regression test (foreign slot must not be freed) | src/lib/testsuite/test_ctrlstatus.c, include/xcectx.h | RED reproduced (ASan double-free, same free_CtrlStatus_ptr path) | ~3k |
|
||||
| 17:55 | Fix: reverted unsafe free-guards (clusterconf+ctrlstatus) + clear ctx->reply.u.get on ALL exits of xce_get_with_condition/xce_resval + NULL-guard handle_get_reply | src/lib/xcectx.c, xceapi.c | GREEN: xcelib_test 244/0, controller_unit_test 166/0; all binaries build clean | ~3k |
|
||||
| 02:27 | Created ../../../../../claude/.claude/projects/-home-rshen-projects-xce-xce/memory/project_bug1077_reply_union_type_confusion.md | — | ~474 |
|
||||
| 02:27 | Edited ../../../../../claude/.claude/projects/-home-rshen-projects-xce-xce/memory/project_bug1077_reply_union_type_confusion.md | inline fix | ~17 |
|
||||
| 02:28 | Session end: 11 writes across 5 files (xcectx.c, xcectx.h, test_ctrlstatus.c, xceapi.c, project_bug1077_reply_union_type_confusion.md) | 7 reads | ~13701 tok |
|
||||
| 17:32 | Edited src/lib/xcectx.c | 3→4 lines | ~32 |
|
||||
| 17:32 | Edited src/lib/xcectx.c | dereferenced() → free_CtrlStatus_ptr() | ~89 |
|
||||
| 17:36 | VERIFIED bug-1077 (libxce reply-union type confusion): confirmed 3-layer fix in tree (revert free-guards + NULL slot on every consumer early-return + handle_get_reply NULL-check); RED→GREEN proven by re-injecting free-guard → ASan double-free at test_ctrlstatus.c:384, restored → GREEN; full libxce suite 291/0, 0 ASan errors | src/lib/xcectx.c, xceapi.c, testsuite/test_ctrlstatus.c | VERIFIED PASS | ~60 |
|
||||
| 17:36 | Session end: 13 writes across 5 files (xcectx.c, xcectx.h, test_ctrlstatus.c, xceapi.c, project_bug1077_reply_union_type_confusion.md) | 7 reads | ~13942 tok |
|
||||
| 02:40 | Edited src/controller/config_repl.c | expanded (+16 lines) | ~346 |
|
||||
| 18:10 | Added NOTICE at config-upgrade restart site: upgrade must snapshot current state (entity-bearing) at config-marker (term,lsn) boundary + install on all members, not per-node WAL re-derive (bug-857) | src/controller/config_repl.c (step 9, do_upgrade_config_with_bytes) | comment-only; refs config-upgrade-snapshot spec + entity-bearing-snapshots hard dep | ~1k |
|
||||
| 02:41 | Session end: 14 writes across 6 files (xcectx.c, xcectx.h, test_ctrlstatus.c, xceapi.c, project_bug1077_reply_union_type_confusion.md) | 8 reads | ~39719 tok |
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue