fix(libxce): bug-1077 ctrlstatus crash from shared reply-union type confusion
ctx->reply.u is a single untagged union shared across opcodes (debugmsg /
get(dataset_t*) / jobid / clusterconf / ctrlstatus alias the same 8 bytes).
Commit 61e1391 added speculative free-guards
`if (ctx->reply.u.ctrlstatus) free_CtrlStatus_ptr(...)` (and the clusterconf
twin). A non-NULL slot does NOT imply a CtrlStatus*: a failover-aborted
reslist/joblist leaves ctx->reply.u.get == &getreply (a dead STACK pointer)
because xce_get_with_condition/xce_resval only cleared the slot on the success
path, not the xce_get_req<0 / cbuf_get<0 early returns. The next ctrlstatus
reply (worker thread) freed that foreign pointer as a CtrlStatus -> ASan
SEGV / stack-use-after-return in free_CtrlStatus (controller.capnp.c:11638).
Fix (root cause, both layers):
- Revert the unsafe free-guards in xce_handle_reply: MOVE_PTR overwrite only,
never free/deref an untagged union slot by type.
- Clear ctx->reply.u.get = NULL on ALL exit paths of xce_get_with_condition
and xce_resval via a single `goto done` cleanup (forget-proof; the bug was
one forgotten failover path). Also drops the now-unused controller.capnp.h.
- Defense-in-depth: handle_get_reply drops a reply when the slot is NULL.
Regression: expose xce_handle_reply and add
test_handle_reply_ctrlstatus_does_not_free_foreign_slot, which poisons the
shared slot with a foreign heap value and proves the ctrlstatus reply handler
does not free it (RED reproduced the exact free_CtrlStatus_ptr double-free;
GREEN after fix). xcelib_test 244/0, controller_unit_test 166/0, all binaries
build clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSxwb2Xt5F84ATgiAZnH8V
This commit is contained in:
parent
61e1391cd0
commit
8da8fd03dd
7 changed files with 943 additions and 767 deletions
|
|
@ -297,3 +297,11 @@ Pure library core (no I/O/threads) + controller callback adapters keeps it simul
|
|||
- WHY ctrl-3 never led: HA election timeout is PRIORITY-STAGGERED. my_priority = index in --peers list (ctrl-1=0, ctrl-2=1, ctrl-3=2; lower = higher priority), and the election delay = my_priority * election_priority_step_ms (ha.h:138, default 500ms → p0 immediate / p1 +500 / p2 +1000). The soak only ever kills ONE node (single-failover), so the surviving pair always contains a lower-index node that out-races ctrl-3. Confirmed: ctrl-3 had 0 CTRL_PRIMARY announcements but DID start 6 elections (it tries, always loses the race). No leadership-transfer/stepdown command exists in xce_test or ha.c, so leadership can't be handed over directly.
|
||||
- HOW to force a specific (lowest-priority) node T to lead, DETERMINISTICALLY and loss-free (run-ha-soak.sh force_elect_node): exploit the RequestVote freshness gate (a behind node cannot win). P=current primary, S=third node. (1) kill S so it stops receiving entries; (2) submit a batch → commits on {P,T}, S now provably behind; (3) kill P → only T alive briefly (T holds ALL commits — they were on {P,T} — so nothing is lost); (4) restart the STILL-BEHIND S → quorum {T fresh, S stale}; S cannot catch up while there is no primary, so it stays freshness-fenced for the whole election → T wins deterministically (NOT by timing); (5) verify no loss, restart P, reconverge. KEY INSIGHT: a behind node can't catch up without a primary, and there's no primary during the election, so the spoiler is guaranteed fenced — this removes all timing-race flakiness.
|
||||
- The soak now tracks SEEN_PRIMARY per node and the verdict ASSERTS all 3 led (PASS line: "all 3 nodes led (ctrl-1,ctrl-2,ctrl-3)"); a node that never leads is a FAIL. force_elect_node is generic (works for any never-led node), run post-loop for each missing node after restoring 3-of-3.
|
||||
|
||||
## Key Learning (2026-06-24, bug-1077) — ctx->reply.u is a SHARED, UNTAGGED union
|
||||
`xce_ctx_t.reply.u` (src/include/xcectx.h) overlays debugmsg/get(dataset_t*)/jobid/clusterconf/ctrlstatus in the SAME 8 bytes. A non-NULL slot does NOT tell you its type. NEVER free or dereference it by type in `xce_handle_reply` (xcectx.c) — `MOVE_PTR` overwrite only (it never touches the old value). The reply slot is owned by the *consumer* (xceapi.c), which MUST clear `u.get = NULL` on EVERY exit path (the get/resval consumers leaked the caller's dead stack `&getreply` on the failover error paths → a later ctrlstatus reply freed it as a CtrlStatus* → stack-use-after-return).
|
||||
|
||||
## Do-Not-Repeat (2026-06-24, bug-1077)
|
||||
- Do NOT add "free the prior occupant" guards on an untagged union slot (this is what commit 61e1391 did incidentally, bundled into an unrelated "fix bug-1060 bug-1067" — it caused a CRITICAL ASan crash). If you want leak-safety on such a slot you need a type tag, not a blind free.
|
||||
- When a request consumer stashes a STACK pointer into a shared ctx field (e.g. `ctx->reply.u.get = &local`), it MUST clear it on ALL exit paths, not just success. Use a single `goto done` cleanup so a new error path can't reintroduce the leak.
|
||||
- Keep speculative "while I'm here" hardening OUT of bugfix commits — it hid a memory-safety regression behind an unrelated commit title.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue