socket_connect sets O_NONBLOCK before connect and bounds the attempt with
poll(timeout); new connect_timeout_ms param (0=DEFAULT_CONNECT_TIMEOUT_MS);
all 5 callers updated (ha.c x3, client.c, xcectx.c). Revalidation Trigger:
shared netio helper signature changed. Broad build links all components;
xcelib_test 278/0, controller_unit_test 194/0. 1.4/1.5 unit tests satisfied
by the TDD output of 1.2/1.3 (test_slowlane.c, test_netio_bounded_connect.c).
ctx->reply.u is a single untagged union shared across opcodes (debugmsg /
get(dataset_t*) / jobid / clusterconf / ctrlstatus alias the same 8 bytes).
Commit 61e1391 added speculative free-guards
`if (ctx->reply.u.ctrlstatus) free_CtrlStatus_ptr(...)` (and the clusterconf
twin). A non-NULL slot does NOT imply a CtrlStatus*: a failover-aborted
reslist/joblist leaves ctx->reply.u.get == &getreply (a dead STACK pointer)
because xce_get_with_condition/xce_resval only cleared the slot on the success
path, not the xce_get_req<0 / cbuf_get<0 early returns. The next ctrlstatus
reply (worker thread) freed that foreign pointer as a CtrlStatus -> ASan
SEGV / stack-use-after-return in free_CtrlStatus (controller.capnp.c:11638).
Fix (root cause, both layers):
- Revert the unsafe free-guards in xce_handle_reply: MOVE_PTR overwrite only,
never free/deref an untagged union slot by type.
- Clear ctx->reply.u.get = NULL on ALL exit paths of xce_get_with_condition
and xce_resval via a single `goto done` cleanup (forget-proof; the bug was
one forgotten failover path). Also drops the now-unused controller.capnp.h.
- Defense-in-depth: handle_get_reply drops a reply when the slot is NULL.
Regression: expose xce_handle_reply and add
test_handle_reply_ctrlstatus_does_not_free_foreign_slot, which poisons the
shared slot with a foreign heap value and proves the ctrlstatus reply handler
does not free it (RED reproduced the exact free_CtrlStatus_ptr double-free;
GREEN after fix). xcelib_test 244/0, controller_unit_test 166/0, all binaries
build clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSxwb2Xt5F84ATgiAZnH8V
xce_failover_rotate reconnected with endpoint_fini+endpoint_init+endpoint_connect
but never re-bound ep->iomgr (reset by endpoint_init) nor re-armed the owner/
msg/err handlers + iomgr_monitor. endpoint_connect then dereferenced a NULL
iomgr (->ssl_ctx) → SEGV at netio.c:672 during live master failover. Added
xce_ctx_reconnect() in xcectx.c that mirrors xcecli_start's establishment
(without restarting iomgr/vtaskmgr or touching ctx->state); rotate delegates to
it. Found via live 3-node E2E. bug-610.
Four bugs fixed:
1. persist.c: array job WAL loop aborted after index [0] when replication
returned -ENODEV (no standbys). Loop guard now uses the return value of
persist_wal_sync_append() which is 0 on local-disk success regardless of
replication status.
2. join.c: WAL-recovered worker slot (status=UNAVAIL) caused add_worker() to
return -EEXIST on reconnect. New EEXIST path in handle_worker_join() looks
up the existing slot by name, updates its endpoint, and clears UNAVAIL so
the normal hook/reply/on_worker_join flow proceeds.
3. xcectx.c: cbuf_get() in xce_hello() blocked indefinitely when the TCP
connection dropped because no error handler was registered. Added
handle_disconnect() which sets ctx->state=XCE_CTX_ERR and calls
cbuf_tryput(-ECONNRESET) to unblock the waiter, allowing
checking_connection() to detect the failure and rejoin.
4. cluster.c: persist_recover() was called before cluster_build_resources()
populated resmgr, causing restore_worker() → add_hostattrs_resvals() to
fail with EEXIST noise. Moved persist_recover() to after HOOK_AFTER_EVENT
so resmgr is fully populated before WAL replay.
