myxce/.kiro/specs/ha-elect-ctx-lifecycle/design.md

Design Document: ha-elect-ctx-lifecycle

Overview

Purpose: Give the HA election carrier (ha_elect_ctx_t) a real, safe lifetime instead of the current deliberate per-round leak, without touching election/PreVote/failover semantics.

Users: Controller maintainers (leak is invisible to operators today — ASAN's detect_leaks=0 default hides it — but it accumulates one carrier per failover/election/PreVote round for the life of the process).

Impact: Adds a reference count to ha_elect_ctx_t and one new generic completion hook to endpoint_t (on_fini, mirroring the existing on_connected fire-once hook). No change to when elections start, how votes are tallied, or how endpoints are torn down — this design only observes the moment a vote endpoint's teardown actually completes and releases the carrier reference that endpoint held.

Goals

• Carrier is freed exactly once, exactly when no vote endpoint can still reference it (Req 1).
• Freeing can never race a live endpoint's read of the carrier (Req 2).
• One mechanism, applied identically to the real-election path (raft_cb_send_vote_request / ha_start_election_async) and the PreVote path (raft_cb_send_pre_vote_request / ha_start_prevote_async) (Req 3).
• Zero observable change to vote tallying, freshness gates, or failover timing (Req 4).

Non-Goals

• Changing endpoint_fini, vote_ep_teardown_async, or any teardown ordering/timing — this design adds an observation point, not a new teardown path.
• Changing the PreVote protocol, ha_elect_ctx_t's field contents, or vote-tally logic.
• A general-purpose object-lifetime framework. The mechanism here is specific to this one carrier; it is not offered as reusable infrastructure for other structs.

Boundary Commitments

This Spec Owns

• The reference count field on ha_elect_ctx_t and its increment/decrement discipline.
• The two capture sites that must increment (raft_cb_send_vote_request, raft_cb_send_pre_vote_request).
• The one release path that must decrement (vote-endpoint fini completion) and the one release path that must decrement on round supersession (ha_start_election_async, ha_start_prevote_async overwriting ha->elect_ctx).
• The new endpoint_t.on_fini hook's existence and its firing point inside endpoint_fini (one field, one conditional call — not a new subsystem).

Out of Boundary

• The PreVote protocol itself (ha-prevote, shipped).
• Any change to election, vote-granting, or failover semantics.
• endpoint_fini's existing teardown mechanics (FREEUP(ep->data), socket/SSL/packet cleanup) — untouched, just preceded by one optional hook call.
• The async endpoint-teardown ordering itself (vote_ep_teardown_async, endpoint_mark_tearing_down, the bug-1197 deferred-vtask path) — this design fires after that ordering completes, whatever path got there.
• The bug-1298 endpoint refcnt mechanism — a different lifetime (the endpoint_t itself) from the one this spec adds (the ha_elect_ctx_t carrier a vote endpoint's ep->data points into). The two refcounts are independent and must not be conflated.

Allowed Dependencies

• endpoint_t (src/include/netio.h, src/lib/netio.c) — may add one field + one call site, may not change existing field semantics.
• ha_elect_ctx_t, vote_ep_ctx_t (src/controller/ha.h, src/controller/ha.c) — internal to ha.c/ha.h, this spec's primary work area.
• Existing atomics discipline on ha->elect_ctx (atomic_store_explicit / atomic_load_explicit, memory_order_release / memory_order_acquire) — extended (one site moves to atomic_exchange_explicit to retrieve the superseded pointer), not replaced.

Revalidation Triggers

• Any new site that assigns ep->data = <vote_ep_ctx_t*> and wires up a vote endpoint outside the two existing capture sites (would need the same increment + hook wiring).
• Any change to endpoint_fini's call sequence that could reorder FREEUP(ep->data) relative to the new on_fini hook.
• Any future feature that gives endpoint_t.on_fini a second consumer (must confirm the two consumers cannot conflict on the same endpoint).
• Any change to the single-node / no-local-config fast-exit paths in ha_start_election_async that currently free(ctx) directly without ever creating a vote endpoint (they must keep doing so — see Architecture).

Architecture

Existing Architecture Analysis

Two sibling capture functions allocate a fresh ha_elect_ctx_t each round and publish it via atomic_store_explicit(&ha->elect_ctx, ..., memory_order_release):

• ha_start_election_async (real election) has two fast-exit paths that already free the ctx correctly: no-local-config bail-out and the single-node case (n_peers == 1, raft_start_election resolves synchronously via on_become_leader). Neither path ever creates a vote endpoint, so the ctx is never shared — a direct free(ctx) is correct today and stays correct; this design does not touch them.
• The multi-node path is where the leak lives: raft_start_election fires on_send_vote_request per peer, each call routing through raft_cb_send_vote_request, which callocs a vote_ep_ctx_t (vctx), sets vctx->ctx = ectx, attaches it as ep->data, and hands the endpoint to iomgr. ha_start_prevote_async → raft_cb_send_pre_vote_request is the structurally identical PreVote sibling.
• vctx is freed exactly once, generically, inside endpoint_fini (src/lib/netio.c:1171, FREEUP(ep->data)) — reached via endpoint_unref (bug-1298 refcount hitting 0) on every teardown path: connect failure, iomgr_monitor failure, the flag-OFF loser-path loop (ha.c:~2142), the flag-ON deferred-vtask path (vote_ep_teardown_async → vote_ep_teardown_vtask_entry), and RDHUP/err-handler paths. This is deliberately the single place vctx memory is reclaimed; the design must not duplicate that free, only observe it. (Line numbers as of current source; g_drive_async_peerctrl_dispatch_enabled now defaults ON per raft-drive-async, so the flag-ON deferred-vtask path is the live default today — both paths still funnel through endpoint_fini exactly once, so the mechanism is unaffected.)
• The carrier itself is currently never freed for the multi-node case: the next round's atomic_store_explicit(&ha->elect_ctx, mc, ...) (both the election and PreVote sites) silently discards the previous pointer, with a comment explicitly documenting the leak as deliberate (UAF avoidance).

Mechanism

Reference-count the carrier. One bias reference represents "this is (or was, pending drain) the published ha->elect_ctx value"; one reference per live vote_ep_ctx_t that captured it.

ha_elect_ctx_t.refcount (new, _Atomic int, starts at 1)

+1  at capture: raft_cb_send_vote_request / raft_cb_send_pre_vote_request,
    at the point ep->data = vctx is committed (i.e. only for vctx instances
    that actually reach a live endpoint — the pre-endpoint_get() alloc-fail
    bailouts never increment, so they need no matching decrement)

-1  at vote-endpoint completion: new endpoint_t.on_fini(ep) hook, fired by
    endpoint_fini() immediately before FREEUP(ep->data), for every teardown
    path (connect-fail, iomgr_monitor-fail, loser-path unref, deferred
    vtask teardown, RDHUP/err-handler) — all of them funnel through
    endpoint_fini exactly once (bug-1298 invariant), so this is the single
    correct observation point regardless of which path got there

-1  at supersession: ha_start_election_async / ha_start_prevote_async, at
    the atomic_store_explicit(&ha->elect_ctx, new_ctx, ...) site — changed
    to atomic_exchange_explicit to retrieve the outgoing pointer, then
    release it (decrement; free at 0) instead of discarding it

free when refcount reaches 0 (pthread_mutex_destroy + sem_destroy + free),
regardless of which of the two release kinds above is the last one to fire

Why a bias reference is required (not pure per-endpoint counting): between atomic_store_explicit(&ha->elect_ctx, ctx, ...) and the first raft_cb_send_vote_request callback actually running, the carrier has zero vote-endpoint references yet but is already live and readable (e.g. ha.c:1638-1639, 2008-2009 inside raft_cb_become_leader read ha->elect_ctx without assuming any endpoint exists — e.g. single-vote-outstanding or mid-fan-out windows). A pure "count of attached endpoints" would allow a spurious free at refcount 0 before the round is superseded. The bias reference removes that window: the carrier is only reclaimed once it is both superseded and drained of endpoint references, in whichever order those two events occur.

Why a generic endpoint_t.on_fini hook (not per-call-site decrements at every endpoint_unref/teardown call): a vctx is only actually freed inside endpoint_fini, but endpoint_unref (which may or may not trigger endpoint_fini synchronously, depending on whether a pool vtask still holds a ref) is called from roughly a dozen sites in ha.c across connect-failure, iomgr-failure, the loser-path loop, and the bug-1197 deferred-teardown-vtask path. Decrementing at each call site would require knowing, at every one of those dozen sites, whether this call is the one that actually drops the count to zero — which duplicates endpoint_fini's own refcount logic and is exactly the kind of coupling Req 2.3/2.4 (uniform coverage of every free path, "including error and abort paths") is trying to make robust against drift. A single hook fired at the one true free point is correct by construction for every current and future teardown path, mirroring the existing on_connected fire-once hook already on endpoint_t.

Architecture Pattern & Boundary Map

sequenceDiagram
    participant ES as ElectionStart (election or PreVote)
    participant CB as SendVoteRequest callback
    participant EP as VoteEndpoint
    participant FI as endpoint_fini

    ES->>ES: allocate carrier, refcount = 1
    ES->>ES: exchange published carrier for new one, get old carrier
    ES->>ES: release old carrier (decrement, free if zero)
    ES->>CB: start election, fan out per peer
    CB->>CB: record carrier on vctx, increment refcount
    CB->>EP: attach vctx as endpoint data, set on_fini hook
    Note over EP: connect, vote RPC, reply, or error -<br/>any teardown path funnels here
    EP->>FI: endpoint_fini runs once refcount reaches zero
    FI->>FI: call on_fini hook if set (decrement carrier refcount, free if zero)
    FI->>FI: free endpoint data (existing behavior, unchanged)

Selected pattern: reference counting with a bias reference, observed via a generic completion hook — same shape as the existing bug-1298 endpoint_t.refcnt mechanism, applied one level up (the carrier a vote endpoint points into, not the endpoint itself). Domain boundaries: carrier lifetime lives entirely in ha.c/ha.h (internal type, per research.md); the only cross-boundary touch is the one generic hook field added to endpoint_t. Existing patterns preserved: endpoint_fini remains the single vctx free site; atomic_store_explicit(..., memory_order_release) / atomic_load_explicit(..., memory_order_acquire) discipline on ha->elect_ctx is preserved (the supersession site becomes atomic_exchange_explicit, same memory order semantics, just also yielding the old value instead of discarding it). New components rationale: on_fini hook — needed because no existing hook fires at endpoint-teardown completion; refcount field — needed because the leak is a multi-referrer lifetime problem, not a single-owner one. Steering compliance: no thread-role change — the hook fires on whatever thread already calls endpoint_fini today (iomgr thread or pool worker, per bug-1197's deferred-teardown-vtask path); the decrement/free is O(1) non-blocking (atomic decrement + conditional free()), so it introduces no new blocking work on any thread class.

File Structure Plan

Modified Files

• src/include/netio.h — add void (*on_fini)(struct endpoint_t *ep); field to endpoint_t, adjacent to the existing on_connected field, with a doc comment cross-referencing it as the completion-side counterpart.
• src/lib/netio.c — in endpoint_fini, add if (ep->on_fini) { ep->on_fini(ep); } immediately before the existing FREEUP(ep->data) line (as-built: the hook is invoked unconditionally when set, not gated on ep->data; vote_ep_on_fini itself null-checks ep->data). No other change to the function.
• src/controller/ha.h — add _Atomic int refcount; to ha_elect_ctx_t (next to the existing done_posted guard field, same struct region).
• src/controller/ha.c:
  • New static helper ha_elect_ctx_release(ha_elect_ctx_t *ctx) — decrement; on reaching 0, pthread_mutex_destroy + sem_destroy + free. Single choke point for both release kinds (endpoint completion, supersession).
  • New static hook function vote_ep_on_fini(endpoint_t *ep) — reads vctx = ep->data, calls ha_elect_ctx_release(vctx->ctx). Wired as ep->on_fini at both capture sites, same place endpoint_set_err_handler is already called.
  • raft_cb_send_vote_request: initialize ectx->refcount is NOT done here (ctx already exists, allocated earlier) — this site only increments (atomic_fetch_add_explicit(&ectx->refcount, 1, memory_order_relaxed)) at the point ep->data = vctx is committed, and sets ep->on_fini.
  • raft_cb_send_pre_vote_request: identical increment + hook wiring at its ep->data = vctx site.
  • ha_start_election_async: at carrier allocation, initialize ctx->refcount = 1. At the multi-node publish site, change atomic_store_explicit(&ha->elect_ctx, ctx, memory_order_release) to atomic_exchange_explicit(&ha->elect_ctx, ctx, memory_order_acq_rel) capturing the return as old_ctx; if old_ctx non-NULL, call ha_elect_ctx_release(old_ctx). The two existing fast-exit free(ctx) paths (no-local-config, single-node) are unchanged — they never share the ctx, so a direct free at refcount 1→destroy is still correct (or, for consistency and to avoid a second free idiom, route them through ha_elect_ctx_release(ctx) too — same result, one less code path to reason about).
  • ha_start_prevote_async: identical refcount = 1 init at allocation; identical atomic_exchange_explicit + ha_elect_ctx_release(old_ctx) at its publish site (the former "Do NOT free a prior ctx" comment site, now ha.c:~8042), comment updated to describe the new mechanism instead of documenting the leak as deliberate.

No New Files

This is a localized extension to two existing files (ha.c/ha.h) plus a minimal, generic addition to the shared endpoint abstraction (netio.h/netio.c). No new translation units.

System Flows

See the sequence diagram in Architecture above — it is the complete flow for both the real-election and PreVote paths (structurally identical, differing only in which capture function runs).

State transitions for ha_elect_ctx_t.refcount (per instance, monotonically decreasing after peak):

stateDiagram-v2
    [*] --> Live: calloc, refcount=1 (bias)
    Live --> Live: vote-ep capture (+1) — 0..n_peers-1 times
    Live --> Live: vote-ep endpoint_fini completes (-1) — any order, any thread
    Live --> Superseded: next round's atomic_exchange (-1, bias released)
    Live --> Freed: refcount reaches 0 while still current (all endpoints drained before next round starts)
    Superseded --> Freed: last outstanding vote-ep drains (-1 reaches 0)
    Freed --> [*]

Requirements Traceability

Requirement	Summary	Components	Interfaces	Flows
1.1	Reclaim once all referencing vote-eps freed	ha_elect_ctx_release, refcount field	—	refcount state diagram
1.2	No bounded wait/timeout — driven by actual release	vote_ep_on_fini hook fired from endpoint_fini	endpoint_t.on_fini	sequence diagram
1.3	Zero ASAN leak over many rounds under leak detection	full mechanism (all sites below)	—	validation: --detect_leaks=1 soak, new test
2.1	Never free while a vctx holds a reference	refcount discipline (+1 capture / -1 completion)	—	sequence diagram
2.2	Carrier still valid at endpoint_fini read time	hook fires from inside endpoint_fini, before FREEUP(ep->data) — vctx itself (thus vctx->ctx) still valid at hook-call time	endpoint_t.on_fini firing order	sequence diagram
2.3	Same mechanism at both capture sites	raft_cb_send_vote_request, raft_cb_send_pre_vote_request both increment + wire hook	—	Modified Files
2.4	Same mechanism on every vctx free path, incl. error/abort	single hook fires from the one true vctx free site (endpoint_fini) regardless of which teardown path reached it	endpoint_t.on_fini	Architecture §Mechanism rationale
3.1	Same lifetime mechanism for election and PreVote	identical refcount/ha_elect_ctx_release/vote_ep_on_fini reused by both paths	—	Modified Files
3.2	Superseded PreVote round's carrier reclaimable like election's	ha_start_prevote_async's publish site uses the same atomic_exchange_explicit + release pattern	—	state diagram
4.1	No change to vote-tally/quorum/freshness behavior	mechanism touches only allocation/free bookkeeping, never votes[] contents or tally logic	—	—
4.2	No change to election/PreVote/failover timing or outcome	hook/refcount work is O(1) non-blocking, off the liveness-critical send/reply path	—	Architecture §Steering compliance
4.3	Full HA soak shows no new loss/divergence/stall	validation plan (Testing Strategy)	—	—

Testing Strategy

• New unit test — leak closure: run N consecutive multi-node election (and separately, PreVote) rounds in the existing controller test harness under ASAN with leak detection enabled (distinct from the suite's current detect_leaks=0 default, per research.md) and assert zero ha_elect_ctx_t bytes leaked. This is Req 1.3's direct acceptance test.
• New unit test — no premature free: construct a round with an in-flight vote endpoint (captured, not yet torn down) and drive a supersession (start a new round); assert the carrier is still valid (not freed) while the outstanding endpoint's teardown has not yet completed, and becomes freed only after its on_fini fires. Directly tests Req 2.1/2.2.
• New unit test — both paths covered: parametrize (or duplicate) the above two tests for ha_start_prevote_async / raft_cb_send_pre_vote_request, confirming Req 3.1/3.2 — no path left on the old leak-forever behavior.
• Regression — existing election/PreVote/failover suites: run unchanged; must stay green with zero behavioral diff (Req 4.1/4.2). Any assertion change here would itself indicate a semantic regression.
• Live — full HA soak: /xce-test-ha sequential and chaotic regression post-merge (Req 4.3), watching specifically for job loss, divergence, or a liveness stall newly attributable to this change (bug-339/bug-340 vote-freshness behavior must be bit-for-bit unaffected, since the fix touches allocation/free timing, not vote-freshness computation).

Risks

Risk	Mitigation
A future second consumer of endpoint_t.on_fini on the same endpoint could silently override this one	Not applicable today (vote endpoints have exactly one hook consumer); flagged as a Revalidation Trigger above so a future feature adding a second hook consumer on a vote endpoint is forced to notice the conflict.
Off-by-one in the increment/decrement pairing (e.g. incrementing at vctx->ctx = ectx instead of at ep->data = vctx) could reintroduce either a leak or a premature free	Increment is placed at the exact point the reference becomes "live" (attached to an endpoint that will reach endpoint_fini); the two pre-endpoint_get() alloc-fail bailouts never increment, matching that they never attach ep->data. Verified by the "no premature free" unit test above.
Relaxed memory order on the increment (memory_order_relaxed) racing the release of the bias reference	The increment happens on the same thread that is about to hand the endpoint to iomgr for connect — happens-before is already established by the existing ep->data = vctx write and subsequent endpoint_connect/iomgr_monitor calls, which themselves use acquire/release on the endpoint's own refcnt. The carrier's refcount only needs atomicity (concurrent increments from a fan-out of N peers, concurrent decrements from N endpoint_fini calls on different threads), not ordering relative to carrier contents — the carrier's field contents are fully populated before the round's atomic_store/atomic_exchange publish (memory_order_release), which is the actual publish barrier readers rely on.