Rongsong Shen
6529dcf08f
Replace the deliberate election-carrier leak with an atomic refcount: a bias reference held by ha->elect_ctx plus one reference per captured vote endpoint, released via a new endpoint_t.on_fini teardown-completion hook (fired once, before ep->data is freed). Both real-election and PreVote capture sites (raft_cb_send_vote_request, raft_cb_send_pre_vote_request) increment on capture; both publish sites (ha_start_election_async, ha_start_prevote_async) now atomic_exchange the outgoing carrier and release it instead of leaking or freeing outright, so a still-outstanding vote endpoint keeps the superseded carrier alive until its own teardown completes. Verified: 758+ controller tests green, isolated LeakSanitizer run shows zero ha_elect_ctx_t leaks across many election/PreVote rounds, no premature-free/UAF under supersession, and a live 10-round HA soak (36 jobs, zero loss, all 3 nodes served as primary).
79 lines
6.6 KiB
Markdown
79 lines
6.6 KiB
Markdown
# Implementation Plan: ha-elect-ctx-lifecycle
|
|
|
|
- [x] 1. Foundation: endpoint teardown-completion hook
|
|
- [x] 1.1 Add a generic completion hook to the endpoint abstraction, fired exactly once when an endpoint's teardown reaches its final free
|
|
- Add an `on_fini` function pointer field to `endpoint_t` (src/include/netio.h), mirroring the existing `on_connected` fire-once hook
|
|
- In `endpoint_fini` (src/lib/netio.c), call the hook (if set) immediately before the existing `FREEUP(ep->data)` line — no other change to the function's sequence
|
|
- New unit test: set `on_fini` on a test endpoint, drive it through teardown, assert the hook fires exactly once and that `ep->data` is still valid (non-freed) at the moment the hook runs
|
|
- Observable: the new unit test passes; every other existing endpoint (which never sets `on_fini`) tears down with zero behavioral change (hook is a no-op NULL check)
|
|
- _Requirements: 2.2_
|
|
|
|
- [x] 2. Core: election carrier reference-counted lifecycle
|
|
- [x] 2.1 Add a reference count to the election carrier and a single release helper
|
|
- Add an `_Atomic int refcount` field to `ha_elect_ctx_t` (src/controller/ha.h)
|
|
- Add a static helper in ha.c that decrements the refcount and, on reaching zero, destroys the mutex/semaphore and frees the carrier
|
|
- New unit test: call the helper once on a freshly allocated carrier (refcount=1) and assert it is freed; call it once on a carrier with refcount=2 and assert it is still valid with refcount=1
|
|
- Observable: the helper is the only place `ha_elect_ctx_t` is destroyed/freed in ha.c
|
|
- _Requirements: 1.1_
|
|
- _Boundary: ElectionCarrierLifecycle (ha.c/ha.h)_
|
|
|
|
- [x] 2.2 (P) Increment the carrier refcount and wire the completion hook at the real-election vote-request capture site
|
|
- In `raft_cb_send_vote_request`, at the point `ep->data = vctx` is committed (only for vctx instances that reach a live endpoint), atomically increment the carrier's refcount and set `ep->on_fini` to a new hook function that releases the carrier via the 2.1 helper
|
|
- The two pre-`endpoint_get()` alloc-fail bailouts in this function do not increment (they never attach `ep->data`) and need no matching release
|
|
- Observable: after a multi-node election round runs to completion and every vote endpoint's teardown finishes, the carrier's refcount reflects only the still-outstanding supersession bias reference (no endpoint-attributable count remains)
|
|
- _Requirements: 1.1, 2.1, 2.3, 2.4_
|
|
- _Boundary: raft_cb_send_vote_request_
|
|
- _Depends: 1.1, 2.1_
|
|
|
|
- [x] 2.3 (P) Increment the carrier refcount and wire the completion hook at the PreVote vote-request capture site
|
|
- Same mechanism as 2.2, applied to `raft_cb_send_pre_vote_request`
|
|
- Observable: identical post-round refcount behavior as 2.2, but for PreVote rounds
|
|
- _Requirements: 1.1, 2.1, 2.3, 2.4, 3.1_
|
|
- _Boundary: raft_cb_send_pre_vote_request_
|
|
- _Depends: 1.1, 2.1_
|
|
|
|
- [x] 2.4 Release the superseded carrier when a new election or PreVote round publishes its own
|
|
- In `ha_start_election_async` and `ha_start_prevote_async`, change the carrier-publish assignment from `atomic_store_explicit` to `atomic_exchange_explicit` to retrieve the outgoing carrier pointer, then release it via the 2.1 helper if non-NULL
|
|
- Route the two existing no-vote-endpoint fast-exit paths in `ha_start_election_async` (no-local-config bail-out, single-node case) through the same 2.1 helper instead of a direct `free()`, for one consistent destruction path
|
|
- Update the "Do NOT free a prior ctx" comment at the PreVote publish site to describe the new mechanism instead of documenting the leak as deliberate
|
|
- Observable: starting a second round while the first round's carrier already has zero endpoint references frees the first round's carrier immediately at supersession; starting a second round while the first still has an outstanding vote endpoint leaves the first carrier alive until that endpoint's teardown completes
|
|
- _Requirements: 1.1, 1.2, 3.1, 3.2_
|
|
- _Boundary: ElectionCarrierLifecycle (ha.c publish sites)_
|
|
- _Depends: 2.1, 2.2, 2.3_
|
|
|
|
- [x] 3. Validation: leak closure and premature-free safety
|
|
- [x] 3.1 (P) Prove zero carrier leaks across many consecutive real-election rounds under leak-detection-enabled ASAN
|
|
- New unit test drives N consecutive multi-node election rounds with ASAN leak detection enabled (distinct from the suite's current `detect_leaks=0` default)
|
|
- Temporarily revert the 2.2/2.4 changes to confirm this test fails (leak reported) pre-fix, then restore and confirm it passes (RED→GREEN)
|
|
- Observable: zero `ha_elect_ctx_t` bytes reported leaked across the run
|
|
- _Requirements: 1.3_
|
|
- _Boundary: election test coverage_
|
|
- _Depends: 2.2, 2.4_
|
|
|
|
- [x] 3.2 (P) Prove zero carrier leaks across many consecutive PreVote rounds under leak-detection-enabled ASAN
|
|
- Same shape as 3.1, driving PreVote rounds via `ha_start_prevote_async` instead
|
|
- Observable: zero `ha_elect_ctx_t` bytes reported leaked across the run
|
|
- _Requirements: 1.3, 3.1, 3.2_
|
|
- _Boundary: PreVote test coverage_
|
|
- _Depends: 2.3, 2.4_
|
|
|
|
- [x] 3.3 (P) Prove the carrier is never freed while an outstanding vote endpoint still references it across a round supersession
|
|
- Construct a round with one captured, not-yet-torn-down vote endpoint, then start a new round (triggering supersession) before that endpoint's teardown completes
|
|
- Assert the original carrier is still valid (not freed) immediately after supersession, and becomes freed only after the outstanding endpoint's `on_fini` fires
|
|
- Covers both the real-election and PreVote capture paths (parametrized or duplicated per design's Testing Strategy)
|
|
- Observable: a naive "free on supersession" implementation would fail this test (UAF or premature free caught by ASAN); the refcounted implementation passes it
|
|
- _Requirements: 2.1, 2.2, 2.3, 3.1, 3.2_
|
|
- _Boundary: no-premature-free test coverage_
|
|
- _Depends: 2.2, 2.3, 2.4_
|
|
|
|
- [x] 3.4 Run the full existing controller unit suite and confirm no behavioral change
|
|
- Run the complete `controller_unit_test` suite; assertions in pre-existing election, PreVote, and failover test files must be byte-for-byte unchanged
|
|
- Observable: full suite green, zero modified assertions in pre-existing election/prevote/failover tests
|
|
- _Requirements: 4.1, 4.2_
|
|
- _Depends: 3.1, 3.2, 3.3_
|
|
|
|
- [x] 3.5 Run a full HA sequential and chaotic regression soak and confirm no new job loss, divergence, or liveness stall
|
|
- Run `/xce-test-ha` sequential and chaotic regression against a live 3-node cluster
|
|
- Observable: zero committed job loss, zero divergence, no liveness stall attributable to this change; bug-339/bug-340 vote-freshness behavior unaffected
|
|
- _Requirements: 4.3_
|
|
- _Depends: 3.4_
|