Rongsong Shen
20e59d0b42
Investigation & fixes (failover committed-job loss): - root-cause bug-851 on an owned 3-node cluster: chaotic failover causes silent committed-prefix divergence (WAL hdr.term always 0; two leaders per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION. - live-validated fixes 848/849/850 (recovery-on-promotion, advertise real applied_lsn, persist-layer contiguity guard); add waldump WAL inspector. Specs (strict raft.tla conformance — library owns the log): - TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6. - raft-log-replication, raft-commit-safety, raft-sim-conformance: extend src/lib/raft to own log/commit/verification via callbacks (specs generated, cross-spec review passed). - ha-term-safety: controller-side adapter + config marker (R1-R7); design done. - config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857). - roadmap + buglog (851/854/857) + .wolf knowledge updated.
39 lines
4.3 KiB
Markdown
39 lines
4.3 KiB
Markdown
# Research / Gap Analysis — ha-elect-ctx-lifecycle
|
||
|
||
## Gap Re-Audit (2026-06-18)
|
||
|
||
> **Prerequisite not met:** this spec has only `brief.md` — **no `requirements.md`** (phase=`initialized`, `ready_for_implementation=false`). A full requirements↔codebase gap map cannot be produced. Run `/kiro-spec-requirements ha-elect-ctx-lifecycle` to formalize EARS requirements, then re-run `/kiro-validate-gap ha-elect-ctx-lifecycle`. The note below is a **pre-requirements gap sketch** derived from `brief.md` + a current code check.
|
||
|
||
### Current State (verified 2026-06-18)
|
||
The deliberate-leak pattern the brief describes is **still present and unfixed** in `src/controller/ha.c`:
|
||
- `ha.c:4229` — "Do NOT free a prior ctx (live vote endpoints may hold a bare pointer to it…)" immediately before `atomic_store(&ha->elect_ctx, mc)`. Each election/PreVote round `calloc`s a fresh `ha_elect_ctx_t` and intentionally leaks the previous one.
|
||
- `ha.c:3978`, `ha.c:4830` — sibling "Do NOT free … may still reference / may already be running" comments on the async endpoint / delaymgr teardown paths (the same UAF caution).
|
||
- No `ha_election_cleanup()` exists (only the aspirational comment the brief cites). Carrier read sites: `ha.c:123-124`, `234-235`, `903-904`.
|
||
- Carrier captured by per-peer vote endpoints (`vote_ep_ctx_t.ctx = ectx`), freed asynchronously by `endpoint_fini` (FREEUP) on the iomgr thread — the root cause of the UAF hazard that makes a naïve free unsafe.
|
||
|
||
### Requirement-to-Asset Map (pre-requirements — tags: MET / Missing / Constraint)
|
||
| Intended capability (from brief) | Current code | Tag |
|
||
|----------------------------------|--------------|-----|
|
||
| Carrier (`ha_elect_ctx_t`) freed when no endpoint references it | leaked by design at `ha.c:4229` | **Missing** |
|
||
| Safe free vs async `endpoint_fini` (`vctx`) teardown | no refcount / no deferred-free ordering | **Missing** |
|
||
| Applies to BOTH real-election and PreVote carrier paths (shared `ha->elect_ctx`) | both paths leak identically | **Missing / Constraint** (fix one without the other leaves the hazard) |
|
||
| No regression of failover correctness (bug-339/bug-340) | current correctness intact; leak is memory-only | **Constraint** |
|
||
|
||
### Severity / Why deferred
|
||
- **Pre-existing + bounded**: one small struct (`sizeof(ha_elect_ctx_t) + n_peers*votes[]`, hundreds of bytes for 3 nodes) per *failover/PreVote event* — a rare event, not steady-state. No unbounded growth under normal operation.
|
||
- Surfaces as ~174 KB / ~1548 allocs under ASAN across the full controller suite's many simulated rounds; the test command uses `detect_leaks=0`, so it does not fail CI today.
|
||
|
||
### Implementation Approach Options (for the eventual design)
|
||
- **Option A — Reference-count the carrier.** Increment per `vctx` that captures it, decrement on each `vctx` free, free at zero. Clean, localized; needs an atomic refcount on `ha_elect_ctx_t` and disciplined inc/dec at every capture/free site. Effort **M**, Risk **Medium** (must cover every `vctx` lifecycle exit, incl. error paths).
|
||
- **Option B — Teardown-ordered deferred free.** Tie the free to confirmed vote-endpoint release (e.g. a generation/epoch the iomgr teardown signals back). Avoids per-capture bookkeeping but needs a reliable "all endpoints of round N released" signal. Effort **M–L**, Risk **Medium**.
|
||
- **Option C — Hybrid.** Refcount as the primitive, plus an epoch guard so a superseded round's carrier is reclaimed promptly once its endpoints drain. Effort **L**, Risk **Medium**.
|
||
|
||
### Research-Needed (carry to design)
|
||
- Enumerate **all** `vctx->ctx = ectx` capture sites and **all** `vctx` free sites (incl. `endpoint_fini` FREEUP, error/abort paths, `raft_cb_send_vote_request`, `ha_send_pre_vote_request`).
|
||
- Confirm the memory-order discipline for the refcount vs the existing `atomic_store(&ha->elect_ctx, …, release)` / acquire reads.
|
||
- Validation must run ASAN with **leak detection ENABLED** (success = zero carrier leaks across many rounds) — distinct from the current `detect_leaks=0` suite default.
|
||
|
||
### Effort / Risk (overall)
|
||
- Effort **M** (single-area C change in `ha.c`/`ha.h`), Risk **Medium** (async-teardown UAF is the real hazard; must not regress bug-339/bug-340 failover correctness).
|
||
|
||
**Next step:** `/kiro-spec-requirements ha-elect-ctx-lifecycle` → `/kiro-spec-design` → implement under leak-detection-enabled ASAN.
|