Rongsong Shen 20e59d0b42 checkpoint: bug-851 root cause + raft TLA+ conformance spec family

Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.

2026-06-21 14:02:15 +00:00

4.3 KiB

Raw Blame History

Research / Gap Analysis — ha-elect-ctx-lifecycle

Gap Re-Audit (2026-06-18)

Prerequisite not met: this spec has only brief.md — no requirements.md (phase=initialized, ready_for_implementation=false). A full requirements↔codebase gap map cannot be produced. Run /kiro-spec-requirements ha-elect-ctx-lifecycle to formalize EARS requirements, then re-run /kiro-validate-gap ha-elect-ctx-lifecycle. The note below is a pre-requirements gap sketch derived from brief.md + a current code check.

Current State (verified 2026-06-18)

The deliberate-leak pattern the brief describes is still present and unfixed in src/controller/ha.c:

ha.c:4229 — "Do NOT free a prior ctx (live vote endpoints may hold a bare pointer to it…)" immediately before atomic_store(&ha->elect_ctx, mc). Each election/PreVote round callocs a fresh ha_elect_ctx_t and intentionally leaks the previous one.
ha.c:3978, ha.c:4830 — sibling "Do NOT free … may still reference / may already be running" comments on the async endpoint / delaymgr teardown paths (the same UAF caution).
No ha_election_cleanup() exists (only the aspirational comment the brief cites). Carrier read sites: ha.c:123-124, 234-235, 903-904.
Carrier captured by per-peer vote endpoints (vote_ep_ctx_t.ctx = ectx), freed asynchronously by endpoint_fini (FREEUP) on the iomgr thread — the root cause of the UAF hazard that makes a naïve free unsafe.

Requirement-to-Asset Map (pre-requirements — tags: MET / Missing / Constraint)

Intended capability (from brief)	Current code	Tag
Carrier (`ha_elect_ctx_t`) freed when no endpoint references it	leaked by design at `ha.c:4229`	Missing
Safe free vs async `endpoint_fini` (`vctx`) teardown	no refcount / no deferred-free ordering	Missing
Applies to BOTH real-election and PreVote carrier paths (shared `ha->elect_ctx`)	both paths leak identically	Missing / Constraint (fix one without the other leaves the hazard)
No regression of failover correctness (bug-339/bug-340)	current correctness intact; leak is memory-only	Constraint

Severity / Why deferred

Pre-existing + bounded: one small struct (sizeof(ha_elect_ctx_t) + n_peers*votes[], hundreds of bytes for 3 nodes) per failover/PreVote event — a rare event, not steady-state. No unbounded growth under normal operation.
Surfaces as ~174 KB / ~1548 allocs under ASAN across the full controller suite's many simulated rounds; the test command uses detect_leaks=0, so it does not fail CI today.

Implementation Approach Options (for the eventual design)

Option A — Reference-count the carrier. Increment per vctx that captures it, decrement on each vctx free, free at zero. Clean, localized; needs an atomic refcount on ha_elect_ctx_t and disciplined inc/dec at every capture/free site. Effort M, Risk Medium (must cover every vctx lifecycle exit, incl. error paths).
Option B — Teardown-ordered deferred free. Tie the free to confirmed vote-endpoint release (e.g. a generation/epoch the iomgr teardown signals back). Avoids per-capture bookkeeping but needs a reliable "all endpoints of round N released" signal. Effort M–L, Risk Medium.
Option C — Hybrid. Refcount as the primitive, plus an epoch guard so a superseded round's carrier is reclaimed promptly once its endpoints drain. Effort L, Risk Medium.

Research-Needed (carry to design)

Enumerate all vctx->ctx = ectx capture sites and all vctx free sites (incl. endpoint_fini FREEUP, error/abort paths, raft_cb_send_vote_request, ha_send_pre_vote_request).
Confirm the memory-order discipline for the refcount vs the existing atomic_store(&ha->elect_ctx, …, release) / acquire reads.
Validation must run ASAN with leak detection ENABLED (success = zero carrier leaks across many rounds) — distinct from the current detect_leaks=0 suite default.

Effort / Risk (overall)

Effort M (single-area C change in ha.c/ha.h), Risk Medium (async-teardown UAF is the real hazard; must not regress bug-339/bug-340 failover correctness).

Next step: /kiro-spec-requirements ha-elect-ctx-lifecycle → /kiro-spec-design → implement under leak-detection-enabled ASAN.

4.3 KiB Raw Blame History Unescape Escape