myxce/.kiro/specs/ha-commit-engine/research.md

# Research Log: ha-commit-engine

Date: 2026-06-11. Discovery status: **complete via umbrella** — this sub-spec inherits the ha-consistency-invariants discovery (`.kiro/specs/ha-consistency-invariants/research.md`: gap analysis, Option C selection, deep-dives §A–§C, locked decisions). No decisions are re-derived here. This log records only the slice-scoped code verification performed for this spec's design.

## Inherited Locked Decisions (umbrella — do not re-litigate)
- Commit engine as single authority; WAIT/RETAIN completion modes (research synthesis)
- Admission cap relative to `n_vcpus` (never absolute); `COMMIT_WAIT_RESERVED` default 2 (research item 3(d–e))
- Blocking condvar on vtask threads accepted; async continuation deferred; no liveness risk (ACKs on iomgr) (research item 3(b))
- Catch-up standbys excluded from quorum accounting and commit counting until `repl_complete_catchup` (umbrella §Quorum Accounting + commit_on_membership_change contract)
- Commit state volatile (no persisted commitIndex); convergence/rejoin owned by ha-convergence-recovery (research §C)
- Own-term gate via WAL_NOOP at promotion — API (`commit_set_writable_lsn`) ships in this spec, consumers in ha-single-leader-role (research item 4)

## Slice Verification (2026-06-11, this spec)

| Fact | Evidence | Implication |
|------|----------|-------------|
| `standby_conn_t.acked_lsn` atomic exists, updated by `repl_on_ack` | src/include/repl.h:24; repl.c:176, store at repl.c:187 | matchIndex bookkeeping ready; engine consumes read-only |
| Dishonest counting: `acked_lsn > 0` | repl.c:419 | replaced by per-LSN engine evaluation |
| `n_active == 0 → success` short-circuit | repl.c:432-435 | removed for n_peers>1 via engine (engine knows n_peers; repl does not) |
| Quorum denominator is `n_active`, not `n_peers` | repl.c:437 | engine uses configured `n_peers` per umbrella commit rule |
| `ha_has_write_quorum` counts `standby_endpoints->count` (includes catch-up) | ha.c:1507-1526; gate call at ha.c:1537 | fix via new `repl_active_standby_count` (ep && !in_catch_up); `in_catch_up` at repl.h:26 |
| Sync barrier: `persist_wal_sync_append` | persist.c:1003-1049; committed_lsn store at 1035; `has_standbys → -ENODEV` degraded-success at ~1027-1031; truncation store at persist.c:1438 | adapter passes `written_lsn`; 1035 store moves into engine; -ENODEV case replaced by engine decision; 1438 store stays |
| ACKs processed on iomgr | `peer_ctrl_msg_handler` → `repl_ack_decode_and_dispatch` (ha.c:2632) → `repl_on_ack` call at ha.c:2748 | `commit_on_ack` feed point; must never block |
| Membership mutation points | repl.h:50-51 (`repl_add_standby`/`repl_remove_standby`), repl.h:76/79 (`repl_start_catchup`/`repl_complete_catchup`); removal call sites ha.c:3176, 3866 | `commit_on_membership_change` call sites |
| vtask pool default 8, `--vcpus` override | cluster.c:1076-1083 | `n_vcpus` input to `commit_engine_init`; init point after vtaskmgr sizing |
| `replicator_t` embedded in `persist_t` (`p->repl`); cluster reaches via `c->persist.repl` | persist.c:1032; ha.c:538, 2748 | engine reaches standby registry via its `persist_t*`; `commit_engine_init(ce, p, ...)` signature suffices |
| `ha_state_t` has `n_peers` and `raft_c`, but no repl pointer | ha.h:180-185 | add explicit `replicator_t *repl` back-pointer (raft_c only set by election path — not reliable for the gate) |
| repl.h/persist.h live in src/include (lib-visible); controller-local headers in src/controller/include | find results; cluster.h, worker.h | `commit.h` goes to src/controller/include; repl.h uses forward-declared `struct commit_engine *` to avoid dependency inversion |
| `repl_wait_majority_ack` callers: persist.c:1032 (walAck), config_repl.c:916 (configAck) | grep | signature gains `entry_lsn`; configAck branch untouched (out of boundary) |
| Existing tests encode old semantics | testsuite/test_repl_majority.c (walAck cases at :85-:177); meson controller_unit_test_src list at meson.build:74-110 | walAck cases must be updated; new test file registered in meson |
| `XCE_ERR_NOT_READY` defined | ha.c:35 (`= E_XCE_NOT_READY`) | reuse for admission-cap rejection |

## Design Synthesis Outcomes (this slice)
- **Adapter placement** (brief-mandated): `repl_wait_majority_ack` stays as the shell; gains `entry_lsn` param because the replicator has no LSN knowledge; engine-less fallback retained ONLY for unit-test contexts and explicitly retired by ha-commit-barrier-rollout. Production walAck always goes through the engine.
- **Engine reachability**: forward-declared `struct commit_engine *ce` on `replicator_t` (keeps src/include/repl.h free of controller headers); engine instance embedded in `cluster_t`; init after vtask sizing + peer count known.
- **Standalone behavior change (intentional)**: with the engine wired, `n_peers <= 1` sync appends now advance `committed_lsn` (previously the -ENODEV path left it unmoved). Raft-correct (self-quorum), satisfies spec req 6.3; flagged in design.
- **Membership-shrink semantics**: re-evaluation = wake + re-check; no forced early-fail (umbrella 4.3 requires fail-not-confirm, satisfied by timeout; reconnection within the deadline may still legitimately commit).
- Review gate: design verified against the umbrella contract — interface verbatim, no contradiction with locked decisions, all 20 slice requirement IDs traced.

---

## Gap Re-Audit (2026-06-18)

**Type:** Post-implementation gap audit (not a pre-impl gap analysis). Roadmap declares `ha-commit-engine` IMPLEMENTED 12/12 (2026-06-12, validate-impl GO, 467 tests green). `tasks.md` shows 6 `[ ]` checkboxes; all 6 are **section/group headers** ("1.", "2.", "3.", "4.", "5.", "6."), not leaf tasks — every leaf sub-task (1.1, 1.2, 2.1, 2.2, 3.1, 3.2, 4.1, 4.2, 5.1, 5.2, 5.3, 6.1) is `[x]`. The "12/12" count is the leaf count. The unchecked group headers are an artifact of the checklist format, documented as such in §Implementation Notes — they are **not** deferred work and **not** core gaps.

### Verdict

**STATUS: CLOSED.** All 7 requirements (20 EARS acceptance criteria) are implemented and wired in production. Verified against current on-disk source (codegraph verbatim reads + grep, 2026-06-18). The engine has since been extended by `ha-commit-durability` (durable self-cap) — that extension strengthens, and does not regress, the ha-commit-engine contract.

### Requirement-to-Asset Map (verified, file:line)

| Req | Criterion | Asset (verified) | Status |
|-----|-----------|------------------|--------|
| 1.1 | confirm only after per-entry majority | `commit_submit(WAIT)` predicate `wait_done_locked` commit.c:184-189; sync path persist.c:1897-1900 → `repl_wait_majority_ack` walAck delegation repl.c:447 → `commit_submit` | MET |
| 1.2 | count ACK toward N only if acked_lsn >= N | `majority_candidate_locked` reads `s->acked_lsn`, sorts desc, candidate = `scratch[n_peers/2-1]` commit.c:107-134 | MET |
| 1.3 | connectivity never substitutes for ACKs | ACK-time eval only; watermarks read from registry at `commit_on_ack` commit.c:366-381; gate (ha_has_write_quorum) is fast-reject only, engine re-verifies | MET |
| 2.1 | no commit advance without majority | `commit_advance_locked` stores only `majority_candidate_locked` result commit.c:145-168 | MET |
| 2.2 | monotonic committed_lsn | `if (cand <= cur) return;` + `atomic_store(max)` commit.c:150-165; XCE_TESTING monotonic assert | MET |
| 2.3 | contiguous, bounded by appended log | candidate clamped to `cap` (durable, ≤ `highest_appended_lsn`) commit.c:136-137, 52-55; cumulative watermarks → contiguous | MET (strengthened: cap is now durable_lsn per ha-commit-durability) |
| 3.1/3.2/3.3 | admission quorum counts only members able to receive; catch-up + disconnected excluded | `repl_active_standby_count` (ep != NULL && !in_catch_up) repl.c:668-686; consumed by `ha_has_write_quorum` ha.c:2049-2058 | MET |
| 4.1 | post-acceptance shrink → fail not confirm | WAIT times out (`-ETIMEDOUT`) when majority never reached; no false-success path; `commit_on_membership_change` re-evals monotonically (shrink never regresses) commit.c:397-409 | MET |
| 4.2 | membership change re-evaluates pending waits | `commit_on_membership_change` wired at 4 mutators: repl.c:130 (add), :162 (remove), :598 (start_catchup), :621 (complete_catchup) | MET |
| 4.3/4.4 | zero peer ACKs never majority (n_peers>1); no zero-active short-circuit | engine derives from `n_peers` not n_active; `k = n_peers/2-1`, absent peers zero-filled commit.c:117-134; legacy `n_active==0→success` short-circuit removed (sync path persist.c:1897 engine branch; configAck has explicit "no n_active shortcut" repl.c:506-510) | MET |
| 5.1 | lag never blocks majority | majority is the only threshold; far-lagging standby sorts low, does not affect `scratch[n_peers/2-1]` commit.c:129-134 | MET |
| 6.1 | reject without quorum | `ha_has_write_quorum` false → `ha_dispatch_gate` -XCE_ERR_NOT_READY ha.c:2031-2059 | MET |
| 6.2 | auto-resume on quorum restore | gate is stateless per request (recomputes `repl_active_standby_count` each call); no reset needed | MET |
| 6.3 | standalone self-quorum | `n_peers <= 1` → `commit_advance_locked` immediate, returns cap commit.c:100-102, 330-332; sync path -ENODEV special case removed on engine branch | MET |
| 7.1/7.2 | n_vcpus-relative admission cap, NOT_READY beyond | `wait_cap = max(1, n_vcpus - COMMIT_WAIT_RESERVED)` commit.c:175-180; `inflight_waits >= cap → -XCE_ERR_NOT_READY` commit.c:313-320 | MET |
| 7.3 | ACK processing unaffected at cap | `commit_on_ack` takes only ce->lock, never the inflight gate; advances + broadcasts regardless of cap commit.c:366-381 | MET |

### The 6 Open `[ ]` Tasks — Classified

All 6 are group-header checkboxes whose every child is `[x]`. None is deferred work; none is a core gap. They are a **checklist-rendering artifact** (parent headers left unchecked while leaves complete).

| Header | Children | Classification | Effort/Risk |
|--------|----------|----------------|-------------|
| 1. Commit engine core | 1.1 [x], 1.2 [x] | DONE (artifact) | none |
| 2. WAIT/self-quorum/zero-peer/cap/gate | 2.1 [x], 2.2 [x] | DONE (artifact) | none |
| 3. Membership re-evaluation | 3.1 [x], 3.2 [x] | DONE (artifact; jointly completed — RED unachievable, §Notes) | none |
| 4. Quorum-accounting fix | 4.1 [x], 4.2 [x] | DONE (artifact) | none |
| 5. Integration (wire engine behind job submit) | 5.1 [x], 5.2 [x], 5.3 [x] | DONE (artifact) | none |
| 6. Validation / dishonest-path audit | 6.1 [x] | DONE (artifact; 6.1 audit record in §Notes) | none |

**Bottom line: 0 deferred follow-ups remain as core gaps. The "deferred follow-ups in §Implementation Notes" referenced by the roadmap are advisory/out-of-scope items, not open tasks** — see the genuine residuals below; all are explicitly out-of-spec-boundary and owned elsewhere.

### Genuine Residuals (advisory, all out-of-boundary or already resolved downstream)

These are the §Implementation Notes "deferred follow-ups". None blocks CLOSED status; each is owned by a later spec or is cosmetic.

1. **LSN-0 first-entry legacy edge** (Notes 5.1, 6.1 audit). The first-ever WAL entry (LSN 0) routes to the legacy branch (persist.c:1903-1929) because `commit_submit` rejects `lsn==0`. — **RESOLVED downstream:** ha-single-leader-role's promotion no-op entry makes LSN 0 never a client write in practice; repl.c:438-440 now confirms LSN 0 immediately. Once-per-cluster-lifetime, no consistency loss. *Effort: 0 (closed). Risk: none.*
2. **`ce->n_peers` not re-derived on dynamic membership change** (Notes 6.1(b)). If `ha->n_peers` is reassigned post-init (e.g. ha.c RDHUP re-election), the engine's `n_peers` is stale. — **Out of boundary** (dynamic membership = future work); current cluster topology is fixed at init. *Option A: assert n_peers stability at re-election callback; Option B: re-derive `ce->n_peers` in a `commit_set_n_peers` setter called from the election path; Option C: leave as documented latent gap. Effort: A ~0.5d, B ~1d (+revalidate barrier-rollout/durability). Risk: LOW today (no dynamic membership in production); MEDIUM if dynamic reconfiguration is ever added without addressing it.*
3. **Pre-existing ASAN leaks in 6 test groups** (Notes 6.1 LEAK FOLLOW-UP: worker_join 33KB, rdhup_integration, failover, start_election_async, liveness, connection_reuse). — **Out of scope** (pre-existing, production-side allocations; suite runs with `detect_leaks=0`). *Option: dedicated leak-cleanup pass. Effort: ~1-2d. Risk: LOW (test-harness hygiene, not a correctness gap).*
4. **Stale comment persist.c:1151-1154 / pre-engine contract text** (Notes 6.1). One-line doc follow-up. *Effort: trivial. Risk: none.*
5. **Untracked `src/lib/testsuite/test_wal_wire.c`** referenced by committed meson.build (Notes 5.1). Repo hygiene. *Effort: trivial (git add). Risk: none.*

### Drift Note (informational)

The engine source has been extended by **ha-commit-durability** (implemented 2026-06-18) AFTER ha-commit-engine: `majority_candidate_locked` now caps the self-contribution at `persist_durable_lsn` (commit.c:93) rather than the appended tip, and a `commit_on_durable_advance` flusher wake was added (commit.c:218). This is additive — it tightens Req 2.3 (committed never beyond durable, not merely never beyond appended) and closes bug-661. The ha-commit-engine contract is unchanged and remains fully satisfied. Original research-table anchors (e.g. repl.c:432, persist.c:1035, ha.c:1507) have drifted; verified-current anchors are in the Requirement-to-Asset Map above.