update documents

.kiro/specs/ha-commit-engine/design.md

@@ -8,7 +8,7 @@
 
 **Users**: Cluster operators gain the guarantee that a confirmed job submission is durable on a majority of members; downstream sub-specs (ha-commit-barrier-rollout, ha-single-leader-role, ha-convergence-recovery) gain the engine interface they build on.
 
-**Impact**: Replaces the counting logic of `repl_wait_majority_ack` (repl.c:386-461) for WAL ACKs with per-LSN engine evaluation; removes the `n_active == 0 → success` short-circuit for multi-member clusters; moves the `committed_lsn` store out of `persist_wal_sync_append` (persist.c:1035) into the engine; fixes `ha_has_write_quorum` (ha.c:1507) to exclude catch-up standbys. Behavior of every other write path is unchanged in this spec.
+**Impact**: Replaces the counting logic of `repl_wait_majority_ack` (repl.c:589, current) for WAL ACKs with per-LSN engine evaluation; removes the `n_active == 0 → success` short-circuit for multi-member clusters; moves the `committed_lsn` store out of `persist_wal_sync_append` (now persist.c:5639) into the engine; fixes `ha_has_write_quorum` (now ha.c:4495) to exclude catch-up standbys. Behavior of every other write path is unchanged in this spec. (Note: `persist_wal_sync_append` itself was later changed by ha-term-safety task 3.1 to route the primary's append through the raft library — `ha_primary_log_append` → `raft_log_append` — before this majority-commit barrier runs; see that spec's design.md.)
 
 ### Goals
 - Entry N commits only when `1 (self) + count(active standbys with acked_lsn >= N) >= n_peers/2 + 1`
@@ -22,7 +22,7 @@
 - Setting `writable_lsn` anywhere (own-term gate consumers are ha-single-leader-role; unset = no gate)
 - RETAIN-mode production consumers (facts roll out in ha-commit-barrier-rollout; the mode itself ships and is tested here)
 - Config-change barrier (`acked_config_lsn` track untouched), demotion/egress/catch-up changes, COMMITTING removal
-- Async continuation of commit waits (umbrella-documented future work; blocking condvar + admission cap is the accepted mechanism)
+- Async continuation of commit waits (umbrella-documented future work; blocking condvar + admission cap is the accepted mechanism) (Note: superseded by raft-apply-decoupling, which added a per-LSN continuation registry, `commit_resolve`/async registration, and a dedicated timeout-sweep thread to `commit_engine_t` — see that spec's design.md)
 
 ## Boundary Commitments
 
@@ -44,7 +44,7 @@
 - `standby_conn_t.acked_lsn` / `in_catch_up` per-standby tracking in `replicator_t` (src/include/repl.h) — read-only consumption; the engine never manages connections
 - `persist_t.committed_lsn` atomic (src/include/persist.h:71) — the engine is the only production writer after init/truncation
 - iomgr/vtaskmgr threading rules per `ha-async-io.md`: `commit_on_ack` and `commit_on_membership_change` are called on the iomgr thread and must never block on I/O or long waits (bounded mutex hold only); `commit_submit(WAIT)` blocks only on vtask threads
-- Dependency direction (umbrella contract): `include/commit.h` depends only on libc/stdatomic types plus forward-declared `persist_t`; `commit.c` is called BY `persist.c` (append/sync path), `repl.c` (ACK + membership events), and `ha.c`/`cluster.c` (init, future promotion); it calls into none of them
+- Dependency direction (umbrella contract): `include/commit.h` depends only on libc/stdatomic types plus forward-declared `persist_t`; `commit.c` is called BY `persist.c` (append/sync path), `repl.c` (ACK + membership events), and `ha.c`/`cluster.c` (init, future promotion); it calls into none of them (Note: superseded — `commit.h` now `#include`s the full `persist.h` and `vtaskmgr.h` rather than only forward-declaring, and `commit.c` now calls INTO `persist.c` (`persist_durable_lsn`, `persist_set_durable_cb`) to implement the durable-majority self-cap; see ha-commit-durability and raft-drive-async design.md)
 
 ### Revalidation Triggers
 - Any change to `commit.h` signatures or return-code contract → revalidate ha-commit-barrier-rollout, ha-single-leader-role, ha-convergence-recovery
@@ -55,10 +55,10 @@
 ## Architecture
 
 ### Existing Architecture Analysis
-- Single-writer model: only the primary appends WAL entries; per-standby `acked_lsn` (Raft matchIndex equivalent) already exists and is updated by `repl_on_ack` (repl.c:176-196) on the iomgr thread (`peer_ctrl_msg_handler` → `repl_ack_decode_and_dispatch` → `repl_on_ack`, ha.c:2739-2748)
+- Single-writer model: only the primary appends WAL entries; per-standby `acked_lsn` (Raft matchIndex equivalent) already exists and is updated by `repl_on_ack` (now repl.c:250-285) on the iomgr thread (`peer_ctrl_msg_handler` → `repl_ack_decode_and_dispatch` → `repl_on_ack`, now ha.c:7249 → ha.c:6630 → repl.c:250)
 - ACKs are cumulative: a standby ACKs its highest contiguously applied LSN (ha-wal-replication contract), so per-standby `acked_lsn` is a watermark — engine evaluation needs no per-entry bitmap
-- Defects replaced: `repl_wait_majority_ack` counts `acked_lsn > 0` (repl.c:419) — any ACK ever; `n_active == 0` short-circuits to success (repl.c:432); quorum denominator is `n_active` (connected standbys) rather than `n_peers` (configured members); `ha_has_write_quorum` (ha.c:1507-1526) counts `standby_endpoints->count`, which includes catch-up standbys
-- `persist_wal_sync_append` (persist.c:1003-1049) is the only sync (blocking) write path — job submit via `persist_job_hook_fun` (persist.c:1125); it blocks a vtask pool thread (default 8, `--vcpus` override, cluster.c:1082); no liveness risk since wakeups come from iomgr
+- Defects replaced: `repl_wait_majority_ack` counted `acked_lsn > 0` — any ACK ever; `n_active == 0` short-circuited to success; quorum denominator was `n_active` (connected standbys) rather than `n_peers` (configured members); `ha_has_write_quorum` counted `standby_endpoints->count`, which included catch-up standbys. (Note: these are pre-spec line citations that no longer resolve — the dishonest-counting branch was fully deleted, not just bypassed: `repl_wait_majority_ack` now unconditionally delegates the walAck branch to `commit_submit`, per ha-commit-barrier-rollout task 2.2. Current `ha_has_write_quorum` is at ha.c:4495-4523.)
+- `persist_wal_sync_append` (now persist.c:5639) is the only sync (blocking) write path — job submit via `persist_job_hook_fun` (now persist.c:6509); it blocks a vtask pool thread (default 8, `--vcpus` override, sized in `cluster.c` ~1743-1745; `commit_engine_init` is wired at cluster.c:1820, not cluster.c:1082 as originally cited); no liveness risk since wakeups come from iomgr (plus, since ha-commit-durability, from the group-commit flusher's durable-advance callback — see below)
 - `replicator_t` is embedded in `persist_t` (`p->repl`), and `ha_state_t`/`cluster_t` reach it via `c->persist.repl` — the engine can reach the standby registry through its `persist_t*`
 
 ### Architecture Pattern & Boundary Map
@@ -164,6 +164,8 @@ Flow decisions: the admission cap rejects before blocking when `inflight_waits >
 
 Under the engine mutex: collect `acked_lsn` of all standbys with `ep != NULL && !in_catch_up` into a scratch array, sort descending. With self counted, entry N has majority when at least `n_peers/2` active standbys have `acked_lsn >= N`; therefore the new commit candidate is `arr[n_peers/2 - 1]` (0-indexed; 0 candidates → no advance) clamped to the highest appended LSN. Store `committed_lsn = max(committed_lsn, candidate)` (2.2); cumulative ACK watermarks make the result contiguous by construction (2.3). Broadcast the condvar; each waiter re-checks its own predicate. Catch-up standbys are excluded from the count (umbrella-locked, conservative — their ACKs count after `repl_complete_catchup` triggers `commit_on_membership_change` re-evaluation).
 
+(Note: superseded by ha-commit-durability, Req 1.2 (bug-661 fix) — the candidate is no longer clamped only to the highest *appended* LSN; self's contribution is additionally capped at `persist_durable_lsn(ce->p)`, the highest LSN fdatasync'd locally, so an entry that is only page-cache-resident cannot count self toward its own majority. See that spec's design.md for the durable-majority barrier.)
+
 ## Requirements Traceability
 
 | Requirement | Summary | Components | Flows |
@@ -202,10 +204,10 @@ Under the engine mutex: collect `acked_lsn` of all standbys with `ep != NULL &&
 | Requirements | 1.1–1.3, 2.1–2.3, 4.1–4.4, 5.1, 6.3, 7.1–7.2 |
 
 **Responsibilities & Constraints**
-- Owns advancement of `persist->committed_lsn` (the atomic stays in `persist_t`; only the engine stores to it in production after init/truncation paths — the store at persist.c:1035 is deleted; the truncation store at persist.c:1438 remains)
-- Commit rule (Raft §5.3, umbrella-verbatim): entry N is committed when `count(self + active standbys with acked_lsn >= N) > n_peers / 2`; advancement is contiguous and monotonic (2.1–2.3)
-- Own-term gate storage: `writable_lsn` defaults to 0 = no gate (this spec never sets it); when set (by ha-single-leader-role), no waiter completes successfully until `committed_lsn >= writable_lsn`
-- Admission cap: at most `max(1, n_vcpus - COMMIT_WAIT_RESERVED)` (reserved default = 2) concurrent WAIT blockers; excess fails fast with `-XCE_ERR_NOT_READY`. Expressed relative to `n_vcpus` only — never absolute (operator tunes `--vcpus`)
+- Owns advancement of `persist->committed_lsn` (the atomic stays in `persist_t`, now declared persist.h:274; only the engine stores to it in production after init/truncation paths — the store formerly in `persist_wal_sync_append`'s engine-wired branch is deleted; truncation stores remain in `persist_truncate_wal_above`, now persist.c:6764 — note the field comment there records several more legitimate `committed_lsn` writers added since, e.g. `persist_committed_lsn_advance_atomic`, all monotonic-max)
+- Commit rule (Raft §5.3, umbrella-verbatim): entry N is committed when `count(self + active standbys with acked_lsn >= N) > n_peers / 2`; advancement is contiguous and monotonic (2.1–2.3) (Note: superseded in part by ha-commit-durability — self's contribution to the count is additionally capped at the local durable watermark, not just the appended tip; see `majority_candidate_locked` in commit.c)
+- Own-term gate storage: `writable_lsn` defaults to 0 = no gate (this spec never sets it); when set (by ha-single-leader-role / ha-term-safety's `ha_arm_own_term_gate`), no waiter completes successfully until `committed_lsn >= writable_lsn`
+- Admission cap: at most `max(1, n_vcpus - COMMIT_WAIT_RESERVED)` (reserved default = 2) concurrent WAIT blockers; excess fails fast with `-XCE_ERR_NOT_READY`. Expressed relative to `n_vcpus` only — never absolute (operator tunes `--vcpus`) (Note: superseded by raft-drive-async task 1.6 — this local cap is now only the fallback used when no `vtaskmgr_t` pool is wired (`commit_set_pool`); the production path folds admission onto the shared vtaskmgr pool's single LivenessAdmission reservation instead. See that spec's design.md.)
 - `n_peers <= 1`: self-ACK constitutes majority; WAIT returns 0 immediately and advances `committed_lsn` to the entry (6.3 — intentional change: standalone nodes now advance `committed_lsn` on sync appends, Raft-correct). For `n_peers > 1`, zero peer ACKs never commit (4.3, 4.4)
 
 **Dependencies**
@@ -287,7 +289,7 @@ Summary (umbrella migration note: "`repl_wait_majority_ack` becomes a thin adapt
 ## Data Models
 
 ### Domain Model (changes only)
-- **commit_engine_t** (in-memory only; commit state volatile, Raft-conformant — umbrella-locked): `{ pthread_mutex_t lock; pthread_cond_t cond; persist_t *p; int n_peers; int n_vcpus; int inflight_waits; uint64_t writable_lsn; uint64_t scratch[MAX_STANDBYS]; }`
+- **commit_engine_t** (in-memory only; commit state volatile, Raft-conformant — umbrella-locked): `{ pthread_mutex_t lock; pthread_cond_t cond; persist_t *p; int n_peers; int n_vcpus; int inflight_waits; uint64_t writable_lsn; uint64_t scratch[MAX_STANDBYS]; }` (Note: superseded — the struct has since grown a `vtaskmgr_t *pool` field (raft-drive-async), a `uint64_t raft_commit_index` advisory field (ha-term-safety), and `commit_continuation_t *continuations`/`n_continuations`/`sweep_thread`/`sweep_stop` for async continuation resolution (raft-apply-decoupling). See `src/controller/include/commit.h` and those specs' design.md.)
 - **replicator_t**: + `struct commit_engine *ce` (NULL until wired)
 - **ha_state_t**: + `replicator_t *repl` (NULL in legacy test contexts)
 - **cluster_t**: + `commit_engine_t commit`