spec(ha-heartbeat-committed-propagation): requirements + design + tasks approved

Live-fresh off-primary committed: primary stamps committed_lsn/config_tag onto each HA heartbeat (OP_nop + the existing-but-unused HeartbeatMsg body); standby monotonically refreshes last_known_committed_* (reconcile-free, term-guarded). Removes the controller-status Req 10 staleness caveat. Vote-floor formula unchanged; safety rests on term-guard + monotonicity (design Security). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-20 11:12:05 +00:00 · 2026-06-20 11:12:05 +00:00 · 9817813f54
commit 9817813f54
parent 54d3293a9e
5 changed files with 405 additions and 0 deletions
--- a/.kiro/specs/ha-heartbeat-committed-propagation/design.md
+++ b/.kiro/specs/ha-heartbeat-committed-propagation/design.md
@ -0,0 +1,204 @@
+# Design Document — ha-heartbeat-committed-propagation
+
+## Overview
+
+**Purpose:** keep a standby controller's knowledge of the cluster's committed position current from the primary's heartbeats, instead of only at a join/failover handshake. The serving primary attaches its committed position (committed LSN + committed configuration version) to every HA heartbeat; each standby consumes it and monotonically advances its `last_known_committed_*` state.
+
+**Users:** operators inspecting a standby via the controller-status `ctrlstatus` query (its off-primary committed becomes near-real-time instead of join-stale or zero), and the HA vote-freshness path (which already reads `last_known_committed_lsn` and gains a fresher, term-guarded value).
+
+**Impact:** the HA heartbeat is a bare `OP_nop` today. This wires up the already-defined-but-unused `HeartbeatMsg{committedLsn, committedConfigTag}` body on that heartbeat (producer + consumer), adds a reconcile-free monotonic updater, and promotes one field to atomic. No new opcode; no change to how committed is produced/replicated/persisted; no change to the vote-floor formula.
+
+### Goals
+- The serving primary advertises its committed LSN + committed config version on every heartbeat (1.1).
+- A standby refreshes its known committed position from heartbeats, monotonically, within one heartbeat interval, with no re-join (1.2, 1.3, 2.x).
+- The propagated value is advisory: it never drives truncation and never weakens the vote-freshness floor (3.x).
+- Existing heartbeat liveness/failover behavior is unchanged (4.x).
+
+### Non-Goals
+- Changing how `committed_lsn` is produced, advanced, replicated, or persisted (commit engine / durability).
+- The join/failover handshake (`cluster_apply_ctrlhello_reply`) that already seeds the standby's committed knowledge — left as-is (it remains the initial source and the only path that reconciles/truncates).
+- Catch-up streaming triggering (stays ACK-driven, per ha-convergence-recovery).
+- The `ctrlstatus` reporting/rendering of committed (owned by controller-status; it already reads `last_known_committed_*`).
+
+## Boundary Commitments
+
+### This Spec Owns
+- The producer: attaching the primary's committed LSN + committed config tag (and its term) to the HA heartbeat request.
+- The consumer: reading that payload on a standby and **monotonically** updating `last_known_committed_lsn` / `last_known_committed_config_tag`, gated by a term check, **without** reconciliation/truncation.
+- The new reconcile-free monotonic updater `ha_observe_heartbeat_committed`.
+- Promoting `last_known_committed_config_tag` to `_Atomic` and migrating its existing writers.
+
+### Out of Boundary
+- The vote-freshness floor **formula** (`max(durable_floor, last_known_committed_lsn)`) — unchanged; this spec only makes its input fresher.
+- The join handshake / RDHUP path and its truncation/reconcile (`cluster_apply_ctrlhello_reply`, `ha_rejoin_reconcile`).
+- Committed production/replication/persistence, WAL truncation, catch-up streaming.
+- The client keepalive `OPCode_heartbeat` / `xce_heartbeat` (a different message — must not be touched or reused).
+- `ctrlstatus` field sourcing/rendering (controller-status).
+
+### Allowed Dependencies
+- The Cap'n Proto schema + codegen: the existing `HeartbeatMsg` struct and the `heartbeatmsg @25` Request-body union member (`src/schema/controller.capnp`) — reused, no shape change.
+- The existing HA heartbeat tick (`primary_heartbeat_fn`, `ha.c`) and the standby `OP_nop` receive path (`cluster.c`).
+- Read access to the primary's `persist.committed_lsn` and `ha->committed_config_tag`; read/write of `ha->last_known_committed_*` and `ha->term`.
+- The request/reply plumbing and a new opcode-less sender alongside `xce_nop_req`.
+
+### Revalidation Triggers
+- Any change to `HeartbeatMsg`'s committed fields or the `heartbeatmsg` body ordinal → re-check producer + consumer.
+- Any change to the voter-floor formula or to who writes `last_known_committed_*` → re-check the term-guard and monotonicity argument (Security section).
+- Any change making the HA heartbeat use a different opcode/body → re-check the consumer's `body_which` branch.
+- Changing `last_known_committed_config_tag`'s type → re-check all writers + the `ctrlstatus` reader.
+
+## Architecture
+
+### Existing Architecture Analysis
+- **HA heartbeat = bare `OP_nop`:** `primary_heartbeat_fn` (`ha.c:5256`) sends `xce_nop_req(ep)` with no body; the standby `OP_nop` handler (`cluster.c:300`) replies a `StateReport` and resets liveness (`ha_on_heartbeat_received`). It reads no request body.
+- **Unused scaffolding:** `HeartbeatMsg{senderPriority@0, term@1, committedLsn@2, committedConfigTag@3}` and the `heartbeatmsg @25` Request-body member exist but have zero `.c` producers/consumers.
+- **Committed knowledge today:** `last_known_committed_lsn` (`_Atomic`, `ha.h:170`) + `last_known_committed_config_tag` (plain, `ha.h:172`) written only at join (`cluster_apply_ctrlhello_reply` — a plain store that ALSO truncates via `ha_rejoin_reconcile`) and an election safe-commit path (`ha.c:925`).
+- **Monotonic precedent:** `commit.c` advances `committed_lsn` with monotonic-max semantics — mirrored here.
+- **Vote floor already consumes it:** `max(durable_floor, last_known_committed_lsn)` (`ha.c:2570`).
+- **Opcode/body independence:** `OP_hello` already carries different bodies (`ctrlhellomsg` vs `replymsg`), so `OP_nop` + `heartbeatmsg` body is consistent with the existing pattern.
+
+### Architecture Pattern & Boundary Map
+Extend-in-place along the existing heartbeat path. **Dependency direction (unchanged):** `schema → libxce (sender) → controller (primary tick + standby receive)`. No upward dependencies; no new opcode.
+
+```mermaid
+graph LR
+    Tick[primary_heartbeat_fn tick] --> Sender[xce_ha_heartbeat_req nop plus HeartbeatMsg body]
+    Sender --> Wire[OP_nop request with heartbeatmsg body]
+    Wire --> Handler[cluster.c OP_nop handler]
+    Handler --> Guard[term guard plus monotonic]
+    Guard --> Observe[ha_observe_heartbeat_committed]
+    Observe --> State[last_known_committed_lsn and config_tag]
+    State --> Floor[vote freshness floor unchanged]
+    State --> Status[ctrlstatus off primary committed]
+```
+
+Key decisions: the primary stamps committed onto the existing heartbeat (no new opcode, Option A); the standby's update is **term-guarded and monotonic** and **reconcile-free**; the vote floor reads the fresher value through the unchanged formula.
+
+## File Structure Plan
+
+### Modified Files
+- `src/include/request.h` + `src/lib/request.c` — add `xce_ha_heartbeat_req(endpoint_t* ep, uint64_t term, uint64_t committed_lsn, uint64_t committed_config_tag)`: sends an `OPCode_nop` request with a `heartbeatmsg` body carrying `{term, committed_lsn, committed_config_tag}`. Generic `xce_nop_req` is left unchanged.
+- `src/controller/ha.c` — `primary_heartbeat_fn`: replace the `xce_nop_req(ep)` heartbeat call with `xce_ha_heartbeat_req(ep, ha->term, persist.committed_lsn, ha->committed_config_tag)` (the `XCE_TESTING` `ha_nop_req_hook` seam is preserved). Add `ha_observe_heartbeat_committed`. Migrate the election-path writer of `last_known_committed_config_tag` (`ha.c:925` area) to `atomic_store`.
+- `src/controller/cluster.c` — `OP_nop` handler: if `request.body_which == Request_body_heartbeatmsg`, apply the term-guard and call `ha_observe_heartbeat_committed`; otherwise behave exactly as today. Liveness reset + `StateReport` reply unchanged.
+- `src/controller/include/ha.h` — change `last_known_committed_config_tag` to `_Atomic uint64_t`; declare `ha_observe_heartbeat_committed`.
+- `src/controller/ha_convergence.c` — `cluster_apply_ctrlhello_reply`: write `last_known_committed_config_tag` via `atomic_store` (type migration only; semantics unchanged).
+- Tests: `src/controller/testsuite/test_ha_heartbeat_committed.c` (new) for producer/consumer + monotonic + term-guard + advisory-only; registered in `controller_test.c` + `controller/meson.build`.
+
+### New Files
+- `src/controller/testsuite/test_ha_heartbeat_committed.c` — unit tests for the consumer/updater. (No new product files — `HeartbeatMsg` C types are generated; logic slots into existing TUs.)
+
+## System Flows
+
+### Heartbeat-borne committed propagation + term-guard
+```mermaid
+sequenceDiagram
+    participant P as Primary
+    participant S as Standby
+    P->>P: read own committed_lsn and committed_config_tag and term
+    P->>S: OP_nop with HeartbeatMsg term committed_lsn config_tag
+    S->>S: reset liveness and reply StateReport as before
+    alt heartbeat term is at least standby term and committed greater than known
+        S->>S: monotonic advance last_known_committed_lsn and config_tag
+    else stale term or not greater or no body
+        S->>S: liveness only no committed update
+    end
+```
+Gating: the standby updates committed only when the heartbeat's advertised term is `>= ha->term` (term-guard) AND the carried LSN exceeds the current `last_known_committed_lsn` (monotonic). Otherwise the heartbeat is processed for liveness exactly as today. The update never reconciles or truncates the log.
+
+## Requirements Traceability
+
+| Requirement | Summary | Components | Interfaces | Flows |
+|-------------|---------|------------|------------|-------|
+| 1.1 | Primary advertises committed on each heartbeat | `xce_ha_heartbeat_req`, `primary_heartbeat_fn` | `HeartbeatMsg{term, committedLsn, committedConfigTag}` body on `OP_nop` | Propagation |
+| 1.2 | Standby consumes it | `cluster.c` `OP_nop` handler, `ha_observe_heartbeat_committed` | `body.heartbeatmsg` read | Propagation |
+| 1.3 | Fresh within one heartbeat interval | per-tick send + receive | heartbeat cadence | Propagation |
+| 2.1 | Advance on higher carried LSN | `ha_observe_heartbeat_committed` | monotonic-max store | Propagation (alt) |
+| 2.2 | Leave unchanged on ≤ (stale/out-of-order/lagging) | `ha_observe_heartbeat_committed` | monotonic guard | Propagation (else) |
+| 2.3 | Config tag advances with LSN, never independently regressed | `ha_observe_heartbeat_committed` | paired update under the same advance | Propagation |
+| 3.1 | No truncation from heartbeat-borne committed | `ha_observe_heartbeat_committed` (reconcile-free) | no `ha_rejoin_reconcile` call | — |
+| 3.2 | No weakening of the vote-freshness floor | vote floor unchanged + term-guard | `max(durable_floor, last_known_committed_lsn)` (unchanged) | Propagation (term-guard) |
+| 3.3 | No change to committed production/replication/persist | consume path | read-only on produced committed | — |
+| 3.4 | Primary uses its own committed | consume gated to non-primary | inbound heartbeats only on standbys | Propagation |
+| 4.1 | Existing liveness behavior unchanged | `OP_nop` handler | `ha_on_heartbeat_received` + `StateReport` untouched | Propagation |
+| 4.2 | Absent/zero committed → plain liveness, no-op | `OP_nop` handler, `ha_observe_heartbeat_committed` | body-absent / term-fail / zero → no update | Propagation (else) |
+
+## Components and Interfaces
+
+| Component | Layer | Intent | Req Coverage | Key Dependencies | Contracts |
+|-----------|-------|--------|--------------|------------------|-----------|
+| `xce_ha_heartbeat_req` | libxce | Send `OP_nop` + `HeartbeatMsg` body | 1.1 | request plumbing, `HeartbeatMsg` codegen (P0) | Service |
+| `primary_heartbeat_fn` call-site | Controller HA | Stamp own committed/term per tick | 1.1, 3.4 | `persist.committed_lsn`, `ha->committed_config_tag`, `ha->term` (P0) | Service |
+| `OP_nop` handler consume branch | Controller | Read body, term-guard, dispatch update | 1.2, 4.1, 4.2 | `ha_observe_heartbeat_committed` (P0) | Service |
+| `ha_observe_heartbeat_committed` | Controller HA | Reconcile-free monotonic update of `last_known_committed_*` | 2.x, 3.1 | `ha->last_known_committed_*` atomics (P0) | Service, State |
+| `last_known_committed_config_tag` → `_Atomic` | Controller HA | Safe concurrent read/write | 2.3 | existing writers (P1) | State |
+
+### libxce — HA heartbeat sender (Service)
+```c
+/* Send the HA heartbeat as OP_nop carrying a HeartbeatMsg body. Clone of the
+ * nop send path + a heartbeatmsg body. Non-blocking, no reply awaited (the
+ * standby's StateReport rides the normal nop reply). Generic xce_nop_req is
+ * left unchanged. */
+int xce_ha_heartbeat_req(endpoint_t* ep, uint64_t term,
+                         uint64_t committed_lsn, uint64_t committed_config_tag);
+```
+- Preconditions: valid endpoint.
+- Postconditions: an `OP_nop` request with `body_which == Request_body_heartbeatmsg` is enqueued; `HeartbeatMsg.term/committedLsn/committedConfigTag` set; `senderPriority` unused (0).
+- Invariants: non-blocking; carries no other state.
+
+### Controller — `ha_observe_heartbeat_committed` (Service / State)
+| Field | Detail |
+|-------|--------|
+| Intent | Monotonically advance a standby's learned committed position from a heartbeat, with no reconciliation |
+| Requirements | 2.1, 2.2, 2.3, 3.1 |
+
+**Responsibilities & Constraints**
+- Advance `last_known_committed_lsn` to the carried LSN only if it is strictly greater (monotonic; single receiver thread, so a relaxed compare-then-`atomic_store`/`acquire`-`release` or `atomic_fetch_max` suffices).
+- Advance `last_known_committed_config_tag` **together with** the LSN (only when the LSN advances); never regress it independently (2.3).
+- **Must NOT** call `ha_rejoin_reconcile`, truncate the WAL, or modify any produced/persisted committed state (3.1, 3.3). This is the explicit distinction from `cluster_apply_ctrlhello_reply`.
+- No-op when the carried LSN is `<=` current or zero (2.2, 4.2).
+
+**Dependencies**
+- Inbound: `cluster.c` `OP_nop` handler — passes the term-checked payload (P0).
+- Outbound: `ha->last_known_committed_lsn` / `last_known_committed_config_tag` atomics (P0).
+
+### Controller — `OP_nop` handler consume branch (Service)
+- After the existing liveness/`StateReport` handling, if `request.body_which == Request_body_heartbeatmsg` and the responder is non-primary: apply the **term-guard** (`hb->term >= atomic_load(ha->term)`) and, if it passes, call `ha_observe_heartbeat_committed(c, hb->committed_lsn, hb->committed_config_tag)`.
+- If the body is absent, the term-guard fails, or committed is zero → no committed update; liveness handling is unchanged (4.1, 4.2).
+- Constraint: this branch must not alter the reply, the liveness reset, or any existing `OP_nop` behavior.
+
+## Data Models
+
+No new persistent data and no schema-shape change. `HeartbeatMsg` (existing) is the request-body DTO; its `term`, `committedLsn`, `committedConfigTag` fields are populated by the producer and read by the consumer. The only in-memory state touched is `ha->last_known_committed_lsn` (already `_Atomic`) and `ha->last_known_committed_config_tag` (**migrated to `_Atomic uint64_t`** so the heartbeat-receive writer, the join writer, the election writer, and the `ctrlstatus` reader interact safely). Both are advisory replicas of the primary's committed markers — never authoritative, never persisted by this path.
+
+## Error Handling
+
+- **Malformed/absent body:** `body_which != Request_body_heartbeatmsg` → treat as a plain liveness heartbeat; no error (4.2).
+- **Stale-term heartbeat:** `hb->term < ha->term` → ignore the committed payload (term-guard); liveness handled normally. This is the safety gate, not an error.
+- **Out-of-order / lower committed:** monotonic no-op (2.2); not an error.
+- **Send failure:** `xce_ha_heartbeat_req` returns the same negative as the nop path; `primary_heartbeat_fn` logs `WARN` exactly as today — no behavior change (4.1).
+- **Observability:** keep heartbeat-committed updates at a low/debug log level (the heartbeat path is hot); do not add per-tick INFO spam.
+
+## Testing Strategy
+
+### Unit (`test_ha_heartbeat_committed.c`)
+- A standby with `last_known_committed_lsn = 4` receiving a heartbeat (`term >= ha->term`) carrying `committed_lsn = 9` → `last_known_committed_lsn == 9` and `last_known_committed_config_tag` updated (1.2, 2.1, 2.3).
+- A heartbeat carrying `committed_lsn = 3` (≤ current 9) → `last_known_committed_lsn` unchanged (2.2).
+- A **stale-term** heartbeat (`hb->term < ha->term`) carrying a higher committed → `last_known_committed_*` unchanged (3.2 term-guard); liveness still reset (4.1).
+- A heartbeat with no `heartbeatmsg` body / zero committed → no committed update, processed as liveness (4.2).
+- `ha_observe_heartbeat_committed` performs no WAL truncation / no reconcile call and mutates nothing beyond `last_known_committed_*` (3.1, 3.3) — assert via a no-truncate / state-snapshot check.
+
+### Integration
+- Producer round-trip: `xce_ha_heartbeat_req` encodes an `OP_nop` whose decoded request has `body_which == heartbeatmsg` with the expected `term/committedLsn/committedConfigTag` (1.1).
+- Vote-floor interaction: with a heartbeat-raised `last_known_committed_lsn`, the existing freshness gate rejects a candidate below it and accepts one at/above it — confirming the floor consumes the fresher value through the unchanged formula and is never lowered (3.2).
+
+### E2E (live `/xce-deploy-ha`)
+- On a healthy 3-node cluster, submit jobs so the primary advances committed; `ctrlstatus` on a **standby** shows `committedLsn` tracking the primary within ~a heartbeat interval (no re-join), closing the controller-status Req 10 staleness caveat (1.3, downstream).
+
+## Security Considerations
+
+The only safety-relevant change is feeding a heartbeat-fresh `last_known_committed_lsn` into the (unchanged) vote-freshness floor `max(durable_floor, last_known_committed_lsn)`.
+
+- **Why raising the floor is safe (Leader Completeness):** committed entries are, by definition, durable on a majority and cluster-wide monotonic (State Machine Safety). A heartbeat-borne committed accepted under the **term-guard** (`hb->term >= ha->term`) therefore reflects a genuinely committed position. Requiring vote candidates to be at least that fresh *strengthens* the guarantee that a new leader holds all committed entries; it never weakens it (3.2). The value only ever raises the floor (monotonic), so the floor cannot drop.
+- **No liveness wedge:** since a majority holds the committed entries, an eligible candidate (`lastLogLsn >= floor`) always exists; a node behind the floor correctly fails its own freshness check and should not lead.
+- **Stale/forged-term defense:** the term-guard rejects committed advertised by a primary whose term is behind the standby's, preventing an old/partitioned primary from inflating the floor. The value is otherwise advisory: it never drives truncation (3.1) and never changes how committed is produced (3.3).
+- **No new authorization surface:** the heartbeat is the existing authenticated primary→standby channel; this only adds three integer fields to a message that already flows.
--- a/.kiro/specs/ha-heartbeat-committed-propagation/requirements.md
+++ b/.kiro/specs/ha-heartbeat-committed-propagation/requirements.md
@ -0,0 +1,55 @@
+# Requirements Document
+
+## Introduction
+
+This feature makes a standby controller's knowledge of the cluster's **committed position** stay current from the primary's **heartbeats**, rather than only being refreshed when the standby (re)joins the primary. Today a standby learns the committed position (committed log position and committed configuration version) only at a join / failover handshake; between joins that knowledge goes stale, and a standby that has applied replicated entries can still report a committed position of zero. The heartbeat already flows continuously between primary and standbys, so it is the natural carrier for a fresh committed marker.
+
+With this feature, the serving primary includes its current committed position in every heartbeat, and each standby uses it to keep its own known committed position up to date — within roughly one heartbeat interval — without waiting for a re-join. This directly removes the staleness caveat documented by the controller-status feature (whose `ctrlstatus` query reports a standby's known committed position): a standby's reported committed position becomes near-real-time instead of join-stale or zero.
+
+The propagated committed marker is **advisory** — it exists for observability and standby state freshness only. It must not change how the committed position is produced, advanced, replicated, or persisted, and it must not feed any correctness-critical decision (log truncation, election/vote safety). Existing heartbeat liveness and failover behavior must be unaffected.
+
+> Full discovery context (problem, approach, scope, boundary candidates, upstream/downstream, constraints) is in `.kiro/specs/ha-heartbeat-committed-propagation/brief.md`.
+
+## Boundary Context
+
+- **In scope**: the serving primary advertising its current committed position (committed log position + committed configuration version) on each heartbeat; a standby consuming that value to keep its own **known committed position** current; monotonic (never-regressing) update of that known position; graceful handling of heartbeats that carry no usable committed value; preserving all existing heartbeat liveness/failover behavior.
+- **Out of scope**: how the committed position is produced, advanced, replicated, or persisted (owned by the commit engine / durability); the join/failover handshake that already seeds the standby's committed knowledge (unchanged, remains the initial source); the catch-up streaming trigger (remains acknowledgement-driven, not heartbeat-driven); the `ctrlstatus` reporting/rendering of the committed position (owned by controller-status, which already reads the standby's known committed position); any change to how a node truncates its log or evaluates vote freshness.
+- **Adjacent expectations**: relies on the existing heartbeat channel and timing (the primary already emits heartbeats and standbys already receive them) and on the existing committed-position semantics, both consumed unchanged. The **downstream** consumer is the controller-status `ctrlstatus` query: once a standby's known committed position is heartbeat-fresh, that query reflects it with no change to controller-status. Vote-freshness evaluation may read the standby's known committed position as part of its floor, but this feature must not let a heartbeat-borne value lower or weaken that floor.
+
+## Requirements
+
+### Requirement 1: Heartbeat-borne committed position propagation
+
+**Objective:** As an operator, I want a standby's known committed position to refresh from the primary's heartbeats, so that it reflects the live cluster committed state without waiting for the standby to re-join.
+
+#### Acceptance Criteria
+1. While a node is the serving primary, when it emits a heartbeat, the Controller shall include in that heartbeat the primary's current committed log position and committed configuration version.
+2. When a standby receives a heartbeat that carries a committed position, the Controller shall update the standby's known committed position (committed log position and committed configuration version) from the values in that heartbeat, subject to the monotonicity rule in Requirement 2.
+3. While a standby is continuously receiving heartbeats from a serving primary, the standby's known committed position shall reflect the primary's committed position within one heartbeat interval, without requiring a re-join handshake.
+
+### Requirement 2: Monotonic, non-regressing committed knowledge
+
+**Objective:** As an operator, I want a standby's known committed position to never move backward, so that observed committed progress stays stable and trustworthy across heartbeats.
+
+#### Acceptance Criteria
+1. When a standby receives a heartbeat whose carried committed log position is greater than the standby's current known committed log position, the Controller shall advance the known committed position to the carried value.
+2. If a standby receives a heartbeat whose carried committed log position is less than or equal to the standby's current known committed log position (for example a delayed or out-of-order heartbeat, or a newly started or lagging primary), then the Controller shall leave the standby's known committed position unchanged.
+3. The standby's known committed configuration version shall be advanced together with the known committed log position and shall never be regressed independently.
+
+### Requirement 3: Advisory-only — no effect on correctness-critical paths
+
+**Objective:** As a maintainer, I want the heartbeat-borne committed value used only for observability and standby state freshness, so that it cannot weaken durability or election-safety guarantees.
+
+#### Acceptance Criteria
+1. The Controller shall not use a heartbeat-borne committed position to truncate, discard, or otherwise modify any write-ahead log entry.
+2. The Controller shall not use a heartbeat-borne committed position to lower or weaken the durability-backed freshness floor used when granting votes.
+3. The Controller shall not change how the committed position is produced, advanced, replicated, or persisted as a result of propagating it on heartbeats.
+4. While a node is the serving primary, the Controller shall continue to use and report its own authoritative committed position; a heartbeat-borne committed value shall be consumed only by non-primary nodes.
+
+### Requirement 4: No regression to heartbeat liveness behavior
+
+**Objective:** As an operator, I want existing heartbeat liveness and failover behavior to be unchanged, so that adding committed propagation does not destabilize the cluster.
+
+#### Acceptance Criteria
+1. The Controller shall preserve existing heartbeat liveness detection, missed-heartbeat tolerance, and proactive-heartbeat timing behavior unchanged when committed propagation is added.
+2. If a received heartbeat carries no usable committed position (for example a zero or absent value), then the Controller shall process it as a liveness heartbeat exactly as before and shall leave the standby's known committed position unchanged, without error.
--- a/.kiro/specs/ha-heartbeat-committed-propagation/research.md
+++ b/.kiro/specs/ha-heartbeat-committed-propagation/research.md
@ -0,0 +1,64 @@
+# Research / Gap Analysis — ha-heartbeat-committed-propagation
+
+> Generated by `/kiro-validate-gap` (2026-06-20). Brownfield; informs design. Requirements: `requirements.md` (Req 1–4). Brief: `brief.md`.
+
+## Current State (verified)
+
+- **The HA primary→standby heartbeat is a bare `OP_nop` with no body.** `primary_heartbeat_fn` sends `xce_nop_req(ep)` (`ha.c:5256`); the standby `OP_nop` handler (`cluster.c:300`) replies a `StateReport` (`ha_nop_reply_state_report`) and resets liveness (`ha_on_heartbeat_received`) — it reads no request body and learns no committed value.
+- **`OPCode_heartbeat` is a SEPARATE, client-side keepalive** (`xce_heartbeat(ctx, hb_interval)`, `xceapi.c:48`; `client.c:233`; dispatch-gate `ha.c:2077/2120/2161`) — NOT the HA heartbeat. **Do not repurpose it** (collision).
+- **The wire already supports a committed-bearing heartbeat, but nothing uses it:** the Request body union has `heartbeatmsg @25: HeartbeatMsg` (`controller.capnp:653`), and `HeartbeatMsg{senderPriority@0, term@1, committedLsn@2, committedConfigTag@3}` (`controller.capnp:761`) is fully defined. Grep finds **zero** `.c` producers or consumers of `heartbeat_msg_t`/`heartbeatmsg`/its `committedLsn` — dead scaffolding.
+- **Standby committed knowledge today:** `ha->last_known_committed_lsn` (`_Atomic uint64_t`, `ha.h:170`) + `ha->last_known_committed_config_tag` (**plain** `uint64_t`, `ha.h:172`) are written only at join via `cluster_apply_ctrlhello_reply` (`ha_convergence.c:372`) — a **plain `atomic_store` (NOT monotonic)** that ALSO calls `ha_rejoin_reconcile` (truncation) — and at an election safe-commit path (`ha.c:925`).
+- **A monotonic-max precedent exists** for committed advance in `commit.c` (`monotonic max() store`, `commit.c:141/151/155`) — mirror it for the heartbeat update.
+- **`last_known_committed_lsn` already feeds the vote-freshness floor**: voter floor = `max(durable_floor, last_known_committed_lsn)` (`ha.c:2570–2586`; established by bug-339 / `project_ha_vote_freshness_invariant`). This is the key safety interaction (below).
+- `xce_nop_req(ep)` is **generic** (also the client `xce_nop`, `xceapi.c:42`) — must not gain a mandatory committed body.
+
+## Requirement-to-Asset Map
+
+| Need (Req) | Existing asset | Gap |
+|---|---|---|
+| 1.1 primary advertises committed on each heartbeat | `primary_heartbeat_fn` sends bare `xce_nop_req`; `HeartbeatMsg{committedLsn,committedConfigTag}` + `heartbeatmsg` body union member exist | **Missing producer**: a HA-heartbeat sender that attaches a `HeartbeatMsg` body carrying `committed_lsn`/`committed_config_tag` |
+| 1.2 standby consumes it | `cluster.c:300` `OP_nop` handler (reads no body); `request.body_which`/`body` union already supports `heartbeatmsg` | **Missing consumer**: read `body.heartbeatmsg` and update `last_known_committed_*` |
+| 1.3 fresh within one heartbeat interval | heartbeat tick cadence (`heartbeat_interval`) | property of 1.1+1.2; testable via tick |
+| 2.1–2.3 monotonic, never regress (LSN + config tag together) | `commit.c` monotonic-max pattern; `last_known_committed_lsn` is `_Atomic` (single receiver thread) | **Constraint**: existing `cluster_apply_ctrlhello_reply` does a PLAIN store — the heartbeat path needs a **monotonic** update; config tag must move with the LSN |
+| 3.1 no truncation from heartbeat-borne committed | `cluster_apply_ctrlhello_reply` calls `ha_rejoin_reconcile` (truncation) | **Critical**: the heartbeat consume path must NOT reconcile/truncate — needs a separate, reconcile-free updater (do NOT reuse `cluster_apply_ctrlhello_reply`) |
+| 3.2 must not weaken the vote-freshness floor | voter floor `max(durable_floor, last_known_committed_lsn)` (`ha.c:2570`) | **Research Needed (highest)**: a heartbeat-fresh (higher) `last_known` RAISES the floor — verify that is safe (Leader Completeness) and cannot deadlock liveness (a standby refusing all candidates) |
+| 3.3 no change to committed production/replication/persist | consume path is read+store only | none — keep it read-only w.r.t. produced committed |
+| 3.4 primary uses its own committed | only standbys process inbound HA heartbeats; primary sends | inherently satisfied; gate the consume to non-primary for clarity |
+| 4.1 existing liveness unchanged | `OP_nop` still resets liveness + replies StateReport | additive body-read only; verify no timing change |
+| 4.2 absent/zero committed → plain liveness, no-op | monotonic update naturally no-ops on 0 / ≤current | handle `body_which != heartbeatmsg` and zero value as liveness-only |
+
+## Implementation Approach Options
+
+**Option A (Extend the `OP_nop` HA heartbeat) — RECOMMENDED.** Add a dedicated HA-heartbeat sender (e.g. `xce_ha_heartbeat_req(ep, committed_lsn, committed_config_tag)`) that sends `OPCode_nop` **with** a `heartbeatmsg` body; the `cluster.c` `OP_nop` handler reads it and monotonically updates `last_known_committed_*` via a new reconcile-free helper. Reuses the already-defined `HeartbeatMsg` + body union member; leaves generic `xce_nop_req` (client nop) untouched.
+*Trade-off*: ✅ minimal, reuses dead scaffolding, no new opcode. ❌ touches the hot HA heartbeat path; an `OP_nop` carrying a `heartbeatmsg@25` body is slightly unconventional (a request may carry any body-union member regardless of opcode — design confirms the pairing).
+
+**Option B (Dedicated HA-heartbeat opcode).** New opcode (distinct from the client `OPCode_heartbeat`) paired with `HeartbeatMsg`, plus dispatch-gate + handler wiring. Cleaner semantic separation. *Trade-off*: ✅ unambiguous. ❌ more churn (opcode, gate, handler, sender swap) for the same effect.
+
+**Option C (Repurpose `OPCode_heartbeat`) — REJECT.** It is the client keepalive (`xce_heartbeat`/`hb_interval`); reusing it collides with existing client semantics.
+
+## Effort & Risk
+
+- **Producer (Req 1.1)**: S — one sender + call-site swap in `primary_heartbeat_fn`.
+- **Consumer + monotonic update (Req 1.2, 2)**: S — one handler branch + a reconcile-free monotonic helper.
+- **Vote-freshness safety review (Req 3.2)**: M — analysis, not much code, but the gating risk.
+- **Combined**: **S–M (3–5 days)**. **Risk Medium** — not the propagation (mechanical) but two interactions: (a) the hot heartbeat path, (b) feeding a fresher `last_known_committed_lsn` into the vote floor.
+
+## Research Items to Carry to Design
+
+1. **Vote-freshness safety (do this first).** Confirm against `raft.tla` Leader Completeness that raising `last_known_committed_lsn` from a heartbeat-advertised committed (which the standby has not necessarily applied/durably stored) is safe to feed the voter floor `max(durable_floor, last_known_committed_lsn)`, and that it cannot wedge liveness (standby refusing otherwise-valid candidates). If risky, gate: only let the heartbeat raise `last_known` up to what the standby can corroborate, or keep the floor on durable state and let only reporting consume the heartbeat value.
+2. **Reconcile-free consume path.** Do NOT reuse `cluster_apply_ctrlhello_reply` (it truncates). Design a monotonic-only `ha_observe_heartbeat_committed`; decide whether to factor a shared monotonic core used by both join and heartbeat (join adds reconcile, heartbeat does not).
+3. **`last_known_committed_config_tag` atomicity.** It is non-atomic (`ha.h:172`); with a new heartbeat-receive writer plus the `ctrlstatus` reader, decide whether to promote to `_Atomic` (and update the 2–3 existing write sites) — or keep best-effort.
+4. **Sender scoping.** Add a HA-specific sender; do NOT alter generic `xce_nop_req`.
+5. **Opcode/body pairing.** Confirm `OP_nop` + `heartbeatmsg` body is accepted end-to-end by the codec/dispatch, or choose Option B's dedicated opcode.
+
+---
+
+## Synthesis / Design Decisions (2026-06-20)
+
+Resolutions of the 5 research items, locked into `design.md`:
+
+1. **Vote-freshness safety — RESOLVED, no vote-path change needed.** The voter floor formula `max(durable_floor, last_known_committed_lsn)` stays UNCHANGED; it already reads `last_known_committed_lsn`. Making that value heartbeat-fresh only RAISES it (monotonic), which *strengthens* the freshness gate (Leader Completeness) rather than weakening it (Req 3.2). Safety rests on three facts: (a) **term-guard** — a standby accepts the heartbeat-borne committed only from a primary whose advertised `term >= ha->term` (`HeartbeatMsg.term@1`), so a stale-term primary cannot inflate the floor; (b) **monotonic** — never regress (Req 2); (c) cluster-wide `committed_lsn` is itself monotonic and majority-durable (State Machine Safety), so any accepted value is genuinely committed. No liveness wedge: committed entries live on a majority, so an eligible candidate (≥ floor) always exists. A node behind the floor correctly fails its own freshness check and should not lead. **Conclusion:** feed the existing floor unchanged; the guard is on the *accept*, not the *use*.
+2. **Reconcile-free consume — RESOLVED.** New `ha_observe_heartbeat_committed(c, lsn, tag)`: monotonic-max update only; does NOT call `ha_rejoin_reconcile`/truncate (Req 3.1). Distinct from `cluster_apply_ctrlhello_reply` (which truncates at join).
+3. **config_tag atomicity — RESOLVED.** Promote `last_known_committed_config_tag` to `_Atomic uint64_t` (`ha.h:172`); update the 2 existing writers (`ha_convergence.c:377`, `ha.c` election path) to `atomic_store`. New heartbeat writer + the `ctrlstatus` reader justify it.
+4. **Sender scoping — RESOLVED.** New `xce_ha_heartbeat_req(ep, term, committed_lsn, committed_config_tag)`; generic `xce_nop_req` untouched (client nop unaffected).
+5. **Opcode/body pairing — RESOLVED.** Send `OPCode_nop` WITH a `heartbeatmsg` body (the Request union already has `heartbeatmsg@25`; opcode and body are independent, as `OP_hello` already carries different bodies). Dispatch still routes by `OPCode_nop`; the handler branches on `body_which == Request_body_heartbeatmsg`. No new opcode (Option A over B).
--- a/.kiro/specs/ha-heartbeat-committed-propagation/spec.json
+++ b/.kiro/specs/ha-heartbeat-committed-propagation/spec.json
@ -0,0 +1,22 @@
+{
+  "feature_name": "ha-heartbeat-committed-propagation",
+  "created_at": "2026-06-20T00:00:00Z",
+  "updated_at": "2026-06-20T00:00:00Z",
+  "language": "en",
+  "phase": "tasks-generated",
+  "approvals": {
+    "requirements": {
+      "generated": true,
+      "approved": true
+    },
+    "design": {
+      "generated": true,
+      "approved": true
+    },
+    "tasks": {
+      "generated": true,
+      "approved": true
+    }
+  },
+  "ready_for_implementation": true
+}
--- a/.kiro/specs/ha-heartbeat-committed-propagation/tasks.md
+++ b/.kiro/specs/ha-heartbeat-committed-propagation/tasks.md
@ -0,0 +1,60 @@
+# Implementation Plan
+
+> TDD is mandatory (project rule): add the failing C test in `testsuite/` (or a piped `xce_test` behavioral script) before the change — red → green → refactor.
+> The work runs along a single request path (atomic migration → libxce sender → primary stamp / standby consume → vote-floor verification → E2E), so tasks are sequential by design; no `(P)` markers. The two foundation tasks (1.1 atomic migration, 1.2 sender) touch independent files but are kept ordered for review simplicity. The HA-critical change is the standby consume + term-guard (task 2.2) and its vote-floor interaction (task 3.2).
+
+- [ ] 1. Foundation: atomic state + heartbeat sender
+- [ ] 1.1 Migrate `last_known_committed_config_tag` to `_Atomic`
+  - Change `ha->last_known_committed_config_tag` from a plain `uint64_t` to `_Atomic uint64_t` (`src/controller/include/ha.h`) and update its existing writers — `cluster_apply_ctrlhello_reply` (`ha_convergence.c`) and the election safe-commit path (`ha.c`) — to `atomic_store_explicit(..., release)`, and any reader (incl. the controller-status `ctrlstatus_fill`) to `atomic_load_explicit(..., acquire)`. Pure type/access migration — no behavior change.
+  - Observable: the controller suite builds clean and passes unchanged; every read/write of the field uses atomic accessors (no plain access remains).
+  - _Requirements: 2.3_
+  - _Boundary: ha.h + last_known_committed_config_tag accessors_
+- [ ] 1.2 Add the HA heartbeat sender `xce_ha_heartbeat_req`
+  - Add `xce_ha_heartbeat_req(endpoint_t* ep, uint64_t term, uint64_t committed_lsn, uint64_t committed_config_tag)` (`src/include/request.h` + `src/lib/request.c`) that sends an `OPCode_nop` request carrying a `heartbeatmsg` body (`HeartbeatMsg{term, committedLsn, committedConfigTag}`; `senderPriority` unused). Leave the generic `xce_nop_req` untouched.
+  - TDD: a round-trip test encodes via the new sender and decodes a request with `body_which == Request_body_heartbeatmsg` carrying the three field values; a guard assertion confirms `xce_nop_req` still sends a bodyless nop.
+  - Observable: the round-trip test decodes the three fields from the new sender, and the bodyless-`xce_nop_req` guard still holds.
+  - _Requirements: 1.1_
+  - _Boundary: libxce sender_
+
+- [ ] 2. Standby consume (reconcile-free, term-guarded, monotonic)
+- [ ] 2.1 Add the reconcile-free monotonic updater `ha_observe_heartbeat_committed`
+  - Add `ha_observe_heartbeat_committed(cluster_t* c, uint64_t committed_lsn, uint64_t committed_config_tag)` (`ha.c`, declared in `ha.h`): advance `last_known_committed_lsn` only when the carried LSN is strictly greater (monotonic; atomic), advancing `last_known_committed_config_tag` together with it; no-op when `<=` current or zero. It must NOT call `ha_rejoin_reconcile`, truncate the WAL, or modify any produced/persisted committed state.
+  - TDD: unit tests assert advance-on-higher (LSN + config tag updated), no-op on equal/lower/zero, config tag never regresses independently, and — critically — that the call performs no WAL truncation / no reconcile and mutates nothing beyond `last_known_committed_*`.
+  - Observable: the unit tests go red → green; the no-truncation / no-mutation assertions hold.
+  - _Requirements: 2.1, 2.2, 2.3, 3.1, 3.3_
+  - _Boundary: ha_observe_heartbeat_committed_
+  - _Depends: 1.1_
+- [ ] 2.2 Wire the standby `OP_nop` consume branch + term-guard
+  - In the `cluster.c` `OP_nop` handler, after the existing liveness reset + `StateReport` reply, if `request.body_which == Request_body_heartbeatmsg` and the responder is non-primary: apply the term-guard (`hb->term >= atomic_load(ha->term)`) and, only if it passes, call `ha_observe_heartbeat_committed(c, hb->committed_lsn, hb->committed_config_tag)`. A missing body, a failed term-guard, or a zero committed value updates nothing and leaves all existing `OP_nop` behavior intact.
+  - TDD: tests confirm a term-valid higher-committed heartbeat advances `last_known_committed_*`; a stale-term heartbeat (`hb->term < ha->term`) carrying a higher committed leaves committed unchanged but STILL resets liveness; a bodyless / zero-committed nop is processed as liveness only; and a primary responder does not consume an inbound committed value.
+  - Observable: the tests pass for the term-valid, stale-term, bodyless, and primary-responder cases, with liveness reset preserved in every case.
+  - _Requirements: 1.2, 3.2, 3.4, 4.1, 4.2_
+  - _Boundary: cluster.c OP_nop handler_
+  - _Depends: 2.1_
+
+- [ ] 3. Primary produce + safety verification
+- [ ] 3.1 Stamp committed + term on the primary heartbeat
+  - In `primary_heartbeat_fn` (`ha.c`), replace the per-standby `xce_nop_req(ep)` heartbeat call with `xce_ha_heartbeat_req(ep, atomic_load(ha->term), atomic_load(persist.committed_lsn), atomic_load(ha->committed_config_tag))`. Preserve the `XCE_TESTING` `ha_nop_req_hook` seam and the existing heartbeat send-failure `WARN` and `last_push_time` bookkeeping.
+  - TDD/observable: a test (or the `ha_nop_req_hook` seam) confirms the primary's heartbeat now carries its current committed LSN / committed config tag / term; the send-failure and liveness-eviction behavior of `primary_heartbeat_fn` is unchanged.
+  - _Requirements: 1.1, 3.4_
+  - _Boundary: primary_heartbeat_fn_
+  - _Depends: 1.2_
+- [ ] 3.2 Verify the vote-freshness floor consumes the fresher value safely
+  - Add an integration test proving the existing freshness gate, with `last_known_committed_lsn` raised by a heartbeat (via `ha_observe_heartbeat_committed`), rejects a vote candidate whose last-log LSN is below the raised floor and accepts one at/above it — confirming the floor formula `max(durable_floor, last_known_committed_lsn)` is read unchanged and is never lowered by this feature. No production change to the vote path.
+  - Observable: the test shows the candidate-below-floor rejected and candidate-at/above-floor accepted after a heartbeat raise, and no edit to the floor formula.
+  - _Requirements: 3.2_
+  - _Boundary: vote-freshness interaction test_
+  - _Depends: 2.1, 2.2_
+
+- [ ] 4. Validation (live cluster)
+- [ ] 4.1 End-to-end: standby committed tracks the primary within a heartbeat interval
+  - On a live 3-node `/xce-deploy-ha` cluster, submit jobs so the primary advances its committed position, then query a STANDBY repeatedly: its `ctrlstatus` `committedLsn` advances to track the primary within roughly one heartbeat interval, without any re-join — closing the controller-status Req 10 staleness caveat. Confirm existing liveness/failover behavior is unaffected.
+  - Observable: the live run shows a standby's `committedLsn` rising to track the primary between joins (not frozen at the join value), and the cluster's liveness/failover behavior is unchanged; manual environment prerequisite (live cluster, real fs).
+  - _Requirements: 1.2, 1.3_
+  - _Depends: 3.1, 2.2_
+
+## Implementation Notes
+- The HA heartbeat is `OPCode_nop` with a `heartbeatmsg` body — opcode and body are independent in this schema (mirrors `OP_hello` carrying `ctrlhellomsg` vs `replymsg`). Dispatch routes by `OPCode_nop` (unchanged); the consumer branches on `body_which`.
+- Do NOT reuse `cluster_apply_ctrlhello_reply` for the heartbeat path — it truncates (`ha_rejoin_reconcile`). The heartbeat updater is reconcile-free (Req 3.1).
+- Do NOT touch the client keepalive `OPCode_heartbeat` / `xce_heartbeat` (a different message) or the generic `xce_nop_req` (client nop).
+- The vote-floor formula is NOT edited — task 3.2 is a verification test only. Safety rests on the term-guard (task 2.2) + monotonicity (task 2.1) + cluster-monotonic committed; see design.md Security Considerations.