myxce/.kiro/specs/ha-heartbeat-committed-propagation/brief.md

Brief: ha-heartbeat-committed-propagation

Opened 2026-06-20 from a controller-status (ctrlstatus) live finding. This is the live-fresh off-primary committed follow-up that controller-status Req 10 explicitly routed out of its own boundary.

Problem

A standby's knowledge of the cluster's committed position — ha->last_known_committed_lsn / ha->last_known_committed_config_tag — is refreshed only at CtrlHello join / RDHUP-failover (cluster_apply_ctrlhello_reply, ha_convergence.c:375) and an election safe-commit path. It is not refreshed on steady-state heartbeats. So between joins, a standby's view of committed lags the live primary, even as the standby keeps applying replicated entries.

This surfaced live in ctrlstatus (controller-status Req 10): a standby reported committedLsn: 0 while appliedLsn: 10 — honest per Req 10.2 (it never learned a committed position) but not useful. Even after a join, the reported committed can sit stale between joins. controller-status Req 10 deliberately reports the honest best-known value and documented this staleness as an out-of-boundary caveat to be fixed here.

The wire already anticipates this: HeartbeatMsg (src/schema/controller.capnp:761) defines committedLsn @2 and committedConfigTag @3, but no code populates or consumes them — the OP_nop heartbeat carries no committed payload, and ha_on_heartbeat_received only updates liveness.

Current State

• HeartbeatMsg{committedLsn, committedConfigTag} fields exist in the schema but are dead (no producer, no consumer).
• last_known_committed_lsn (ha.h:170, _Atomic) + last_known_committed_config_tag (ha.h:172, plain) written only at join/RDHUP + election safe-commit.
• OP_nop heartbeat receive path (cluster.c:300) resets liveness only.
• controller-status ctrlstatus_fill already reports last_known_committed_* off-primary (controller-status task 6.2) — it will automatically reflect a fresher value once this spec lands.

Desired Outcome

The primary stamps its current committed_lsn / committed_config_tag into every heartbeat; each standby consumes them in the heartbeat receive path and monotonically advances last_known_committed_lsn / last_known_committed_config_tag. Off-primary committed then tracks the live primary within one heartbeat interval, and ctrlstatus on a standby shows a near-real-time committed position instead of a join-stale (or zero) value.

Approach

Populate the already-defined HeartbeatMsg.committedLsn/committedConfigTag on the primary's heartbeat sender; consume them on the standby's heartbeat receive path with a monotonic update of last_known_committed_*. No change to how committed is produced/replicated/persisted — this only propagates an already-committed marker as liveness-adjacent metadata.

Scope

• In: primary heartbeat sender populates committedLsn/committedConfigTag; standby heartbeat receive path consumes them → monotonic update of last_known_committed_lsn/last_known_committed_config_tag; unit + integration tests (standby's learned committed advances on heartbeat without a re-join); memory-ordering consistency for the non-atomic last_known_committed_config_tag vs the ctrlstatus reader.
• Out: how committed_lsn is produced/advanced (ha-commit-engine / commit durability); the ctrlstatus reporting itself (controller-status Req 10, already done); WAL catch-up triggering (stays ACK-driven per ha-convergence-recovery); election safe-commit derivation (unchanged — must NOT use heartbeat-borne committed for truncation, per the existing "never truncate from stale ctrlhello/heartbeat" rule).

Boundary Candidates

• Primary heartbeat sender (stamp committed into HeartbeatMsg).
• Standby heartbeat receive (consume + monotonic last_known_* update).
• The monotonicity + memory-ordering contract on last_known_committed_*.

Out of Boundary

• Committed-LSN production / quorum advance (commit engine).
• ctrlstatus rendering and field sourcing (controller-status).
• Any use of heartbeat-borne committed for WAL truncation or vote-freshness floor changes (explicitly forbidden — truncation/freshness must stay durable-backed; this value is advisory/reporting only).

Upstream / Downstream

• Upstream: ha-heartbeat-liveness (owns HeartbeatMsg + the heartbeat tick/receive path), ha-commit-engine (committed_lsn semantics).
• Downstream: controller-status Req 10 (off-primary committed becomes live-fresh; removes the documented staleness caveat); ha-vote-freshness-invariant (voter floor max(durable_floor, last_known_committed_lsn) benefits from a fresher last_known_committed_lsn, but freshness must remain correct — verify no safety regression).

Existing Spec Touchpoints

• Extends: ha-heartbeat-liveness (heartbeat content/behavior).
• Adjacent: ha-repl-hardening (standby replication state), controller-status (the consumer that motivated this), ha-convergence-recovery (catch-up stays ACK-driven, not heartbeat-driven).

Constraints

• last_known_committed_* updates must be monotonic (never regress on an out-of-order/stale heartbeat).
• Heartbeat path is hot — keep the added work to two field reads + two conditional stores.
• The heartbeat-borne committed is advisory (reporting/liveness): it must never feed WAL truncation or weaken the durable-backed vote-freshness floor.
• last_known_committed_config_tag is non-atomic (ha.h:172); reconcile the writer/reader memory ordering with the ctrlstatus best-effort read (consider promoting to _Atomic).