myxce/.kiro/specs/ha-heartbeat-committed-propagation/design.md

Design Document — ha-heartbeat-committed-propagation

Overview

Purpose: keep a standby controller's knowledge of the cluster's committed position current from the primary's heartbeats, instead of only at a join/failover handshake. The serving primary attaches its committed position (committed LSN + committed configuration version) to every HA heartbeat; each standby consumes it and monotonically advances its last_known_committed_* state.

Users: operators inspecting a standby via the controller-status ctrlstatus query (its off-primary committed becomes near-real-time instead of join-stale or zero), and the HA vote-freshness path (which already reads last_known_committed_lsn and gains a fresher, term-guarded value).

Impact: the HA heartbeat is a bare OP_nop today. This wires up the already-defined-but-unused HeartbeatMsg{committedLsn, committedConfigTag} body on that heartbeat (producer + consumer), adds a reconcile-free monotonic updater, and promotes one field to atomic. No new opcode; no change to how committed is produced/replicated/persisted; no change to the vote-floor formula.

Goals

• The serving primary advertises its committed LSN + committed config version on every heartbeat (1.1).
• A standby refreshes its known committed position from heartbeats, monotonically, within one heartbeat interval, with no re-join (1.2, 1.3, 2.x).
• The propagated value is advisory: it never drives truncation and never weakens the vote-freshness floor (3.x).
• Existing heartbeat liveness/failover behavior is unchanged (4.x).

Non-Goals

• Changing how committed_lsn is produced, advanced, replicated, or persisted (commit engine / durability).
• The join/failover handshake (cluster_apply_ctrlhello_reply) that already seeds the standby's committed knowledge — left as-is (it remains the initial source and the only path that reconciles/truncates).
• Catch-up streaming triggering (stays ACK-driven, per ha-convergence-recovery).
• The ctrlstatus reporting/rendering of committed (owned by controller-status; it already reads last_known_committed_*).

Boundary Commitments

This Spec Owns

• The producer: attaching the primary's committed LSN + committed config tag (and its term) to the HA heartbeat request.
• The consumer: reading that payload on a standby and monotonically updating last_known_committed_lsn / last_known_committed_config_tag, gated by a term check, without reconciliation/truncation.
• The new reconcile-free monotonic updater ha_observe_heartbeat_committed.
• Promoting last_known_committed_config_tag to _Atomic and migrating its existing writers.

Out of Boundary

• The vote-freshness floor formula (max(durable_floor, last_known_committed_lsn)) — unchanged; this spec only makes its input fresher.
• The join handshake / RDHUP path and its truncation/reconcile (cluster_apply_ctrlhello_reply, ha_rejoin_reconcile).
• Committed production/replication/persistence, WAL truncation, catch-up streaming.
• The client keepalive OPCode_heartbeat / xce_heartbeat (a different message — must not be touched or reused).
• ctrlstatus field sourcing/rendering (controller-status).

Allowed Dependencies

• The Cap'n Proto schema + codegen: the existing HeartbeatMsg struct and the heartbeatmsg @25 Request-body union member (src/schema/controller.capnp) — reused, no shape change.
• The existing HA heartbeat tick (primary_heartbeat_fn, ha.c) and the standby OP_nop receive path (cluster.c).
• Read access to the primary's persist.committed_lsn and ha->committed_config_tag; read/write of ha->last_known_committed_* and ha->term.
• The request/reply plumbing and a new opcode-less sender alongside xce_nop_req.

Revalidation Triggers

• Any change to HeartbeatMsg's committed fields or the heartbeatmsg body ordinal → re-check producer + consumer.
• Any change to the voter-floor formula or to who writes last_known_committed_* → re-check the term-guard and monotonicity argument (Security section).
• Any change making the HA heartbeat use a different opcode/body → re-check the consumer's body_which branch.
• Changing last_known_committed_config_tag's type → re-check all writers + the ctrlstatus reader.

Architecture

Existing Architecture Analysis

• HA heartbeat = bare OP_nop (pre-feature baseline, since replaced by this spec — see File Structure Plan): primary_heartbeat_fn (now ha.c:9665; line drifted from unrelated growth added by later specs elsewhere in the file) sent xce_nop_req(ep) with no body; the standby OP_nop handler (now cluster.c:427, same drift) replied a StateReport and reset liveness (ha_on_heartbeat_received). It read no request body.
• Unused scaffolding: HeartbeatMsg{senderPriority@0, term@1, committedLsn@2, committedConfigTag@3} and the heartbeatmsg @25 Request-body member exist but have zero .c producers/consumers. (Verified current: schema unchanged.)
• Committed knowledge at baseline: last_known_committed_lsn (_Atomic, now ha.h:183) written at join (cluster_apply_ctrlhello_reply — a store that ALSO truncates via ha_rejoin_reconcile) and at the election safe-commit path (ha.c:925 area, still current); last_known_committed_config_tag (plain pre-task-1.1, now ha.h:185) written only at join. Correction: this bullet originally implied a third writer of last_known_committed_config_tag at the election path — verified false; that path writes last_known_committed_lsn and the separate committed_config_tag field (Req 8.1, set on quorum ACK), not last_known_committed_config_tag.
• Monotonic precedent: commit.c advances committed_lsn with monotonic-max semantics — mirrored here.
• Vote floor already consumes it: max(durable_floor, last_known_committed_lsn), computed in ha_vote_self_freshness (now ha.c:5310; citation corrected — durable_floor is supplied by the voter-path caller, e.g. cluster.c:544).
• Opcode/body independence: OP_hello already carries different bodies (ctrlhellomsg vs replymsg), so OP_nop + heartbeatmsg body is consistent with the existing pattern.

Architecture Pattern & Boundary Map

Extend-in-place along the existing heartbeat path. Dependency direction (unchanged): schema → libxce (sender) → controller (primary tick + standby receive). No upward dependencies; no new opcode.

graph LR
    Tick[primary_heartbeat_fn tick] --> Sender[xce_ha_heartbeat_req nop plus HeartbeatMsg body]
    Sender --> Wire[OP_nop request with heartbeatmsg body]
    Wire --> Handler[cluster.c OP_nop handler]
    Handler --> Guard[term guard plus monotonic]
    Guard --> Observe[ha_observe_heartbeat_committed]
    Observe --> State[last_known_committed_lsn and config_tag]
    State --> Floor[vote freshness floor unchanged]
    State --> Status[ctrlstatus off primary committed]

Key decisions: the primary stamps committed onto the existing heartbeat (no new opcode, Option A); the standby's update is term-guarded and monotonic and reconcile-free; the vote floor reads the fresher value through the unchanged formula.

File Structure Plan

Modified Files

• src/include/request.h + src/lib/request.c — add xce_ha_heartbeat_req(endpoint_t* ep, uint64_t term, uint64_t committed_lsn, uint64_t committed_config_tag): sends an OPCode_nop request with a heartbeatmsg body carrying {term, committed_lsn, committed_config_tag}. Generic xce_nop_req is left unchanged.
• src/controller/ha.c — primary_heartbeat_fn: replace the xce_nop_req(ep) heartbeat call with xce_ha_heartbeat_req(ep, ha->term, persist.committed_lsn, ha->committed_config_tag) (the XCE_TESTING ha_nop_req_hook seam is preserved). Add ha_observe_heartbeat_committed. Migrate the election-path writer of last_known_committed_config_tag (ha.c:925 area) to atomic_store.
• src/controller/cluster.c — OP_nop handler: if request.body_which == Request_body_heartbeatmsg, apply the term-guard and call ha_observe_heartbeat_committed; otherwise behave exactly as today. Liveness reset + StateReport reply unchanged.
• src/controller/include/ha.h — change last_known_committed_config_tag to _Atomic uint64_t; declare ha_observe_heartbeat_committed.
• src/controller/ha_convergence.c — cluster_apply_ctrlhello_reply: write last_known_committed_config_tag via atomic_store (type migration only; semantics unchanged).
• Tests: src/controller/testsuite/test_ha_heartbeat_committed.c (new) for producer/consumer + monotonic + term-guard + advisory-only; registered in controller_test.c + controller/meson.build.

New Files

• src/controller/testsuite/test_ha_heartbeat_committed.c — unit tests for the consumer/updater. (No new product files — HeartbeatMsg C types are generated; logic slots into existing TUs.)

System Flows

Heartbeat-borne committed propagation + term-guard

sequenceDiagram
    participant P as Primary
    participant S as Standby
    P->>P: read own committed_lsn and committed_config_tag and term
    P->>S: OP_nop with HeartbeatMsg term committed_lsn config_tag
    S->>S: reset liveness and reply StateReport as before
    alt heartbeat term is at least standby term and committed greater than known
        S->>S: monotonic advance last_known_committed_lsn and config_tag
    else stale term or not greater or no body
        S->>S: liveness only no committed update
    end

Gating: the standby updates committed only when the heartbeat's advertised term is >= ha->term (term-guard) AND the carried LSN exceeds the current last_known_committed_lsn (monotonic). Otherwise the heartbeat is processed for liveness exactly as today. The update never reconciles or truncates the log.

Requirements Traceability

Requirement	Summary	Components	Interfaces	Flows
1.1	Primary advertises committed on each heartbeat	xce_ha_heartbeat_req, primary_heartbeat_fn	HeartbeatMsg{term, committedLsn, committedConfigTag} body on OP_nop	Propagation
1.2	Standby consumes it	cluster.c OP_nop handler, ha_observe_heartbeat_committed	body.heartbeatmsg read	Propagation
1.3	Fresh within one heartbeat interval	per-tick send + receive	heartbeat cadence	Propagation
2.1	Advance on higher carried LSN	ha_observe_heartbeat_committed	monotonic-max store	Propagation (alt)
2.2	Leave unchanged on ≤ (stale/out-of-order/lagging)	ha_observe_heartbeat_committed	monotonic guard	Propagation (else)
2.3	Config tag advances with LSN, never independently regressed	ha_observe_heartbeat_committed	paired update under the same advance	Propagation
3.1	No truncation from heartbeat-borne committed	ha_observe_heartbeat_committed (reconcile-free)	no ha_rejoin_reconcile call	—
3.2	No weakening of the vote-freshness floor	vote floor unchanged + term-guard	max(durable_floor, last_known_committed_lsn) (unchanged)	Propagation (term-guard)
3.3	No change to committed production/replication/persist	consume path	read-only on produced committed	—
3.4	Primary uses its own committed	consume gated to non-primary	inbound heartbeats only on standbys	Propagation
4.1	Existing liveness behavior unchanged	OP_nop handler	ha_on_heartbeat_received + StateReport untouched	Propagation
4.2	Absent/zero committed → plain liveness, no-op	OP_nop handler, ha_observe_heartbeat_committed	body-absent / term-fail / zero → no update	Propagation (else)

Components and Interfaces

Component	Layer	Intent	Req Coverage	Key Dependencies	Contracts
xce_ha_heartbeat_req	libxce	Send OP_nop + HeartbeatMsg body	1.1	request plumbing, HeartbeatMsg codegen (P0)	Service
primary_heartbeat_fn call-site	Controller HA	Stamp own committed/term per tick	1.1, 3.4	persist.committed_lsn, ha->committed_config_tag, ha->term (P0)	Service
OP_nop handler consume branch	Controller	Read body, term-guard, dispatch update	1.2, 4.1, 4.2	ha_observe_heartbeat_committed (P0)	Service
ha_observe_heartbeat_committed	Controller HA	Reconcile-free monotonic update of last_known_committed_*	2.x, 3.1	ha->last_known_committed_* atomics (P0)	Service, State
last_known_committed_config_tag → _Atomic	Controller HA	Safe concurrent read/write	2.3	existing writers (P1)	State

libxce — HA heartbeat sender (Service)

/* Send the HA heartbeat as OP_nop carrying a HeartbeatMsg body. Clone of the
 * nop send path + a heartbeatmsg body. Non-blocking, no reply awaited (the
 * standby's StateReport rides the normal nop reply). Generic xce_nop_req is
 * left unchanged. */
int xce_ha_heartbeat_req(endpoint_t* ep, uint64_t term,
                         uint64_t committed_lsn, uint64_t committed_config_tag);

• Preconditions: valid endpoint.
• Postconditions: an OP_nop request with body_which == Request_body_heartbeatmsg is enqueued; HeartbeatMsg.term/committedLsn/committedConfigTag set; senderPriority unused (0).
• Invariants: non-blocking; carries no other state.

Controller — ha_observe_heartbeat_committed (Service / State)

Field	Detail
Intent	Monotonically advance a standby's learned committed position from a heartbeat, with no reconciliation
Requirements	2.1, 2.2, 2.3, 3.1

Responsibilities & Constraints

• Advance last_known_committed_lsn to the carried LSN only if it is strictly greater (monotonic; single receiver thread, so a relaxed compare-then-atomic_store/acquire-release or atomic_fetch_max suffices).
• Advance last_known_committed_config_tag together with the LSN (only when the LSN advances); never regress it independently (2.3).
• Must NOT call ha_rejoin_reconcile, truncate the WAL, or modify any produced/persisted committed state (3.1, 3.3). This is the explicit distinction from cluster_apply_ctrlhello_reply.
• No-op when the carried LSN is <= current or zero (2.2, 4.2).

Dependencies

• Inbound: cluster.c OP_nop handler — passes the term-checked payload (P0).
• Outbound: ha->last_known_committed_lsn / last_known_committed_config_tag atomics (P0).

Controller — OP_nop handler consume branch (Service)

• After the existing liveness/StateReport handling, if request.body_which == Request_body_heartbeatmsg and the responder is non-primary: apply the term-guard (hb->term >= atomic_load(ha->term)) and, if it passes, call ha_observe_heartbeat_committed(c, hb->committed_lsn, hb->committed_config_tag).
• If the body is absent, the term-guard fails, or committed is zero → no committed update; liveness handling is unchanged (4.1, 4.2).
• Constraint: this branch must not alter the reply, the liveness reset, or any existing OP_nop behavior.

Data Models

No new persistent data and no schema-shape change. HeartbeatMsg (existing) is the request-body DTO; its term, committedLsn, committedConfigTag fields are populated by the producer and read by the consumer. The only in-memory state touched is ha->last_known_committed_lsn (already _Atomic) and ha->last_known_committed_config_tag (migrated to _Atomic uint64_t so the heartbeat-receive writer, the join writer, the election writer, and the ctrlstatus reader interact safely). Both are advisory replicas of the primary's committed markers — never authoritative, never persisted by this path.

Error Handling

• Malformed/absent body: body_which != Request_body_heartbeatmsg → treat as a plain liveness heartbeat; no error (4.2).
• Stale-term heartbeat: hb->term < ha->term → ignore the committed payload (term-guard); liveness handled normally. This is the safety gate, not an error.
• Out-of-order / lower committed: monotonic no-op (2.2); not an error.
• Send failure: xce_ha_heartbeat_req returns the same negative as the nop path; primary_heartbeat_fn logs WARN exactly as today — no behavior change (4.1).
• Observability: keep heartbeat-committed updates at a low/debug log level (the heartbeat path is hot); do not add per-tick INFO spam.

Testing Strategy

Unit (test_ha_heartbeat_committed.c)

• A standby with last_known_committed_lsn = 4 receiving a heartbeat (term >= ha->term) carrying committed_lsn = 9 → last_known_committed_lsn == 9 and last_known_committed_config_tag updated (1.2, 2.1, 2.3).
• A heartbeat carrying committed_lsn = 3 (≤ current 9) → last_known_committed_lsn unchanged (2.2).
• A stale-term heartbeat (hb->term < ha->term) carrying a higher committed → last_known_committed_* unchanged (3.2 term-guard); liveness still reset (4.1).
• A heartbeat with no heartbeatmsg body / zero committed → no committed update, processed as liveness (4.2).
• ha_observe_heartbeat_committed performs no WAL truncation / no reconcile call and mutates nothing beyond last_known_committed_* (3.1, 3.3) — assert via a no-truncate / state-snapshot check.

Integration

• Producer round-trip: xce_ha_heartbeat_req encodes an OP_nop whose decoded request has body_which == heartbeatmsg with the expected term/committedLsn/committedConfigTag (1.1).
• Vote-floor interaction: with a heartbeat-raised last_known_committed_lsn, the existing freshness gate rejects a candidate below it and accepts one at/above it — confirming the floor consumes the fresher value through the unchanged formula and is never lowered (3.2).

E2E (live /xce-deploy-ha)

• On a healthy 3-node cluster, submit jobs so the primary advances committed; ctrlstatus on a standby shows committedLsn tracking the primary within ~a heartbeat interval (no re-join), closing the controller-status Req 10 staleness caveat (1.3, downstream).

Security Considerations

The only safety-relevant change is feeding a heartbeat-fresh last_known_committed_lsn into the (unchanged) vote-freshness floor max(durable_floor, last_known_committed_lsn).

• Why raising the floor is safe (Leader Completeness): committed entries are, by definition, durable on a majority and cluster-wide monotonic (State Machine Safety). A heartbeat-borne committed accepted under the term-guard (hb->term >= ha->term) therefore reflects a genuinely committed position. Requiring vote candidates to be at least that fresh strengthens the guarantee that a new leader holds all committed entries; it never weakens it (3.2). The value only ever raises the floor (monotonic), so the floor cannot drop.
• No liveness wedge: since a majority holds the committed entries, an eligible candidate (lastLogLsn >= floor) always exists; a node behind the floor correctly fails its own freshness check and should not lead.
• Stale/forged-term defense: the term-guard rejects committed advertised by a primary whose term is behind the standby's, preventing an old/partitioned primary from inflating the floor. The value is otherwise advisory: it never drives truncation (3.1) and never changes how committed is produced (3.3).
• No new authorization surface: the heartbeat is the existing authenticated primary→standby channel; this only adds three integer fields to a message that already flows.