myxce/.kiro/specs/ha-heartbeat-committed-propagation/research.md

Research / Gap Analysis — ha-heartbeat-committed-propagation

Generated by /kiro-validate-gap (2026-06-20). Brownfield; informs design. Requirements: requirements.md (Req 1–4). Brief: brief.md.

Current State (verified)

• The HA primary→standby heartbeat is a bare OP_nop with no body. primary_heartbeat_fn sends xce_nop_req(ep) (ha.c:5256); the standby OP_nop handler (cluster.c:300) replies a StateReport (ha_nop_reply_state_report) and resets liveness (ha_on_heartbeat_received) — it reads no request body and learns no committed value.
• OPCode_heartbeat is a SEPARATE, client-side keepalive (xce_heartbeat(ctx, hb_interval), xceapi.c:48; client.c:233; dispatch-gate ha.c:2077/2120/2161) — NOT the HA heartbeat. Do not repurpose it (collision).
• The wire already supports a committed-bearing heartbeat, but nothing uses it: the Request body union has heartbeatmsg @25: HeartbeatMsg (controller.capnp:653), and HeartbeatMsg{senderPriority@0, term@1, committedLsn@2, committedConfigTag@3} (controller.capnp:761) is fully defined. Grep finds zero .c producers or consumers of heartbeat_msg_t/heartbeatmsg/its committedLsn — dead scaffolding.
• Standby committed knowledge today: ha->last_known_committed_lsn (_Atomic uint64_t, ha.h:170) + ha->last_known_committed_config_tag (plain uint64_t, ha.h:172) are written only at join via cluster_apply_ctrlhello_reply (ha_convergence.c:372) — a plain atomic_store (NOT monotonic) that ALSO calls ha_rejoin_reconcile (truncation) — and at an election safe-commit path (ha.c:925).
• A monotonic-max precedent exists for committed advance in commit.c (monotonic max() store, commit.c:141/151/155) — mirror it for the heartbeat update.
• last_known_committed_lsn already feeds the vote-freshness floor: voter floor = max(durable_floor, last_known_committed_lsn) (ha.c:2570–2586; established by bug-339 / project_ha_vote_freshness_invariant). This is the key safety interaction (below).
• xce_nop_req(ep) is generic (also the client xce_nop, xceapi.c:42) — must not gain a mandatory committed body.

Requirement-to-Asset Map

Need (Req)	Existing asset	Gap
1.1 primary advertises committed on each heartbeat	primary_heartbeat_fn sends bare xce_nop_req; HeartbeatMsg{committedLsn,committedConfigTag} + heartbeatmsg body union member exist	Missing producer: a HA-heartbeat sender that attaches a HeartbeatMsg body carrying committed_lsn/committed_config_tag
1.2 standby consumes it	cluster.c:300 OP_nop handler (reads no body); request.body_which/body union already supports heartbeatmsg	Missing consumer: read body.heartbeatmsg and update last_known_committed_*
1.3 fresh within one heartbeat interval	heartbeat tick cadence (heartbeat_interval)	property of 1.1+1.2; testable via tick
2.1–2.3 monotonic, never regress (LSN + config tag together)	commit.c monotonic-max pattern; last_known_committed_lsn is _Atomic (single receiver thread)	Constraint: existing cluster_apply_ctrlhello_reply does a PLAIN store — the heartbeat path needs a monotonic update; config tag must move with the LSN
3.1 no truncation from heartbeat-borne committed	cluster_apply_ctrlhello_reply calls ha_rejoin_reconcile (truncation)	Critical: the heartbeat consume path must NOT reconcile/truncate — needs a separate, reconcile-free updater (do NOT reuse cluster_apply_ctrlhello_reply)
3.2 must not weaken the vote-freshness floor	voter floor max(durable_floor, last_known_committed_lsn) (ha.c:2570)	Research Needed (highest): a heartbeat-fresh (higher) last_known RAISES the floor — verify that is safe (Leader Completeness) and cannot deadlock liveness (a standby refusing all candidates)
3.3 no change to committed production/replication/persist	consume path is read+store only	none — keep it read-only w.r.t. produced committed
3.4 primary uses its own committed	only standbys process inbound HA heartbeats; primary sends	inherently satisfied; gate the consume to non-primary for clarity
4.1 existing liveness unchanged	OP_nop still resets liveness + replies StateReport	additive body-read only; verify no timing change
4.2 absent/zero committed → plain liveness, no-op	monotonic update naturally no-ops on 0 / ≤current	handle body_which != heartbeatmsg and zero value as liveness-only

Implementation Approach Options

Option A (Extend the OP_nop HA heartbeat) — RECOMMENDED. Add a dedicated HA-heartbeat sender (e.g. xce_ha_heartbeat_req(ep, committed_lsn, committed_config_tag)) that sends OPCode_nop with a heartbeatmsg body; the cluster.c OP_nop handler reads it and monotonically updates last_known_committed_* via a new reconcile-free helper. Reuses the already-defined HeartbeatMsg + body union member; leaves generic xce_nop_req (client nop) untouched. Trade-off: ✅ minimal, reuses dead scaffolding, no new opcode. ❌ touches the hot HA heartbeat path; an OP_nop carrying a heartbeatmsg@25 body is slightly unconventional (a request may carry any body-union member regardless of opcode — design confirms the pairing).

Option B (Dedicated HA-heartbeat opcode). New opcode (distinct from the client OPCode_heartbeat) paired with HeartbeatMsg, plus dispatch-gate + handler wiring. Cleaner semantic separation. Trade-off: ✅ unambiguous. ❌ more churn (opcode, gate, handler, sender swap) for the same effect.

Option C (Repurpose OPCode_heartbeat) — REJECT. It is the client keepalive (xce_heartbeat/hb_interval); reusing it collides with existing client semantics.

Effort & Risk

• Producer (Req 1.1): S — one sender + call-site swap in primary_heartbeat_fn.
• Consumer + monotonic update (Req 1.2, 2): S — one handler branch + a reconcile-free monotonic helper.
• Vote-freshness safety review (Req 3.2): M — analysis, not much code, but the gating risk.
• Combined: S–M (3–5 days). Risk Medium — not the propagation (mechanical) but two interactions: (a) the hot heartbeat path, (b) feeding a fresher last_known_committed_lsn into the vote floor.

Research Items to Carry to Design

1. Vote-freshness safety (do this first). Confirm against raft.tla Leader Completeness that raising last_known_committed_lsn from a heartbeat-advertised committed (which the standby has not necessarily applied/durably stored) is safe to feed the voter floor max(durable_floor, last_known_committed_lsn), and that it cannot wedge liveness (standby refusing otherwise-valid candidates). If risky, gate: only let the heartbeat raise last_known up to what the standby can corroborate, or keep the floor on durable state and let only reporting consume the heartbeat value.
2. Reconcile-free consume path. Do NOT reuse cluster_apply_ctrlhello_reply (it truncates). Design a monotonic-only ha_observe_heartbeat_committed; decide whether to factor a shared monotonic core used by both join and heartbeat (join adds reconcile, heartbeat does not).
3. last_known_committed_config_tag atomicity. It is non-atomic (ha.h:172); with a new heartbeat-receive writer plus the ctrlstatus reader, decide whether to promote to _Atomic (and update the 2–3 existing write sites) — or keep best-effort.
4. Sender scoping. Add a HA-specific sender; do NOT alter generic xce_nop_req.
5. Opcode/body pairing. Confirm OP_nop + heartbeatmsg body is accepted end-to-end by the codec/dispatch, or choose Option B's dedicated opcode.

---

Synthesis / Design Decisions (2026-06-20)

Resolutions of the 5 research items, locked into design.md:

1. Vote-freshness safety — RESOLVED, no vote-path change needed. The voter floor formula max(durable_floor, last_known_committed_lsn) stays UNCHANGED; it already reads last_known_committed_lsn. Making that value heartbeat-fresh only RAISES it (monotonic), which strengthens the freshness gate (Leader Completeness) rather than weakening it (Req 3.2). Safety rests on three facts: (a) term-guard — a standby accepts the heartbeat-borne committed only from a primary whose advertised term >= ha->term (HeartbeatMsg.term@1), so a stale-term primary cannot inflate the floor; (b) monotonic — never regress (Req 2); (c) cluster-wide committed_lsn is itself monotonic and majority-durable (State Machine Safety), so any accepted value is genuinely committed. No liveness wedge: committed entries live on a majority, so an eligible candidate (≥ floor) always exists. A node behind the floor correctly fails its own freshness check and should not lead. Conclusion: feed the existing floor unchanged; the guard is on the accept, not the use.
2. Reconcile-free consume — RESOLVED. New ha_observe_heartbeat_committed(c, lsn, tag): monotonic-max update only; does NOT call ha_rejoin_reconcile/truncate (Req 3.1). Distinct from cluster_apply_ctrlhello_reply (which truncates at join).
3. config_tag atomicity — RESOLVED. Promote last_known_committed_config_tag to _Atomic uint64_t (ha.h:172); update the 2 existing writers (ha_convergence.c:377, ha.c election path) to atomic_store. New heartbeat writer + the ctrlstatus reader justify it.
4. Sender scoping — RESOLVED. New xce_ha_heartbeat_req(ep, term, committed_lsn, committed_config_tag); generic xce_nop_req untouched (client nop unaffected).
5. Opcode/body pairing — RESOLVED. Send OPCode_nop WITH a heartbeatmsg body (the Request union already has heartbeatmsg@25; opcode and body are independent, as OP_hello already carries different bodies). Dispatch still routes by OPCode_nop; the handler branches on body_which == Request_body_heartbeatmsg. No new opcode (Option A over B).