myxce/.kiro/specs/ha-heartbeat-committed-propagation/tasks.md

Implementation Plan

TDD is mandatory (project rule): add the failing C test in testsuite/ (or a piped xce_test behavioral script) before the change — red → green → refactor. The work runs along a single request path (atomic migration → libxce sender → primary stamp / standby consume → vote-floor verification → E2E), so tasks are sequential by design; no (P) markers. The two foundation tasks (1.1 atomic migration, 1.2 sender) touch independent files but are kept ordered for review simplicity. The HA-critical change is the standby consume + term-guard (task 2.2) and its vote-floor interaction (task 3.2).

• 1. Foundation: atomic state + heartbeat sender

• 1.1 Migrate last_known_committed_config_tag to _Atomic

  • Change ha->last_known_committed_config_tag from a plain uint64_t to _Atomic uint64_t (src/controller/include/ha.h) and update its existing writers — cluster_apply_ctrlhello_reply (ha_convergence.c) and the election safe-commit path (ha.c) — to atomic_store_explicit(..., release), and any reader (incl. the controller-status ctrlstatus_fill) to atomic_load_explicit(..., acquire). Pure type/access migration — no behavior change.
  • Observable: the controller suite builds clean and passes unchanged; every read/write of the field uses atomic accessors (no plain access remains).
  • Requirements: 2.3
  • Boundary: ha.h + last_known_committed_config_tag accessors

• 1.2 Add the HA heartbeat sender xce_ha_heartbeat_req

  • Add xce_ha_heartbeat_req(endpoint_t* ep, uint64_t term, uint64_t committed_lsn, uint64_t committed_config_tag) (src/include/request.h + src/lib/request.c) that sends an OPCode_nop request carrying a heartbeatmsg body (HeartbeatMsg{term, committedLsn, committedConfigTag}; senderPriority unused). Leave the generic xce_nop_req untouched.
  • TDD: a round-trip test encodes via the new sender and decodes a request with body_which == Request_body_heartbeatmsg carrying the three field values; a guard assertion confirms xce_nop_req still sends a bodyless nop.
  • Observable: the round-trip test decodes the three fields from the new sender, and the bodyless-xce_nop_req guard still holds.
  • Requirements: 1.1
  • Boundary: libxce sender

• 2. Standby consume (reconcile-free, term-guarded, monotonic)

• 2.1 Add the reconcile-free monotonic updater ha_observe_heartbeat_committed

  • Add ha_observe_heartbeat_committed(cluster_t* c, uint64_t committed_lsn, uint64_t committed_config_tag) (ha.c, declared in ha.h): advance last_known_committed_lsn only when the carried LSN is strictly greater (monotonic; atomic), advancing last_known_committed_config_tag together with it; no-op when <= current or zero. It must NOT call ha_rejoin_reconcile, truncate the WAL, or modify any produced/persisted committed state.
  • TDD: unit tests assert advance-on-higher (LSN + config tag updated), no-op on equal/lower/zero, config tag never regresses independently, and — critically — that the call performs no WAL truncation / no reconcile and mutates nothing beyond last_known_committed_*.
  • Observable: the unit tests go red → green; the no-truncation / no-mutation assertions hold.
  • Requirements: 2.1, 2.2, 2.3, 3.1, 3.3
  • Boundary: ha_observe_heartbeat_committed
  • Depends: 1.1

• 2.2 Wire the standby OP_nop consume branch + term-guard

  • In the cluster.c OP_nop handler, after the existing liveness reset + StateReport reply, if request.body_which == Request_body_heartbeatmsg and the responder is non-primary: apply the term-guard (hb->term >= atomic_load(ha->term)) and, only if it passes, call ha_observe_heartbeat_committed(c, hb->committed_lsn, hb->committed_config_tag). A missing body, a failed term-guard, or a zero committed value updates nothing and leaves all existing OP_nop behavior intact.
  • TDD: tests confirm a term-valid higher-committed heartbeat advances last_known_committed_*; a stale-term heartbeat (hb->term < ha->term) carrying a higher committed leaves committed unchanged but STILL resets liveness; a bodyless / zero-committed nop is processed as liveness only; and a primary responder does not consume an inbound committed value.
  • Observable: the tests pass for the term-valid, stale-term, bodyless, and primary-responder cases, with liveness reset preserved in every case.
  • Requirements: 1.2, 3.2, 3.4, 4.1, 4.2
  • Boundary: cluster.c OP_nop handler
  • Depends: 2.1

• 3. Primary produce + safety verification

• 3.1 Stamp committed + term on the primary heartbeat

  • In primary_heartbeat_fn (ha.c), replace the per-standby xce_nop_req(ep) heartbeat call with xce_ha_heartbeat_req(ep, atomic_load(ha->term), atomic_load(persist.committed_lsn), atomic_load(ha->committed_config_tag)). Preserve the XCE_TESTING ha_nop_req_hook seam and the existing heartbeat send-failure WARN and last_push_time bookkeeping.
  • TDD/observable: a test (or the ha_nop_req_hook seam) confirms the primary's heartbeat now carries its current committed LSN / committed config tag / term; the send-failure and liveness-eviction behavior of primary_heartbeat_fn is unchanged.
  • Requirements: 1.1, 3.4
  • Boundary: primary_heartbeat_fn
  • Depends: 1.2

• 3.2 Verify the vote-freshness floor consumes the fresher value safely

  • Add an integration test proving the existing freshness gate, with last_known_committed_lsn raised by a heartbeat (via ha_observe_heartbeat_committed), rejects a vote candidate whose last-log LSN is below the raised floor and accepts one at/above it — confirming the floor formula max(durable_floor, last_known_committed_lsn) is read unchanged and is never lowered by this feature. No production change to the vote path.
  • Observable: the test shows the candidate-below-floor rejected and candidate-at/above-floor accepted after a heartbeat raise, and no edit to the floor formula.
  • Requirements: 3.2
  • Boundary: vote-freshness interaction test
  • Depends: 2.1, 2.2

• 4. Validation (live cluster)

• 4.1 End-to-end: standby committed tracks the primary within a heartbeat interval

  • On a live 3-node /xce-deploy-ha cluster, submit jobs so the primary advances its committed position, then query a STANDBY repeatedly: its ctrlstatus committedLsn advances to track the primary within roughly one heartbeat interval, without any re-join — closing the controller-status Req 10 staleness caveat. Confirm existing liveness/failover behavior is unaffected.
  • Observable: the live run shows a standby's committedLsn rising to track the primary between joins (not frozen at the join value), and the cluster's liveness/failover behavior is unchanged; manual environment prerequisite (live cluster, real fs).
  • Requirements: 1.2, 1.3
  • Depends: 3.1, 2.2

Implementation Notes

• 4.1 VERIFIED 2026-07-02 (previously HAND-OFF, blocked on overlayfs — unblocked by ha-commit-durability 6.1 confirming ~/tmp is a genuine ext4 fs in this sandbox): deployed a 3-node cluster to ~/tmp/xce-ctrlstatus-deploy, queried standby ctrl-2's ctrlstatus before/after submitting 5 more jobs (4s wait, inside the 3000ms heartbeat interval): committedLsn rose 4 → 9, exactly matching the 5 new jobs, with NO re-join — confirms committed propagates to standbys via heartbeat, not just at join. Then killed the primary (ctrl-1) and confirmed liveness/failover unaffected: wait-primary succeeded, term advanced 3 → 5 (new primary elected), and committedLsn continued advancing post-failover (9 → 10). PASS on both halves of the Observable. Found and logged (unrelated, low-priority) during this session: bug-1400 (xce_test cmd_jobsub SEGVs on malformed CLI args) and bug-1401 (ctrlstatus reply decode leak in decode_CtrlStatus_ptr).
• The HA heartbeat is OPCode_nop with a heartbeatmsg body — opcode and body are independent in this schema (mirrors OP_hello carrying ctrlhellomsg vs replymsg). Dispatch routes by OPCode_nop (unchanged); the consumer branches on body_which.
• Do NOT reuse cluster_apply_ctrlhello_reply for the heartbeat path — it truncates (ha_rejoin_reconcile). The heartbeat updater is reconcile-free (Req 3.1).
• Do NOT touch the client keepalive OPCode_heartbeat / xce_heartbeat (a different message) or the generic xce_nop_req (client nop).
• 1.1 LEARNING: ha.h is src/controller/ha.h (NOT src/controller/include/ha.h). last_known_committed_config_tag is now _Atomic uint64_t (ha.h:172), sibling of last_known_committed_lsn (ha.h:170); read via atomic_load_explicit(acquire), write via atomic_store_explicit(release).
• TEST-RUN LEARNING (2.x/3.x): the meson test wrapper (xce-build.sh controller) is NON-DETERMINISTIC ("OK"/"FAIL"/"Timeout") because of ~149 PRE-EXISTING LeakSanitizer leaks from ha_start_election_async/ha_start_prevote_async (test_ha_start_election_async.c, test_ha_rdhup_integration.c) — unrelated to this spec (identical count on a stashed base). AUTHORITATIVE verification = run the binary directly: cd src && ASAN_OPTIONS=detect_leaks=0 ./xce.build/controller/controller_unit_test, then check rc=0 and zero :FAIL: markers.
• 3.1 LEARNING: primary stamps its OWN ha->term / c->persist.committed_lsn / ha->committed_config_tag (all _Atomic, acquire-load once per tick) — NOT the standby last_known_*. Test seam: new ha_heartbeat_req_hook(ep,term,lsn,tag) (XCE_TESTING, NULL default) with precedence new-hook → legacy ha_nop_req_hook → real xce_ha_heartbeat_req, so existing ha_nop_req_hook liveness tests stay green.
• 1.2 LEARNING: the C-side heartbeatmsg body support PRE-EXISTED (no typemap edit needed) — heartbeat_msg_t typedef typemap.h:17, xce_request_t.heartbeatmsg member typemap.h:287, struct heartbeat_msg_s{sender_priority,term,committed_lsn,committed_config_tag} typemap.h:445 (order matches schema @0..@3); enum Request_body_heartbeatmsg == 22 (rides OPCode_nop). The codec (encode/decode/free_HeartbeatMsg + the request arm) was generated; only producer/consumer were missing. Test round-trip pattern: heap-calloc packet consumed+freed by xce_decode (caller does NOT free pkt), decoded body freed by xce_free_request (see test_wal_wire.c). Sender uses xce_request(ep, &req) (non-blocking).
• 4.1 HAND-OFF (live-cluster manual validation — run on a real-fs deploy, NOT the overlayfs sandbox): on a /xce-deploy-ha 3-node cluster, (1) jobsub a steady trickle of jobs so the primary's committedLsn keeps advancing; (2) repeatedly ctrlstatus a STANDBY (any-node hint view) and watch its committedLsn — it should RISE to track the primary within ~one heartbeat interval (no re-join), instead of being frozen at the join value or 0 (the controller-status Req 10 staleness this spec removes); (3) confirm liveness/failover is unaffected (kill the primary, a standby still takes over normally). PASS = standby committedLsn tracks the live primary between joins, with no liveness regression. Rebuild first so the cluster has the producer (task 3.1) + consumer (task 2.2).
• The vote-floor formula is NOT edited — task 3.2 is a verification test only. Safety rests on the term-guard (task 2.2) + monotonicity (task 2.1) + cluster-monotonic committed; see design.md Security Considerations.