# Implementation Plan

> TDD is mandatory (project rule): add the failing C test in `testsuite/` (or a piped `xce_test` behavioral script) before the change — red → green → refactor.
> The work runs along a single request path (atomic migration → libxce sender → primary stamp / standby consume → vote-floor verification → E2E), so tasks are sequential by design; no `(P)` markers. The two foundation tasks (1.1 atomic migration, 1.2 sender) touch independent files but are kept ordered for review simplicity. The HA-critical change is the standby consume + term-guard (task 2.2) and its vote-floor interaction (task 3.2).

- [ ] 1. Foundation: atomic state + heartbeat sender
- [x] 1.1 Migrate `last_known_committed_config_tag` to `_Atomic`
  - Change `ha->last_known_committed_config_tag` from a plain `uint64_t` to `_Atomic uint64_t` (`src/controller/include/ha.h`) and update its existing writers — `cluster_apply_ctrlhello_reply` (`ha_convergence.c`) and the election safe-commit path (`ha.c`) — to `atomic_store_explicit(..., release)`, and any reader (incl. the controller-status `ctrlstatus_fill`) to `atomic_load_explicit(..., acquire)`. Pure type/access migration — no behavior change.
  - Observable: the controller suite builds clean and passes unchanged; every read/write of the field uses atomic accessors (no plain access remains).
  - _Requirements: 2.3_
  - _Boundary: ha.h + last_known_committed_config_tag accessors_
- [x] 1.2 Add the HA heartbeat sender `xce_ha_heartbeat_req`
  - Add `xce_ha_heartbeat_req(endpoint_t* ep, uint64_t term, uint64_t committed_lsn, uint64_t committed_config_tag)` (`src/include/request.h` + `src/lib/request.c`) that sends an `OPCode_nop` request carrying a `heartbeatmsg` body (`HeartbeatMsg{term, committedLsn, committedConfigTag}`; `senderPriority` unused). Leave the generic `xce_nop_req` untouched.
  - TDD: a round-trip test encodes via the new sender and decodes a request with `body_which == Request_body_heartbeatmsg` carrying the three field values; a guard assertion confirms `xce_nop_req` still sends a bodyless nop.
  - Observable: the round-trip test decodes the three fields from the new sender, and the bodyless-`xce_nop_req` guard still holds.
  - _Requirements: 1.1_
  - _Boundary: libxce sender_

- [ ] 2. Standby consume (reconcile-free, term-guarded, monotonic)
- [x] 2.1 Add the reconcile-free monotonic updater `ha_observe_heartbeat_committed`
  - Add `ha_observe_heartbeat_committed(cluster_t* c, uint64_t committed_lsn, uint64_t committed_config_tag)` (`ha.c`, declared in `ha.h`): advance `last_known_committed_lsn` only when the carried LSN is strictly greater (monotonic; atomic), advancing `last_known_committed_config_tag` together with it; no-op when `<=` current or zero. It must NOT call `ha_rejoin_reconcile`, truncate the WAL, or modify any produced/persisted committed state.
  - TDD: unit tests assert advance-on-higher (LSN + config tag updated), no-op on equal/lower/zero, config tag never regresses independently, and — critically — that the call performs no WAL truncation / no reconcile and mutates nothing beyond `last_known_committed_*`.
  - Observable: the unit tests go red → green; the no-truncation / no-mutation assertions hold.
  - _Requirements: 2.1, 2.2, 2.3, 3.1, 3.3_
  - _Boundary: ha_observe_heartbeat_committed_
  - _Depends: 1.1_
- [x] 2.2 Wire the standby `OP_nop` consume branch + term-guard
  - In the `cluster.c` `OP_nop` handler, after the existing liveness reset + `StateReport` reply, if `request.body_which == Request_body_heartbeatmsg` and the responder is non-primary: apply the term-guard (`hb->term >= atomic_load(ha->term)`) and, only if it passes, call `ha_observe_heartbeat_committed(c, hb->committed_lsn, hb->committed_config_tag)`. A missing body, a failed term-guard, or a zero committed value updates nothing and leaves all existing `OP_nop` behavior intact.
  - TDD: tests confirm a term-valid higher-committed heartbeat advances `last_known_committed_*`; a stale-term heartbeat (`hb->term < ha->term`) carrying a higher committed leaves committed unchanged but STILL resets liveness; a bodyless / zero-committed nop is processed as liveness only; and a primary responder does not consume an inbound committed value.
  - Observable: the tests pass for the term-valid, stale-term, bodyless, and primary-responder cases, with liveness reset preserved in every case.
  - _Requirements: 1.2, 3.2, 3.4, 4.1, 4.2_
  - _Boundary: cluster.c OP_nop handler_
  - _Depends: 2.1_

- [ ] 3. Primary produce + safety verification
- [x] 3.1 Stamp committed + term on the primary heartbeat
  - In `primary_heartbeat_fn` (`ha.c`), replace the per-standby `xce_nop_req(ep)` heartbeat call with `xce_ha_heartbeat_req(ep, atomic_load(ha->term), atomic_load(persist.committed_lsn), atomic_load(ha->committed_config_tag))`. Preserve the `XCE_TESTING` `ha_nop_req_hook` seam and the existing heartbeat send-failure `WARN` and `last_push_time` bookkeeping.
  - TDD/observable: a test (or the `ha_nop_req_hook` seam) confirms the primary's heartbeat now carries its current committed LSN / committed config tag / term; the send-failure and liveness-eviction behavior of `primary_heartbeat_fn` is unchanged.
  - _Requirements: 1.1, 3.4_
  - _Boundary: primary_heartbeat_fn_
  - _Depends: 1.2_
- [x] 3.2 Verify the vote-freshness floor consumes the fresher value safely
  - Add an integration test proving the existing freshness gate, with `last_known_committed_lsn` raised by a heartbeat (via `ha_observe_heartbeat_committed`), rejects a vote candidate whose last-log LSN is below the raised floor and accepts one at/above it — confirming the floor formula `max(durable_floor, last_known_committed_lsn)` is read unchanged and is never lowered by this feature. No production change to the vote path.
  - Observable: the test shows the candidate-below-floor rejected and candidate-at/above-floor accepted after a heartbeat raise, and no edit to the floor formula.
  - _Requirements: 3.2_
  - _Boundary: vote-freshness interaction test_
  - _Depends: 2.1, 2.2_

- [x] 4. Validation (live cluster)  <!-- VERIFIED 2026-07-02 -->
- [x] 4.1 End-to-end: standby committed tracks the primary within a heartbeat interval  <!-- VERIFIED 2026-07-02 on a real-fs (~/tmp, ext4) 3-node deploy — see Implementation Notes -->
  - On a live 3-node `/xce-deploy-ha` cluster, submit jobs so the primary advances its committed position, then query a STANDBY repeatedly: its `ctrlstatus` `committedLsn` advances to track the primary within roughly one heartbeat interval, without any re-join — closing the controller-status Req 10 staleness caveat. Confirm existing liveness/failover behavior is unaffected.
  - Observable: the live run shows a standby's `committedLsn` rising to track the primary between joins (not frozen at the join value), and the cluster's liveness/failover behavior is unchanged; manual environment prerequisite (live cluster, real fs).
  - _Requirements: 1.2, 1.3_
  - _Depends: 3.1, 2.2_

## Implementation Notes
- 4.1 VERIFIED 2026-07-02 (previously HAND-OFF, blocked on overlayfs — unblocked by ha-commit-durability 6.1 confirming `~/tmp` is a genuine ext4 fs in this sandbox): deployed a 3-node cluster to `~/tmp/xce-ctrlstatus-deploy`, queried standby ctrl-2's `ctrlstatus` before/after submitting 5 more jobs (4s wait, inside the 3000ms heartbeat interval): `committedLsn` rose `4 → 9`, exactly matching the 5 new jobs, with NO re-join — confirms committed propagates to standbys via heartbeat, not just at join. Then killed the primary (ctrl-1) and confirmed liveness/failover unaffected: `wait-primary` succeeded, term advanced `3 → 5` (new primary elected), and `committedLsn` continued advancing post-failover (`9 → 10`). PASS on both halves of the Observable. Found and logged (unrelated, low-priority) during this session: bug-1400 (xce_test cmd_jobsub SEGVs on malformed CLI args) and bug-1401 (ctrlstatus reply decode leak in `decode_CtrlStatus_ptr`).
- The HA heartbeat is `OPCode_nop` with a `heartbeatmsg` body — opcode and body are independent in this schema (mirrors `OP_hello` carrying `ctrlhellomsg` vs `replymsg`). Dispatch routes by `OPCode_nop` (unchanged); the consumer branches on `body_which`.
- Do NOT reuse `cluster_apply_ctrlhello_reply` for the heartbeat path — it truncates (`ha_rejoin_reconcile`). The heartbeat updater is reconcile-free (Req 3.1).
- Do NOT touch the client keepalive `OPCode_heartbeat` / `xce_heartbeat` (a different message) or the generic `xce_nop_req` (client nop).
- 1.1 LEARNING: `ha.h` is `src/controller/ha.h` (NOT `src/controller/include/ha.h`). `last_known_committed_config_tag` is now `_Atomic uint64_t` (`ha.h:172`), sibling of `last_known_committed_lsn` (`ha.h:170`); read via `atomic_load_explicit(acquire)`, write via `atomic_store_explicit(release)`.
- TEST-RUN LEARNING (2.x/3.x): the meson test wrapper (`xce-build.sh controller`) is NON-DETERMINISTIC ("OK"/"FAIL"/"Timeout") because of ~149 PRE-EXISTING LeakSanitizer leaks from `ha_start_election_async`/`ha_start_prevote_async` (`test_ha_start_election_async.c`, `test_ha_rdhup_integration.c`) — unrelated to this spec (identical count on a stashed base). AUTHORITATIVE verification = run the binary directly: `cd src && ASAN_OPTIONS=detect_leaks=0 ./xce.build/controller/controller_unit_test`, then check rc=0 and zero `:FAIL:` markers.
- 3.1 LEARNING: primary stamps its OWN `ha->term` / `c->persist.committed_lsn` / `ha->committed_config_tag` (all `_Atomic`, acquire-load once per tick) — NOT the standby `last_known_*`. Test seam: new `ha_heartbeat_req_hook(ep,term,lsn,tag)` (`XCE_TESTING`, NULL default) with precedence new-hook → legacy `ha_nop_req_hook` → real `xce_ha_heartbeat_req`, so existing `ha_nop_req_hook` liveness tests stay green.
- 1.2 LEARNING: the C-side `heartbeatmsg` body support PRE-EXISTED (no typemap edit needed) — `heartbeat_msg_t` typedef `typemap.h:17`, `xce_request_t.heartbeatmsg` member `typemap.h:287`, `struct heartbeat_msg_s{sender_priority,term,committed_lsn,committed_config_tag}` `typemap.h:445` (order matches schema `@0..@3`); enum `Request_body_heartbeatmsg == 22` (rides `OPCode_nop`). The codec (`encode/decode/free_HeartbeatMsg` + the request arm) was generated; only producer/consumer were missing. Test round-trip pattern: heap-`calloc` packet consumed+freed by `xce_decode` (caller does NOT free pkt), decoded body freed by `xce_free_request` (see `test_wal_wire.c`). Sender uses `xce_request(ep, &req)` (non-blocking).
- 4.1 HAND-OFF (live-cluster manual validation — run on a real-fs deploy, NOT the overlayfs sandbox): on a `/xce-deploy-ha` 3-node cluster, (1) `jobsub` a steady trickle of jobs so the primary's committedLsn keeps advancing; (2) repeatedly `ctrlstatus` a STANDBY (any-node hint view) and watch its `committedLsn` — it should RISE to track the primary within ~one heartbeat interval (no re-join), instead of being frozen at the join value or 0 (the controller-status Req 10 staleness this spec removes); (3) confirm liveness/failover is unaffected (kill the primary, a standby still takes over normally). PASS = standby committedLsn tracks the live primary between joins, with no liveness regression. Rebuild first so the cluster has the producer (task 3.1) + consumer (task 2.2).
- The vote-floor formula is NOT edited — task 3.2 is a verification test only. Safety rests on the term-guard (task 2.2) + monotonicity (task 2.1) + cluster-monotonic committed; see design.md Security Considerations.