myxce/.kiro/specs/ha-heartbeat-committed-propagation/spec.json

{
  "feature_name": "ha-heartbeat-committed-propagation",
  "created_at": "2026-06-20T00:00:00Z",
  "updated_at": "2026-06-20T00:00:00Z",
  "language": "en",
  "phase": "implementation-complete",
  "approvals": {
    "requirements": {
      "generated": true,
      "approved": true
    },
    "design": {
      "generated": true,
      "approved": true
    },
    "tasks": {
      "generated": true,
      "approved": true
    }
  },
  "ready_for_implementation": true,
  "implementation": {
    "completed": true,
    "tasks_done": "1.1, 1.2, 2.1, 2.2, 3.1, 3.2, 4.1",
    "notes": "Code + safety verification implemented, reviewed, committed (f4327ce, 282a82f, 8bfa861, 90e7179, d534f83, 3b46562); controller + lib suites green; the vote-floor safety property (Req 3.2) is verified by an executable test against the real ha_vote_self_freshness gate. 4.1 (live E2E, Req 1.2/1.3) VERIFIED 2026-07-02 on a real-fs (~/tmp, ext4) 3-node deploy: standby committedLsn tracked the primary within one heartbeat interval with no re-join, and continued advancing correctly across a failover. See tasks.md Implementation Notes."
  },
  "validate_impl": {
    "decision": "GO",
    "validated_at": "2026-07-18T09:45:00Z",
    "note": "RETRY of a validation pass that crashed silently mid-live-soak ~4 hours earlier with no artifacts/logs (stale empty deploy dir /home/claude/tmp/xce-ha-heartbeat-committed-deploy found and removed at start of this pass; no evidence it was a product bug rather than an environment/harness loss). Prior state: all 11 tasks (1.1-4.1) were [x] in tasks.md, 4.1 already live-verified 2026-07-02, but phase was stuck at \"tasks-generated\" with no validate_impl record (same administrative-oversight pattern as ha-wal-replication / ha-entity-snapshots / ha-commit-durability). This pass ran immediately after two sibling specs: raft-drive-async (NO-GO, bug-1447 ASAN SEGV in netio.c SSL handshake) and ha-wal-replication (GO, clean 10/10 soak) -- extra weight was given to reproducing or ruling out bug-1447's signature in this spec's own live soak.\n\nMECHANICAL: Fresh ASAN build (nix develop + xce-build.sh controller asan), binaries confirmed with session-fresh mtimes (controller_unit_test, xce_test, xce_controller). Standalone controller_unit_test run directly with ASAN_OPTIONS=detect_leaks=0 (authoritative over meson's aggregate line per this project's convention): 972 Tests, 0 Failures, 0 Ignored, OK -- zero \":FAIL:\" occurrences, exit 0. All heartbeat-owned tests explicitly confirmed PASS: test_ha_heartbeat_committed.c's 8 tests (test_observe_advances_on_higher, test_observe_noop_on_equal_or_lower, test_observe_noop_on_zero, test_observe_no_reconcile_no_other_mutation, test_consume_term_valid_advances, test_consume_stale_term_ignored, test_consume_primary_does_not_consume, test_consume_null_or_zero_noop) and test_ha_vote_floor_heartbeat.c's 4 tests (test_heartbeat_raises_vote_floor_to_max, test_candidate_below_raised_floor_rejected, test_candidate_at_or_above_raised_floor_accepted, test_lower_heartbeat_never_lowers_floor). TBD/TODO grep on request.c/request.h/ha.c/ha.h/cluster.c/both test files: only pre-existing, already-flagged-unrelated hits (ha.c:1844 ha-term-safety stub; cluster.c group-building and config-generation TODOs that predate and are untouched by this feature). Secrets grep clean.\n\nINTEGRATION (cross-task wiring confirmed by direct code read against design.md's Architecture + File Structure Plan): producer -- primary_heartbeat_fn (ha.c) calls xce_ha_heartbeat_req(ep, ha->term, persist.committed_lsn, ha->committed_config_tag) (request.h:33 decl, request.c:1257 impl) instead of bare xce_nop_req, sending OP_nop with a heartbeatmsg body; consumer -- cluster.c's OP_nop handler (~line 300-375) checks request.body_which == Request_body_heartbeatmsg and calls ha_consume_heartbeat_committed(c, ...) (ha.c:8090-8170), which: (1) gates out CTRL_PRIMARY nodes (Req 3.4), (2) runs the term-guard via the raft library's real UpdateTerm path (zero-entry ha_consume_append_entries call, not a hand-rolled check) and rejects hb->term < myterm (Req 3.2), (3) calls the reconcile-free monotonic updater ha_observe_heartbeat_committed (ha.c:8033, ha.h:1327) which advances last_known_committed_lsn/_config_tag only on strictly-greater LSN with no WAL truncation and no ha_rejoin_reconcile call (Req 2.1/2.2/3.1/3.3/4.2), and (4) calls ha_standby_advance_committed_and_apply -- a later hardening addition (commit 40f510fb, 2026-07-11, bug-1433/1434 fix) that drives this node's own committed_lsn + apply pump from the heartbeat-learned position; this call is additive to the design's Component boundary, not a violation (design.md's Allowed Dependencies already cover read/write of last_known_committed_* and ha->term). Both _Atomic migration writers confirmed: ha.c:8048 (election path) and ha_convergence.c:368 (cluster_apply_ctrlhello_reply join path), both atomic_store_explicit. ha.h declares both new functions (ha_observe_heartbeat_committed:1327, ha_consume_heartbeat_committed:1346) plus the _Atomic field (ha.h:185). Vote-floor formula (ha.c:2570, max(durable_floor, last_known_committed_lsn)) confirmed unchanged -- this spec only makes its input fresher, per design.md's explicit Out-of-Boundary note. All 6 implementation commits (f4327ce, 282a82f, 8bfa861, 90e7179, d534f83, 3b46562) plus the later hardening commit (40f510fb) confirmed present in git history.\n\nCOVERAGE: all 4 requirement sections map to completed tasks with no gaps -- Req 1 (producer) -> tasks 1.2, 3.1; Req 2 (monotonicity) -> task 2.1; Req 3 (advisory-only/term-guard/no-reconcile) -> tasks 2.1, 2.2, 3.2; Req 4 (liveness/no-regression) -> tasks 2.1, 2.2, 4.1. File Structure Plan MINOR drift (non-blocking): design.md's New Files section only names test_ha_heartbeat_committed.c; the vote-floor safety integration test landed as a second file, test_ha_vote_floor_heartbeat.c (both registered in controller_test.c + controller/meson.build) -- functionally equivalent/additive coverage of the same Req 3.2 property design.md calls for, same category of drift already accepted on the ha-wal-replication sibling. design.md's ha.h path is written as src/controller/include/ha.h; actual file is src/controller/ha.h (doc-only path drift, already flagged in tasks.md's own Implementation Notes as a non-blocking learning, not re-litigated here).\n\nRUNTIME LIVENESS (the blocking check, given the bug-1447 sibling finding and this being a retry of a crashed prior attempt): ran the canonical /xce-test-ha SEQUENTIAL 10-round soak (fresh session binaries) against a real ext4 deploy dir (~/tmp/xce-ha-heartbeat-committed-soak, confirmed ext4 via df -T before the run, cleaned up after). Result: PASS -- 10/10 rounds, 33 jobs submitted, zero committed-job loss, all 3 nodes (ctrl-1, ctrl-2, ctrl-3) served as primary at least once (leadership-coverage maneuver satisfied: ctrl-1 rounds 1-4/6-9, ctrl-2 rounds 1-4/7-10, ctrl-3 rounds 5/6/10). Explicitly grepped the full soak log and all per-node logs for bug-1447's exact signature (AddressSanitizer, DEADLYSIGNAL, SEGV, ssl3_read_bytes, handle_ssl_handshake, io_worker): zero matches across all six markers. Also grepped broadly for any other crash/abort signature (Segmentation fault, core dumped, Aborted, double free, heap-buffer-overflow, use-after-free): zero matches. This does not prove bug-1447 is fixed or absent globally (it is a rare, timing-dependent race in a different subsystem, raft-drive-async's reconnect-offload path, which this spec's boundary does not touch) -- but it does establish that this spec's own heartbeat-borne committed-propagation path runs clean through a full supported failover lifecycle, satisfying this spec's own regression contract (Req 4).\n\nOWNERSHIP: bug-1447 remains raft-drive-async/netio.c territory (see that spec's own NO-GO record); noted here only as a risk actively checked for and ruled out in this run, not adopted as this spec's blocker. No new bugs found or logged during this validation pass; the prior silent crash (~4 hours before this retry) left no diagnosable artifacts and could not be attributed to a specific cause -- treated as an environment/harness loss, not a product regression, since this clean retry reproduced nothing.",
    "known_limitations": [
      "bug-1447 (open, NOT-FIXED, owned by raft-drive-async / src/lib/netio.c, not this spec): a timing-dependent ASAN SEGV in handle_ssl_handshake/io_worker triggered by raft-drive-async's own reconnect-offload cadence. Did not reproduce in this spec's clean 10/10 run, but as a shared-codebase race condition it cannot be ruled out for all future runs by one clean pass. No action item for ha-heartbeat-committed-propagation; tracked entirely under raft-drive-async's remediation.",
      "The prior validation attempt (~4 hours before this retry) crashed silently mid-soak with no logs or diagnosable artifacts (only an empty stale deploy-dir marker was found). Could not be root-caused or attributed to a product bug; this retry's clean, fully-instrumented 10/10 soak found nothing reproducing it, so it is treated as a one-off environment/harness loss rather than a regression, but it was not positively explained.",
      "File Structure Plan drift (non-blocking, doc-only): design.md's New Files section names only test_ha_heartbeat_committed.c; the vote-floor safety integration test landed as a second registered file, test_ha_vote_floor_heartbeat.c. Coverage is equivalent/additive; a future doc pass could update design.md to match.",
      "design.md's File Structure Plan lists the ha.h path as src/controller/include/ha.h; the actual file is src/controller/ha.h. Cosmetic path drift already flagged in tasks.md's own Implementation Notes, not a functional gap."
    ]
  }
}