From 5508b0b51c8e1de83bf7c76c9b79a01c89276d4f Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Sat, 20 Jun 2026 12:32:34 +0000 Subject: [PATCH] docs(ha-heartbeat-committed-propagation): mark 4.1 pending live-cluster validation (manual env) 1.1-3.2 committed + unit/integration-green (incl. vote-floor safety test). 4.1 (off-primary committed tracks primary E2E) needs a real-fs /xce-deploy-ha run; sandbox overlayfs fdatasync errno=5 (bug-795) breaks the commit path. Hand-off recipe recorded. Co-Authored-By: Claude Opus 4.8 (1M context) --- .kiro/specs/ha-heartbeat-committed-propagation/spec.json | 7 ++++++- .kiro/specs/ha-heartbeat-committed-propagation/tasks.md | 2 ++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/.kiro/specs/ha-heartbeat-committed-propagation/spec.json b/.kiro/specs/ha-heartbeat-committed-propagation/spec.json index ca3922e4..6d0777f2 100644 --- a/.kiro/specs/ha-heartbeat-committed-propagation/spec.json +++ b/.kiro/specs/ha-heartbeat-committed-propagation/spec.json @@ -18,5 +18,10 @@ "approved": true } }, - "ready_for_implementation": true + "ready_for_implementation": true, + "implementation": { + "completed": false, + "tasks_done": "1.1, 1.2, 2.1, 2.2, 3.1, 3.2", + "notes": "Code + safety verification implemented, reviewed, committed (f4327ce, 282a82f, 8bfa861, 90e7179, d534f83, 3b46562); controller + lib suites green; the vote-floor safety property (Req 3.2) is verified by an executable test against the real ha_vote_self_freshness gate. 4.1 (live E2E, Req 1.2/1.3) BLOCKED on manual real-fs env — sandbox overlayfs fdatasync errno=5 (bug-795) breaks the commit path the primary must advance. Hand-off recipe in tasks.md Implementation Notes." + } } diff --git a/.kiro/specs/ha-heartbeat-committed-propagation/tasks.md b/.kiro/specs/ha-heartbeat-committed-propagation/tasks.md index 2cba45e1..ea7aa915 100644 --- a/.kiro/specs/ha-heartbeat-committed-propagation/tasks.md +++ b/.kiro/specs/ha-heartbeat-committed-propagation/tasks.md @@ -52,6 +52,7 @@ - Observable: the live run shows a standby's `committedLsn` rising to track the primary between joins (not frozen at the join value), and the cluster's liveness/failover behavior is unchanged; manual environment prerequisite (live cluster, real fs). - _Requirements: 1.2, 1.3_ - _Depends: 3.1, 2.2_ + - _Blocked: live-cluster manual-env prerequisite — must run on a real-fs `/xce-deploy-ha` deployment (e.g. /tmp/xce-ha-deploy). The sandbox is overlayfs (`fdatasync errno=5`, bug-795), which breaks the commit/durability path the primary must advance for a standby to learn fresh committed. 1.1-3.2 are committed + unit/integration-green (incl. the vote-floor safety test). Hand-off recipe in Implementation Notes._ ## Implementation Notes - The HA heartbeat is `OPCode_nop` with a `heartbeatmsg` body — opcode and body are independent in this schema (mirrors `OP_hello` carrying `ctrlhellomsg` vs `replymsg`). Dispatch routes by `OPCode_nop` (unchanged); the consumer branches on `body_which`. @@ -61,4 +62,5 @@ - TEST-RUN LEARNING (2.x/3.x): the meson test wrapper (`xce-build.sh controller`) is NON-DETERMINISTIC ("OK"/"FAIL"/"Timeout") because of ~149 PRE-EXISTING LeakSanitizer leaks from `ha_start_election_async`/`ha_start_prevote_async` (`test_ha_start_election_async.c`, `test_ha_rdhup_integration.c`) — unrelated to this spec (identical count on a stashed base). AUTHORITATIVE verification = run the binary directly: `cd src && ASAN_OPTIONS=detect_leaks=0 ./xce.build/controller/controller_unit_test`, then check rc=0 and zero `:FAIL:` markers. - 3.1 LEARNING: primary stamps its OWN `ha->term` / `c->persist.committed_lsn` / `ha->committed_config_tag` (all `_Atomic`, acquire-load once per tick) — NOT the standby `last_known_*`. Test seam: new `ha_heartbeat_req_hook(ep,term,lsn,tag)` (`XCE_TESTING`, NULL default) with precedence new-hook → legacy `ha_nop_req_hook` → real `xce_ha_heartbeat_req`, so existing `ha_nop_req_hook` liveness tests stay green. - 1.2 LEARNING: the C-side `heartbeatmsg` body support PRE-EXISTED (no typemap edit needed) — `heartbeat_msg_t` typedef `typemap.h:17`, `xce_request_t.heartbeatmsg` member `typemap.h:287`, `struct heartbeat_msg_s{sender_priority,term,committed_lsn,committed_config_tag}` `typemap.h:445` (order matches schema `@0..@3`); enum `Request_body_heartbeatmsg == 22` (rides `OPCode_nop`). The codec (`encode/decode/free_HeartbeatMsg` + the request arm) was generated; only producer/consumer were missing. Test round-trip pattern: heap-`calloc` packet consumed+freed by `xce_decode` (caller does NOT free pkt), decoded body freed by `xce_free_request` (see `test_wal_wire.c`). Sender uses `xce_request(ep, &req)` (non-blocking). +- 4.1 HAND-OFF (live-cluster manual validation — run on a real-fs deploy, NOT the overlayfs sandbox): on a `/xce-deploy-ha` 3-node cluster, (1) `jobsub` a steady trickle of jobs so the primary's committedLsn keeps advancing; (2) repeatedly `ctrlstatus` a STANDBY (any-node hint view) and watch its `committedLsn` — it should RISE to track the primary within ~one heartbeat interval (no re-join), instead of being frozen at the join value or 0 (the controller-status Req 10 staleness this spec removes); (3) confirm liveness/failover is unaffected (kill the primary, a standby still takes over normally). PASS = standby committedLsn tracks the live primary between joins, with no liveness regression. Rebuild first so the cluster has the producer (task 3.1) + consumer (task 2.2). - The vote-floor formula is NOT edited — task 3.2 is a verification test only. Safety rests on the term-guard (task 2.2) + monotonicity (task 2.1) + cluster-monotonic committed; see design.md Security Considerations.