myxce/.kiro/specs/ha-wal-replication/spec.json

{
  "feature_name": "ha-wal-replication",
  "created_at": "2026-06-10T17:00:00Z",
  "updated_at": "2026-07-18T02:05:00Z",
  "language": "en",
  "phase": "implementation-complete",
  "approvals": {
    "requirements": {
      "generated": true,
      "approved": true
    },
    "design": {
      "generated": true,
      "approved": true
    },
    "tasks": {
      "generated": true,
      "approved": true
    }
  },
  "ready_for_implementation": true,
  "validate_impl": {
    "decision": "GO",
    "validated_at": "2026-07-18T02:05:00Z",
    "note": "Prior state: all 20 tasks (1.1-5.5) were [x] in tasks.md but phase was stuck at \"tasks-generated\" with no validate_impl record (same administrative-oversight pattern as ha-entity-snapshots / ha-commit-durability). Real implementation commits exist under the ha-wal-replication tag (18e351e repl_broadcast send, 81dbc4d registration bridge, 566c91c standby ACK reply, 861b746 primary ACK handler, ab270fa/e2fba08 tests, plus two later hardening commits: 708f227 durable-before-ACK, 64ab210 CRC-chain ownership fix, 46bd207 test-seed fix) -- this pass focused on cross-task/feature-level checks per kiro-validate-impl scope, run immediately after a sibling spec (raft-drive-async) validated NO-GO on a shared-codebase ASAN crash (bug-1447), so extra weight was given to reproducing or ruling out that crash signature in this spec's own live soak.\n\nMECHANICAL: Fresh ASAN build (nix develop + xce-build.sh controller asan) clean, zero compiler errors; meson Ok:3 Fail:0. Standalone controller_unit_test binary run directly (authoritative over meson's aggregate line per this project's own convention): 972 Tests, 0 Failures, 0 Ignored, OK -- zero \":FAIL:\" occurrences. WAL-replication-owned tests explicitly confirmed passing: test_repl_add_standby_max, test_repl_add_standby_idempotent, test_repl_registration_lifecycle, test_broadcast_one_standby_sends_entry, test_wal_ack_reply_updates_acked_lsn_and_signals, test_repl_broadcast_rejects_checkpoint, test_pin_repl_broadcast_full_fanout_skips_empty_and_catchup, test_repl_on_ack_wired_updates_watermark_and_commits. TBD/TODO grep on repl.c/ha.c/cluster.c/test_repl_unit.c: only the pre-existing, already-flagged-unrelated ha-term-safety stub at ha.c:1844. Secrets grep clean.\n\nINTEGRATION (cross-task wiring confirmed by direct code read, matching design.md's architecture end-to-end): standby registration on connect (ha.c:1556 raft_cb_become_leader, ha.c:8374 on_standby_connected) -> repl_add_standby; deregistration on disconnect (ha.c:6787 vote_ep_err_handler, ha.c:8422 standby_ep_err_handler_primary, ha.c:8814 eviction path) -> repl_remove_standby; standby applies entry then sends WalAckReply (ha.c:2601 ha_send_wal_ack_reply, wired from ha_apply_wal_catchup_entry, called from cluster.c:192 handle_wal_catchup_entry with ep+c passed through per task 3.1); primary ACK dispatch (ha.c:5928-5940 repl_ack_decode_and_dispatch's RequestReply_data_walackreplymsg arm -> repl_on_ack -> acked_lsn update + ack_cond signal, repl.c:190-226). repl_broadcast (repl.c:236-316) is a real TCP send via xce_wal_batch_req_raw, not a stub -- catch-up skip and MAX_PENDING_ACKS eviction guards both present and exercised by tests. Note: repl_broadcast's live signature now also carries prev_log_term (a later ha-term-safety-era addition for AppendEntries term-safety) beyond what design.md's Component 1 describes -- additive, not a boundary violation; the core WAL-transmission contract this spec committed to is unchanged.\n\nCOVERAGE: all 5 requirement sections map to completed tasks with no gaps -- Req 1 (Registration) -> tasks 2.1-2.3; Req 2 (Broadcast) -> tasks 1.1-1.2; Req 3 (Standby Receipt/ACK) -> tasks 3.1-3.3; Req 4 (Majority Commit) -> tasks 4.1-4.2; Req 5 (Correctness/Observability) -> tasks 5.1-5.5, with 5.1's in-order-apply guarantee satisfied by the adjacent bug-850 contiguity guard in ha_apply_single_entry (ha.c, returns -EAGAIN on an ahead-of-tail entry, silently deferred to catch-up) -- correctly relied upon rather than re-implemented, matching design.md's Out of Boundary note that catch-up streaming is ha-repl-hardening's territory. Minor logging deviation: repl_broadcast's per-send trace uses TRACE not literally \"DEBUG3\" (Req 5.2) -- cosmetic, not a functional gap.\n\nDESIGN: Architecture matches design.md's wire-protocol diagram exactly (repl_broadcast -> xce_wal_batch_req_raw -> standby apply -> WalAckReply -> repl_on_ack -> majority_ack_cond). File Structure Plan MISMATCH (non-blocking): design.md specified CREATE src/controller/testsuite/test_ha_wal_repl.c; the actual implementation consolidated the same test coverage into the pre-existing test_repl_unit.c (plus test_p2_integration.c, test_commit_engine.c, test_convergence_verification.c for cross-spec-integration pins) instead of a new file -- functionally equivalent, all Boundary Commitments (xce_wal_batch_req_raw single-entry mode, existing repl_add_standby/repl_remove_standby API, existing WalAckReply struct, no schema changes) held. Blocked tasks: none.\n\nRUNTIME LIVENESS (the blocking check, given the sibling bug-1447 finding): ran the canonical /xce-test-ha SEQUENTIAL 10-round soak (fresh binaries from this session's build) against a real ext4 deploy dir (/home/claude/tmp/xce-ha-deploy, cleaned up after). Result: PASS -- 10/10 rounds, 33 jobs submitted, zero committed-job loss, walcompare committed-prefix consistent every round, all 3 nodes (ctrl-1/ctrl-2/ctrl-3) served as primary at least once (leadership-coverage maneuver satisfied). Explicitly grepped all per-node logs for AddressSanitizer/DEADLYSIGNAL/SEGV/ssl3_read_bytes/handle_ssl_handshake/io_worker (bug-1447's exact signature): zero matches -- the sibling's netio.c SSL-handshake-race crash did NOT reproduce in this run. This does not prove bug-1447 is fixed or absent (it is a rare, timing-dependent race that took the sibling until round 9/10 to trigger once, and ha-wal-replication's own boundary does not touch the raft-drive-async reconnect-offload path that triggered it) -- but it does establish that ha-wal-replication's own WAL-transmission/ACK/commit path runs clean through a full supported failover lifecycle on the current codebase, which is this spec's own regression contract (Req 4, 5.5).\n\nOWNERSHIP: bug-1447 remains raft-drive-async/netio.c territory (see that spec's own NO-GO record); it is noted here only as a risk that was actively checked for and ruled out in this run, not adopted as this spec's blocker.",
    "known_limitations": [
      "bug-1447 (open, NOT-FIXED, owned by raft-drive-async / src/lib/netio.c, not this spec): a timing-dependent ASAN SEGV in handle_ssl_handshake/io_worker triggered once by raft-drive-async's own reconnect-offload cadence at round 9/10 of its sequential soak. Did not reproduce in this spec's clean 10/10 run, but as a shared-codebase race condition it cannot be ruled out for all future runs by one clean pass. No action item for ha-wal-replication; tracked entirely under raft-drive-async's remediation.",
      "File Structure Plan drift (non-blocking): design.md specified a new test_ha_wal_repl.c; actual tests landed in test_repl_unit.c (+ cross-spec integration pins in test_p2_integration.c/test_commit_engine.c/test_convergence_verification.c). Coverage is equivalent; a future doc pass could update design.md's File Structure Plan to match, but this is cosmetic.",
      "Req 5.2's DEBUG3-level broadcast log is implemented as TRACE level in the current logging framework's naming, not a literal \"DEBUG3\" call site -- functionally satisfies the intent (per-send LSN/opcode visibility at a low log level) but is a naming mismatch against the requirement's literal wording."
    ]
  }
}