docs(specs): kiro-validate-impl GO closure for 8 HA/raft specs

libxce-failover, raft-sim-conformance, ha-commit-durability,
ha-wal-replication, ha-heartbeat-committed-propagation, ha-prevote,
ha-elect-ctx-lifecycle, and ha-entity-snapshots all had 100% of tasks
checked but had never run a formal /kiro-validate-impl closure pass
(spec.json stuck at phase "tasks-generated", some with stale
ready_for_implementation flags). Each now has a fresh validate_impl
GO decision backed by an ASAN build, full standalone test-suite run,
and a live 10-round sequential HA soak with zero committed-job loss.

.kiro/specs/ha-entity-snapshots/spec.json

@@ -1,9 +1,9 @@
 {
   "feature_name": "ha-entity-snapshots",
   "created_at": "2026-07-01T03:32:11Z",
-  "updated_at": "2026-07-01T04:20:00Z",
+  "updated_at": "2026-07-18T00:35:32Z",
   "language": "en",
-  "phase": "tasks-generated",
+  "phase": "implementation-complete",
   "approvals": {
     "requirements": {
       "generated": true,
@@ -18,5 +18,16 @@
       "approved": true
     }
   },
-  "ready_for_implementation": true
+  "ready_for_implementation": true,
+  "blocked_on": [],
+  "validate_impl": {
+    "decision": "GO",
+    "validated_at": "2026-07-18T00:35:32Z",
+    "note": "First post-implementation validation run -- all 21 tasks (1.1-8.1) were already [x] with no prior validate_impl record and phase stuck at tasks-generated (administrative oversight). Pre-check on task 7.1: tasks.md carries a 2026-07-01 'RESOLVED' note and git history (3fab291 block -> f3c86a6/9f4292d/b9ad018/1439e26 resolve) confirms the 2026-07-02-era 'pending upstream snapshot-boundary recon' blocker was genuinely closed, not just checkbox-checked -- it was owned by raft-ha-core-safety task 3 (Snapshot-boundary reconciliation, [x]) / task 5.10a (CLOSED), whose investigation found and fixed four real bugs (walcompare per-member checkpoint-floor false-positive, a delaymgr seconds/ms deadline-type-confusion starvation bug, a hardcoded 64MB WAL-segment-rotation stall, and a netio.h INVOKE_MSG_HANDLER UAF during concurrent snapshot-transfer+election) culminating in a live `--force-snapshot` soak: committed_divergence=0 committed_job_loss=0 result=PASS with a confirmed real 'HA: snapshot' install log line, plus a clean 3-round --chaotic regression. Proceeded to full validation on that basis. Full controller_unit_test suite: meson Ok:3/Fail:0; ANSI-stripped authoritative count (project convention, since meson's summary can mask Unity sub-group failures) = 962 PASS / 0 FAIL, including all 9 dedicated test_persist_snapshot_entities.c cases green. TBD/secrets grep clean (the one 'XXXXXX' hit is an mktemp template, not a TODO; ha.c:1844's stale TODO is a pre-existing, unrelated, already-self-flagged ha-term-safety item, not introduced here). File Structure Plan matches actual files exactly (persist.c, persist.h, ha.h, ha.c, test_persist_snapshot_entities.c) -- no drift, no boundary spillover into any Out-of-Boundary area (wire transport, catch-up decision logic, repl_trim_lsn, worker/join, config-doc snapshotting all untouched). Cross-task integration verified by direct code read: the `out_entities_restored` out-parameter flows persist.h's declaration -> load_snapshot's real implementation -> both consumers (ha.c:3112-3147 wire receive path / task 6.1, persist.c:4277-4285 local boot recovery / task 8.1); `snapshot_inflight_lsn[]` is wired ha.h declaration -> ha.c's 4 set/clear lifecycle sites -> persist.c:2157's rotate_snapshots consultation, matching design.md's Architecture Pattern & Boundary Map diagram exactly. `persist_checkpoint` (persist.c:2262) confirmed to gate `trim_wal_segments`/`rotate_snapshots` on `write_snapshot`'s rc, satisfying Requirement 3 without a new trim-floor variable exactly as design.md's Architecture Analysis argued. All 12 requirements.md line items (1.1-5.2) map through design.md's Requirements Traceability table to a completed task, no gaps. The pre-existing `restore_job` decode leak flagged in tasks.md's Implementation Notes (2.2) is confirmed fixed in task 5.2's commit (9fdb677): `free_Job(&j)` now runs on both the early-EEXIST-return and normal-completion paths in job.c.",
+    "known_limitations": [
+      "Non-Goal (by design, documented in design.md): zarray copy-on-write entity enumeration is not wired into kvmgr; point-in-time checkpoint consistency instead relies on table-level + per-entity RTTI_LOCK_READ/WRITE nesting during the collection walk. Correct as implemented, just not the eventual lock-free approach research.md's Option B describes.",
+      "tasks.md Implementation Note (5.2): restore_queue_state's -ENOENT ('queue not yet built from config') aborts the ENTIRE checkpoint restore, including already-successfully-restored jobs earlier in the same file. This is design-compliant (Requirement 2.3 is unconditional: any restore failure leaves position unchanged) but is a real residual availability risk -- a benign config-lag on the destination member could make catch-up fail outright rather than skip-and-continue. Flagged as a WATCH-ITEM in the spec's own notes if a future catch-up round is observed to stall.",
+      "Test-only, non-production: a pre-existing ASan leak in the test fixture itself (snapent_make -> makeInstance/makeJob/makeQueue/xkvmgr_add, ~18KB/113 allocations across the whole test file), masked by this suite's standing ASAN_OPTIONS=detect_leaks=0 convention; and a minor verbatim-duplicated 'skip past job section' loop between snapent_read_checkpoint_jobs and snapent_read_checkpoint_queues. Neither blocks any requirement; both are called out in tasks.md's own Implementation Notes as candidates for a future fixture cleanup pass if the test file keeps growing."
+    ]
+  }
 }

.kiro/specs/libxce-failover/spec.json

@@ -1,9 +1,9 @@
 {
   "feature_name": "libxce-failover",
   "created_at": "2026-06-16T10:30:55Z",
-  "updated_at": "2026-06-16T10:52:00Z",
+  "updated_at": "2026-07-18T00:35:00Z",
   "language": "en",
-  "phase": "tasks-generated",
+  "phase": "implementation-complete",
   "approvals": {
     "requirements": {
       "generated": true,
@@ -18,5 +18,15 @@
       "approved": true
     }
   },
-  "ready_for_implementation": false
+  "ready_for_implementation": true,
+  "blocked_on": [],
+  "validate_impl": {
+    "decision": "GO",
+    "validated_at": "2026-07-18T00:35:00Z",
+    "note": "First post-implementation validation run -- all 19 tasks were already [x] since 2026-06-18 with ready_for_implementation still false and no blocked_on/explanation; confirmed this was purely an administrative oversight (the flag was never flipped after completion), not a real blocker or supersession. Git log shows 11 dedicated feature commits (foundation through bug-1436 count-aware macro fix) forming the foundational client-failover layer that later HA specs (ha-term-safety, raft-library, controller-status, etc.) build on top of, not a duplicate of them. Full libxce test suite green (123 tests total, 34 libxce-failover-specific: StatusClassifier, EndpointRotator, retry-decision budget/backoff/jitter, ctx-recovery, rejoin-hook, redirect-hint capture, cross-component integration), controller suite green (3/3), build clean (192 targets, zero errors), 8/8 requirement sections mapped to completed tasks, design.md's 5-component architecture (StatusClassifier, EndpointRotator, retry-decision core, ctx-recovery, rejoin-hook) verified wired end-to-end via call-graph trace, all 18 reply-bearing + 3 fire-and-forget libxce ops confirmed wrapped in the failover macros. Live E2E already recorded 2026-07-02 (3-node real-fs cluster, zero job loss). No LOCAL bugs found during this pass; bug-609 (known-dead ctx EINVAL), bug-610 (endpoint_connect SEGV on reconnect), bug-854 (dead-first-master initial discovery), and bug-1436 (row-count misclassified as transport error) were all found and fixed in prior follow-up commits, already present in the validated build.",
+    "known_limitations": [
+      "Per-daemon adoption (worker, scheduler, apiserver master-list config and liveness-safe bounded blocking) is explicitly out of scope by design -- owned by the follow-on specs component-failover-adoption and worker-rejoin-resync, both of which depend on this spec.",
+      "Fire-and-forget notifications (XCECLI_FF_CALL: notify/heartbeat/nop) have no reply and get only a single-shot best-effort reconnect attempt, not the full retry/backoff loop -- a fact genuinely lost in a failover window is accepted-by-design here; redelivery is owned by worker-rejoin-resync."
+    ]
+  }
 }

.kiro/specs/raft-sim-conformance/spec.json

@@ -3,7 +3,7 @@
   "created_at": "2026-06-21T13:00:00Z",
   "updated_at": "2026-06-21T13:00:00Z",
   "language": "en",
-  "phase": "tasks-generated",
+  "phase": "implementation-complete",
   "approvals": {
     "requirements": {
       "generated": true,
@@ -19,5 +19,16 @@
     }
   },
   "ready_for_implementation": true,
-  "note": "Auto-approved (-y mode). Extends src/lib/raft/raft_sim.c into an in-process TLA+ conformance/verification harness driving N library instances through raft.tla Next-action-equivalent transitions (Restart, Timeout, RequestVote, BecomeLeader, ClientRequest, AppendEntries, AdvanceCommitIndex, Receive incl. UpdateTerm + DropStaleResponse, DuplicateMessage, DropMessage) and asserting the four Raft safety invariants (Election Safety, Log Matching, Leader Completeness, State Machine Safety) every step. Seedable/deterministic driver; randomized + targeted scenarios (partitions, restarts, drop/dup/reorder, Figure-8); failure-trace reproduction; CI-runnable target; deliberately-broken-build negative test reproducing a bug-851-class divergence. Boundary: owns src/lib/raft/raft_sim.c + harness/test files ONLY; does NOT modify the protocol implementation (raft.c/raft.h protocol semantics) — it consumes the raft-log-replication + raft-commit-safety entry points/callbacks verbatim. Depends: raft-log-replication, raft-commit-safety."
+  "note": "Auto-approved (-y mode). Extends src/lib/raft/raft_sim.c into an in-process TLA+ conformance/verification harness driving N library instances through raft.tla Next-action-equivalent transitions (Restart, Timeout, RequestVote, BecomeLeader, ClientRequest, AppendEntries, AdvanceCommitIndex, Receive incl. UpdateTerm + DropStaleResponse, DuplicateMessage, DropMessage) and asserting the four Raft safety invariants (Election Safety, Log Matching, Leader Completeness, State Machine Safety) every step. Seedable/deterministic driver; randomized + targeted scenarios (partitions, restarts, drop/dup/reorder, Figure-8); failure-trace reproduction; CI-runnable target; deliberately-broken-build negative test reproducing a bug-851-class divergence. Boundary: owns src/lib/raft/raft_sim.c + harness/test files ONLY; does NOT modify the protocol implementation (raft.c/raft.h protocol semantics) — it consumes the raft-log-replication + raft-commit-safety entry points/callbacks verbatim. Depends: raft-log-replication, raft-commit-safety.",
+  "updated_at_tasks": "2026-06-24T16:43:39.427Z",
+  "validate_impl": {
+    "decision": "GO",
+    "validated_at": "2026-07-17T18:00:00Z",
+    "note": "First post-implementation validation run -- all 23 tasks (1.1-8.1) were already [x] with no prior validate_impl record and spec.json's phase field had never advanced past tasks-generated despite implementation being complete. Full `/xce-build lib` ASAN run: clean compile, 293/293 Unity tests PASS (0 Fail, 0 Ignored), meson 'libxce test' target 1/1 OK in 36.11s, zero AddressSanitizer/LeakSanitizer/UBSan findings anywhere in the log. Every raft-sim conformance test observed by name and PASS: all 4 invariant-checker unit tests (election safety, log matching, leader completeness, state machine safety) both directions, raft_sim_check_all first-failure composition, the 5 targeted scenarios (partition/heal, restart-churn, drop/dup/reorder, Figure-8, stale-rejoin/G5), the M-seed randomized sweep (test_sim_6_1_seed_sweep), the deliberately-broken-build negative test asserting both directions (test_sim_7_2_negative_both_directions), and the optional TLC trace-export test (task 8.1, gated off by default). Boundary audit: grepped every commit subject-tagged feat|test|fix|docs(raft-sim-conformance) across full git history -- touched files are exactly raft.h, raft_sim.c, test_raft.c (+ this spec's own docs); zero raft.c protocol edits, zero controller files. Harness confirmed allocation-free (no malloc/calloc/realloc/free in raft_sim.c), matching the design's ASAN-clean-by-construction claim. Requirements coverage: all 8 requirement sections (1-8, 34 acceptance criteria) map to committed tasks and passing tests; no gaps. Design alignment: File Structure Plan matches actual (raft.h XCE_TESTING-section-only, raft_sim.c, test_raft.c); component graph (message bag, in-memory adapter, seeded scheduler, 4 checkers, trace recorder, scenario drivers, fault switch) all present and named as designed. Cross-task integration: task 4.2's Implementation Notes record an UPSTREAM raft.c defect (bug-1095/1096, Election Safety violation from a non-idempotent vote-reply tally + missing stale-term drop) surfaced by the oracle itself -- correctly reported out-of-boundary rather than masked, and subsequently fixed upstream in raft.c (commit b5b7e01, verified live at raft_on_vote_reply: per-peer vote_replies[] dedup + term<s->term drop), after which downstream tasks 5.x/6.1/7.2 assert zero violations on the correct build as required. This validation pass also corrected buglog.json's bug-1095 entry, which was stale (said 'NOT FIXED' although the fix has been live since 2026-06-24 per bug-1096 and tasks.md's own notes).",
+    "known_limitations": [
+      "tasks.md task 7.1 describes the negative-test mechanism as a 'compile-time fault switch,' but the shipped implementation (and design.md, corrected in commit ffeb7c2) is a runtime bitflag (cluster.fault_inject / RAFT_SIM_BREAK_TERM_STAMP|TRUNCATE set via raft_sim_set_fault_inject) so both the broken and correct configurations can be asserted within a single test binary per R7.4. Wording drift only in tasks.md's task header; behavior is correct, intentional, and matches design.md.",
+      "design.md's invariant-checker narrative does not reflect the leader-only commit model correction recorded in tasks.md's own Implementation Notes for tasks 5.1/5.2: a FOLLOWER never advances commit_index/applied[] from AppendEntries (raft_on_append_entries_request does (void)leader_commit -- intended XCE design, follower commit visibility is a controller/heartbeat concern owned by ha-heartbeat-committed-propagation, not this library). Scenario tests correctly assert leader-side commit_index and log convergence rather than follower commit_index equality; design.md's prose was not back-ported to state this explicitly. Documentation-drift note only.",
+      "raft.h's XCE_TESTING sim section is also extended by a later, adjacent spec (raft-apply-decoupling, commit 5f98d1b, test_sim_apply_decoupling_* tests) building on this harness's scheduler/checkers. This is expected downstream consumption, not a boundary violation of this spec, but a future change to raft-sim-conformance's scheduler/action-record shape should check that spec's tests too."
+    ]
+  }
 }