diff --git a/.kiro/specs/ha-commit-durability/spec.json b/.kiro/specs/ha-commit-durability/spec.json index 7edadcc5..e54de69c 100644 --- a/.kiro/specs/ha-commit-durability/spec.json +++ b/.kiro/specs/ha-commit-durability/spec.json @@ -1,9 +1,9 @@ { "feature_name": "ha-commit-durability", "created_at": "2026-06-17T23:40:42Z", - "updated_at": "2026-06-19T00:00:00Z", + "updated_at": "2026-07-18T00:55:00Z", "language": "en", - "phase": "tasks-generated", + "phase": "implementation-complete", "reopened_at": "2026-06-19T00:00:00Z", "reopened_reason": "bug-795: total-cluster-outage (cold-restart-all) committed-job loss surfaced by the xce-test-e2e-primitives soak; Phase 6 added (6.1 reproduce-on-real-fs gates the rest).", "approvals": { @@ -20,5 +20,17 @@ "approved": true } }, - "ready_for_implementation": true + "ready_for_implementation": true, + "blocked_on": [], + "validate_impl": { + "decision": "GO", + "validated_at": "2026-07-18T00:55:00Z", + "note": "Prior state: all 28 tasks (1-6.4) were [x] in tasks.md but phase was stuck at \"tasks-generated\" with no validate_impl record (administrative oversight, same pattern as ha-entity-snapshots). Per-task Implementation Notes in tasks.md already document extensive self-review of each task (including a corrective REWORK note for the never-self-terminate invariant and a cross-task fix, bug-760, for an adjacent test regression) -- this pass focuses on cross-task/feature-level checks.\n\nMECHANICAL: Full ASAN build clean (nix develop + xce-build.sh controller asan), zero compiler errors. Meson test summary Ok:3 Fail:0. Standalone controller_unit_test binary (authoritative per this project's own meson-Fail:0-masks-Unity-subgroups convention): 0 \":FAIL:\" occurrences, 791 individual \": PASS\" test lines, all green (process exit code alone is not gating -- pre-existing unrelated ASAN leak-report noise at exit is a documented quirk, not a functional failure). TBD/TODO grep on the feature's owned files (persist.c, persist.h, commit.c, ha.c, cluster.c, test_commit_durability.c, test-commit-durability.sh) found only pre-existing, unrelated items not introduced by this spec: cluster.c's 3 TODOs are an unrelated config-generation-tool note and a config-group FIXME, and ha.c:1844 is the same pre-existing ha-term-safety-owned stub TODO already flagged as unrelated by the ha-entity-snapshots validate_impl pass. Secrets grep clean.\n\nSTALENESS RE-CHECK (the key judgment call): this spec's own live evidence (deterministic RED->GREEN E2E and rolling-soak zero-loss) was captured 2026-06-18/2026-06-19, and the dedicated cold-restart-all real-fs revalidation (task 6.1-6.4) on 2026-07-02 -- both materially predate today. In the interim, persist.c/ha.c/cluster.c/commit.c (the exact files this spec owns) received substantial further edits: bug-1433/1434 (2026-07-11, added persist_committed_lsn_advance_atomic -- a CAS-safe committed_lsn advance, and removed a force-flush call reachable from a never-blocks vtask class), bug-1417/1419 (vote-index mapping, WAL replay-tally dedup), the job-array-single-wal-record series (new WAL_JOB_ARRAY_SUBMIT record type + apply/replay changes), and bug-1445 (2026-07-17, first-boot promotion-check-vs-own-first-replay race in raft_cb_become_leader -- the SAME function this spec's promotion-eligibility gate lives in). Given this density of churn in exactly the owned files, the old live evidence alone was judged insufficient without a fresh check. Code-level verification: majority_candidate_locked (commit.c:135) still caps the self contribution at persist_durable_lsn (unchanged); persist_committed_lsn_advance_atomic (persist.c:5881) and its two call sites explicitly preserve \"committed_lsn never advances past durable_lsn\" in their own code comments, i.e. the later CAS fix was written to respect this spec's invariant rather than bypass it; bug-1445's new check is an orthogonal job-array live-apply-fence re-check added alongside, not a modification of, this spec's recovery-truncated/durable-floor promotion gate.\n\nFRESH LIVE RE-VALIDATION (this session, genuine ext4 -- confirmed /home/claude, backing device /dev/nvme0n1p2 ext4, NOT the container's overlay /): (a) --deterministic strace mode returned a false RED (3/3 probes \"0 fdatasync in-window\"). Diagnosed, not accepted at face value: this sandbox's date is BusyBox date, which does not support GNU date's %N (nanoseconds) -- confirmed by direct invocation, it emits the literal unexpanded string \"%N\", breaking the script's sub-second window arithmetic. Cross-checked strace's own (unaffected) microsecond timestamps against the controller's committed_lsn advance log lines for all 3 probes: fdatasync consistently PRECEDED the commit-advance/ack by 16-43ms in every probe -- i.e. durable-before-ack genuinely holds on the current build; the script's verdict was a test-harness false positive, not a product regression. Logged as bug-1446 (test-harness only). (b) 4-round rolling SIGKILL+restart soak plus a full cold-restart-all cycle on the same ext4 deploy: 8/8 PASS, 0 job loss -- all 20 committed jobs survived both rolling failover and cold-restart-all. This directly reconfirms Req 1.5 and the Phase-6 total-outage guarantee on today's code, not just June/early-July code.\n\nINTEGRATION: durable_lsn -> majority_candidate_locked self-cap -> committed_lsn -> commit barrier -> client reply chain verified intact end to end (code read + the live soak's observed ordering above). recovery-verdict -> promotion-gate -> freshness-floor chain verified intact (persist_recovery_verdict/persist_durable_floor still consumed by ha.c's promotion gate and cluster.c's vote/PreVote freshness comparisons). No boundary spillover: the only two touches to raft_cb_become_leader since this spec landed (bug-1433/1434's force-flush removal, bug-1445's first-boot fence re-check) are both additive/orthogonal, not modifications of this spec's owned gate logic. src/controller/CLAUDE.md's \"Commit Durability\" guarantee note (Protocol Verification Checklist) is current and accurate, and is explicitly cross-referenced by the newer bug-1434 code comments.\n\nCOVERAGE: all 3 requirements sections (1.1-1.6 Durable Commit, 2.1-2.6 Recovery Integrity and Leadership Fence, 3.1-3.3 Deterministic Durability Verification) map through design.md's Requirements Traceability table to a completed task; no gaps. File Structure Plan matches actual files (persist.c/.h, commit.c, ha.c, cluster.c, config.h, main.c, test_commit_durability.c, test-commit-durability.sh) with no drift. Blocked tasks: none.", + "known_limitations": [ + "bug-1396 (open, low-priority, unconfirmed): an extra/duplicate committed job appeared after one cold-restart-all cycle in the original 2026-07-02 5-run validation -- the opposite symptom of the loss this spec targets (over-counting, not loss), not yet root-caused. Does not block this GO (it is not a durability-invariant violation) but remains open for a future investigation.", + "persist_apply_pump_run_sync (flagged by bug-1433, not introduced by this spec): still reachable from raft_cb_become_leader on a VTASK_LIVENESS (never-blocks) vtask and performs bounded pread() -- lower risk than the fdatasync this spec's own group-commit flusher correctly keeps off that path, but strictly still blocking I/O. Tracked as its own follow-up in bug-1433, adjacent to but outside this spec's boundary.", + "bug-1446 (this session, test-harness only): src/tools/test-commit-durability.sh's --deterministic mode produces a false RED in any environment whose `date` lacks GNU date's %N support (e.g. BusyBox date, as in this sandbox) because its submit-to-ack window arithmetic silently breaks on the unexpanded literal \"%N\". The soak mode used for this session's live re-validation is unaffected and remains the authoritative live check in such environments. A future fix should make the script detect/require GNU date or use an alternative sub-second time source, and fail loud on an incompatible date rather than emitting a false verdict.", + "The group-commit batching mechanism's interaction with the newer job-array-single-wal-record feature (multi-entry array WAL records) was confirmed consistent at the shared committed_lsn/durable_lsn-invariant level (code read) but was not independently re-exercised by a dedicated array-specific durability soak in this pass; job-array-single-wal-record carries its own validate_impl track for that." + ] + } } diff --git a/.kiro/specs/ha-elect-ctx-lifecycle/spec.json b/.kiro/specs/ha-elect-ctx-lifecycle/spec.json index 6cd22153..e146b1d6 100644 --- a/.kiro/specs/ha-elect-ctx-lifecycle/spec.json +++ b/.kiro/specs/ha-elect-ctx-lifecycle/spec.json @@ -1,9 +1,9 @@ { "feature_name": "ha-elect-ctx-lifecycle", "created_at": "2026-06-16T03:10:00Z", - "updated_at": "2026-07-11T05:30:00Z", + "updated_at": "2026-07-18T09:56:00Z", "language": "en", - "phase": "tasks-generated", + "phase": "implementation-complete", "approvals": { "requirements": { "generated": true, @@ -18,5 +18,15 @@ "approved": true } }, - "ready_for_implementation": true + "ready_for_implementation": true, + "validate_impl": { + "decision": "GO", + "validated_at": "2026-07-18T09:56:00Z", + "note": "Cross-task/feature-level validation of a spec whose 13/13 tasks.md leaf checkboxes (1.1, 2.1-2.4, 3.1-3.5) had been complete since before 2026-07-11, but whose spec.json phase was never advanced past tasks-generated (same administrative-oversight pattern as several sibling specs validated earlier today: ha-wal-replication, ha-heartbeat-committed-propagation, ha-prevote). This is the LAST of 9 Group-1 validate-impl closure passes run this session. Ran immediately after ha-prevote (GO) -- extra weight was given to the fact that ha-prevote's own soak already independently exercised this spec's fix (commit 6529dcf) via test_many_prevote_rounds_no_carrier_leak, giving this pass a useful cross-check without treating it as sufficient on its own.\n\nCROSS-TASK CONSISTENCY / DESIGN ALIGNMENT: confirmed live on-disk source matches design.md's File Structure Plan exactly, field-for-field. src/include/netio.h:103/110 -- on_fini is declared immediately adjacent to on_connected with the documented cross-reference comment (as specified). src/lib/netio.c:1112-1113 -- endpoint_fini calls ep->on_fini(ep) immediately before FREEUP(ep->data) (line 1116), matching the designed firing point exactly. src/controller/ha.h:126 -- _Atomic int refcount added to ha_elect_ctx_t with a doc comment that reproduces the design's bias-reference rationale verbatim. src/controller/ha.c: ha_elect_ctx_release (line 150, decrement + pthread_mutex_destroy/sem_destroy/free at zero) and vote_ep_on_fini (line 180, reads vctx from ep->data, releases vctx->ctx) are the single choke points design.md specifies; both raft_cb_send_vote_request (line 282 increment, line 293 hook wire) and raft_cb_send_pre_vote_request (line 421 increment, line 432 hook wire) apply the identical increment+wire pattern -- Requirement 3's 'uniform coverage' claim holds by direct source inspection, not just test titles. Supersession release sites present at lines 7106-7108 and 7376-7378 (atomic_exchange_explicit retrieving the outgoing pointer, then ha_elect_ctx_release on it) matching the design's 'exchange instead of store' mechanism for both ha_start_election_async and ha_start_prevote_async. Two additional release call sites (7423, 7442) cover error/abort paths per Requirement 2.4. Dependency direction and Boundary Commitments honored: no change to endpoint_fini's existing teardown mechanics beyond the one conditional hook call, no change to PreVote/election/failover semantics, the unrelated bug-1298 endpoint refcnt mechanism is untouched and not conflated with the new carrier refcount. No Revalidation Trigger has fired (no second on_fini consumer exists, no new vote endpoint capture site was added outside the two documented ones).\n\nMECHANICAL: Fresh ASAN build (nix develop + xce-build.sh controller asan) -- zero compiler/linker errors, xce_controller/controller_unit_test/xce_test binaries confirmed with session-fresh mtimes. Standalone controller_unit_test binary run directly (authoritative over meson's aggregate line per this project's convention): 972 Tests, 0 Failures, 0 Ignored, OK, exit 0 -- zero regressions across the full suite, including existing vote-tally/quorum/freshness-gate tests (bug-339/bug-340 vote-freshness fixes unaffected). LEAK-SPECIFIC CHECK (this spec's core deliverable is a leak fix, so this is the load-bearing evidence): re-ran test_ha_elect_ctx_lifecycle.c in isolation with ASAN_OPTIONS=detect_leaks=1 (leak detection is off by project-wide default per cerebrum.md convention; this spec's own tests are written to enable it explicitly for exactly this purpose) -- 5 Tests, 0 Failures, 0 Ignored, OK, exit 0: test_release_at_refcount_one_frees_no_crash, test_release_at_refcount_two_leaves_valid, test_many_election_rounds_no_carrier_leak (20 rounds x 2 votes), test_many_prevote_rounds_no_carrier_leak (20 rounds x 2 votes), test_carrier_survives_supersession_with_outstanding_endpoint -- zero leaked ha_elect_ctx_t allocations across 40 real-election + 40 PreVote vote endpoints, and no premature-free UAF. TBD/TODO/FIXME/HACK/XXX grep on the spec's boundary files (ha.c, ha.h, netio.h, netio.c, test_ha_elect_ctx_lifecycle.c): one match (ha.c:1844, 'TODO(ha-term-safety task 2.x)') -- belongs to a different spec's stub, not this one's boundary. Secrets grep: clean.\n\nRUNTIME LIVENESS: ran the canonical /xce-test-ha SEQUENTIAL 10-round soak (fresh session binaries) against a real ext4 deploy dir. FIRST ATTEMPT FAILED at round 1: 'no new primary elected after kill (no serving primary at term>2)'. Root-caused via the controller logs, not assumed: cold-start election was unusually slow this run (ctrl-1 reached CTRL_PRIMARY term=2 only ~10.1s after process start), and the driver's round-1 kill fired ~1.7s after that -- before ctrl-1's just-started config-push handshake to the two standbys ('ctrlhello sent' at the same timestamp as promotion) could complete. Both standbys were still at tag=0 (no local config) at kill time, so per ha-config-bootstrap's own Req 2.8 guard they correctly refused to start PreVote/RDHUP re-election ('no local config (tag=0, n_peers=3) -- voter only, not starting PreVote/RDHUP re-election') rather than risk electing an unconfigured primary. This is a test-harness timing race (round 1 has no guard waiting for initial standby config receipt before killing the primary) -- not a code defect in this spec's boundary (no code this spec owns touches config propagation or the ha-config-bootstrap tag gate). RETRY: same command, fresh deploy dir, immediately after -- PASS: 10/10 rounds, 36 jobs submitted, zero job loss, zero committed-prefix divergence, all 3 nodes (ctrl-1/ctrl-2/ctrl-3) served as primary at least once. Grepped both the failed and the passing run's driver + all per-node controller logs for bug-1447's exact signature (AddressSanitizer, DEADLYSIGNAL, SEGV, ssl3_read_bytes, handle_ssl_handshake, io_worker) and other crash signatures (Segmentation fault, core dumped, Aborted, double free, heap-buffer-overflow, use-after-free): zero matches in either run. No leftover xce_controller processes after either run (driver's own cleanup on FAIL, and clean exit on PASS, both confirmed via ps).\n\nOWNERSHIP: bug-1447 remains raft-drive-async/netio.c territory, not reproduced here. The round-1 config-push/kill-timing race is a test-harness gap in run-ha-soak.sh's sequential mode (no wait for initial 3-of-3 config convergence before the first kill), not owned by this spec -- flagged here as a known_limitation for whoever next touches the driver or ha-config-bootstrap, not adopted as a blocker for this spec's own GO. No new bugs found or fixed during this validation pass; no code changes were made -- this was a read/build/test/soak-only pass.", + "known_limitations": [ + "Test-harness timing race in .claude/skills/xce-test-ha/run-ha-soak.sh sequential mode: round 1's kill is not gated on the standbys having received their initial config push, so on a slow cold-start election (observed here: ~10s to CTRL_PRIMARY) round 1 can fire before config propagates, hitting ha-config-bootstrap's Req 2.8 voter-only guard and failing the round with 'no serving primary at term>2'. Reproduced once in this session, did not reproduce on an immediate retry (10/10 clean). Not owned by this spec (no code this spec touches affects config propagation or the tag=0 guard) -- flagged for whoever next touches run-ha-soak.sh or ha-config-bootstrap; a fix would add a short 3-of-3-config-received wait before round 1's first kill.", + "bug-1447 (open, NOT-FIXED, owned by raft-drive-async / src/lib/netio.c, not this spec): a timing-dependent ASAN SEGV in handle_ssl_handshake/io_worker. Did not reproduce in either soak run in this pass (failed or passing). No action item for this spec.", + "Requirement 4.3 names both sequential AND chaotic /xce-test-ha regression modes. Per the explicit validation-scope instruction for this pass, only the sequential 10-round soak was (re-)run; chaotic mode was not independently re-run for this spec. This mirrors the evidence scope of the sibling ha-prevote validation (also sequential-only) and is a low-risk gap given this is a pure memory-lifetime fix with zero semantic surface on election/failover outcomes (chaotic mode's specific hazard class -- double-fault commit-prefix divergence timing -- is orthogonal to carrier refcounting)." + ] + } } diff --git a/.kiro/specs/ha-entity-snapshots/spec.json b/.kiro/specs/ha-entity-snapshots/spec.json index 87df3f85..85390fcd 100644 --- a/.kiro/specs/ha-entity-snapshots/spec.json +++ b/.kiro/specs/ha-entity-snapshots/spec.json @@ -1,9 +1,9 @@ { "feature_name": "ha-entity-snapshots", "created_at": "2026-07-01T03:32:11Z", - "updated_at": "2026-07-01T04:20:00Z", + "updated_at": "2026-07-18T00:35:32Z", "language": "en", - "phase": "tasks-generated", + "phase": "implementation-complete", "approvals": { "requirements": { "generated": true, @@ -18,5 +18,16 @@ "approved": true } }, - "ready_for_implementation": true + "ready_for_implementation": true, + "blocked_on": [], + "validate_impl": { + "decision": "GO", + "validated_at": "2026-07-18T00:35:32Z", + "note": "First post-implementation validation run -- all 21 tasks (1.1-8.1) were already [x] with no prior validate_impl record and phase stuck at tasks-generated (administrative oversight). Pre-check on task 7.1: tasks.md carries a 2026-07-01 'RESOLVED' note and git history (3fab291 block -> f3c86a6/9f4292d/b9ad018/1439e26 resolve) confirms the 2026-07-02-era 'pending upstream snapshot-boundary recon' blocker was genuinely closed, not just checkbox-checked -- it was owned by raft-ha-core-safety task 3 (Snapshot-boundary reconciliation, [x]) / task 5.10a (CLOSED), whose investigation found and fixed four real bugs (walcompare per-member checkpoint-floor false-positive, a delaymgr seconds/ms deadline-type-confusion starvation bug, a hardcoded 64MB WAL-segment-rotation stall, and a netio.h INVOKE_MSG_HANDLER UAF during concurrent snapshot-transfer+election) culminating in a live `--force-snapshot` soak: committed_divergence=0 committed_job_loss=0 result=PASS with a confirmed real 'HA: snapshot' install log line, plus a clean 3-round --chaotic regression. Proceeded to full validation on that basis. Full controller_unit_test suite: meson Ok:3/Fail:0; ANSI-stripped authoritative count (project convention, since meson's summary can mask Unity sub-group failures) = 962 PASS / 0 FAIL, including all 9 dedicated test_persist_snapshot_entities.c cases green. TBD/secrets grep clean (the one 'XXXXXX' hit is an mktemp template, not a TODO; ha.c:1844's stale TODO is a pre-existing, unrelated, already-self-flagged ha-term-safety item, not introduced here). File Structure Plan matches actual files exactly (persist.c, persist.h, ha.h, ha.c, test_persist_snapshot_entities.c) -- no drift, no boundary spillover into any Out-of-Boundary area (wire transport, catch-up decision logic, repl_trim_lsn, worker/join, config-doc snapshotting all untouched). Cross-task integration verified by direct code read: the `out_entities_restored` out-parameter flows persist.h's declaration -> load_snapshot's real implementation -> both consumers (ha.c:3112-3147 wire receive path / task 6.1, persist.c:4277-4285 local boot recovery / task 8.1); `snapshot_inflight_lsn[]` is wired ha.h declaration -> ha.c's 4 set/clear lifecycle sites -> persist.c:2157's rotate_snapshots consultation, matching design.md's Architecture Pattern & Boundary Map diagram exactly. `persist_checkpoint` (persist.c:2262) confirmed to gate `trim_wal_segments`/`rotate_snapshots` on `write_snapshot`'s rc, satisfying Requirement 3 without a new trim-floor variable exactly as design.md's Architecture Analysis argued. All 12 requirements.md line items (1.1-5.2) map through design.md's Requirements Traceability table to a completed task, no gaps. The pre-existing `restore_job` decode leak flagged in tasks.md's Implementation Notes (2.2) is confirmed fixed in task 5.2's commit (9fdb677): `free_Job(&j)` now runs on both the early-EEXIST-return and normal-completion paths in job.c.", + "known_limitations": [ + "Non-Goal (by design, documented in design.md): zarray copy-on-write entity enumeration is not wired into kvmgr; point-in-time checkpoint consistency instead relies on table-level + per-entity RTTI_LOCK_READ/WRITE nesting during the collection walk. Correct as implemented, just not the eventual lock-free approach research.md's Option B describes.", + "tasks.md Implementation Note (5.2): restore_queue_state's -ENOENT ('queue not yet built from config') aborts the ENTIRE checkpoint restore, including already-successfully-restored jobs earlier in the same file. This is design-compliant (Requirement 2.3 is unconditional: any restore failure leaves position unchanged) but is a real residual availability risk -- a benign config-lag on the destination member could make catch-up fail outright rather than skip-and-continue. Flagged as a WATCH-ITEM in the spec's own notes if a future catch-up round is observed to stall.", + "Test-only, non-production: a pre-existing ASan leak in the test fixture itself (snapent_make -> makeInstance/makeJob/makeQueue/xkvmgr_add, ~18KB/113 allocations across the whole test file), masked by this suite's standing ASAN_OPTIONS=detect_leaks=0 convention; and a minor verbatim-duplicated 'skip past job section' loop between snapent_read_checkpoint_jobs and snapent_read_checkpoint_queues. Neither blocks any requirement; both are called out in tasks.md's own Implementation Notes as candidates for a future fixture cleanup pass if the test file keeps growing." + ] + } } diff --git a/.kiro/specs/ha-heartbeat-committed-propagation/spec.json b/.kiro/specs/ha-heartbeat-committed-propagation/spec.json index 6d0777f2..104b28de 100644 --- a/.kiro/specs/ha-heartbeat-committed-propagation/spec.json +++ b/.kiro/specs/ha-heartbeat-committed-propagation/spec.json @@ -3,7 +3,7 @@ "created_at": "2026-06-20T00:00:00Z", "updated_at": "2026-06-20T00:00:00Z", "language": "en", - "phase": "tasks-generated", + "phase": "implementation-complete", "approvals": { "requirements": { "generated": true, @@ -20,8 +20,19 @@ }, "ready_for_implementation": true, "implementation": { - "completed": false, - "tasks_done": "1.1, 1.2, 2.1, 2.2, 3.1, 3.2", - "notes": "Code + safety verification implemented, reviewed, committed (f4327ce, 282a82f, 8bfa861, 90e7179, d534f83, 3b46562); controller + lib suites green; the vote-floor safety property (Req 3.2) is verified by an executable test against the real ha_vote_self_freshness gate. 4.1 (live E2E, Req 1.2/1.3) BLOCKED on manual real-fs env — sandbox overlayfs fdatasync errno=5 (bug-795) breaks the commit path the primary must advance. Hand-off recipe in tasks.md Implementation Notes." + "completed": true, + "tasks_done": "1.1, 1.2, 2.1, 2.2, 3.1, 3.2, 4.1", + "notes": "Code + safety verification implemented, reviewed, committed (f4327ce, 282a82f, 8bfa861, 90e7179, d534f83, 3b46562); controller + lib suites green; the vote-floor safety property (Req 3.2) is verified by an executable test against the real ha_vote_self_freshness gate. 4.1 (live E2E, Req 1.2/1.3) VERIFIED 2026-07-02 on a real-fs (~/tmp, ext4) 3-node deploy: standby committedLsn tracked the primary within one heartbeat interval with no re-join, and continued advancing correctly across a failover. See tasks.md Implementation Notes." + }, + "validate_impl": { + "decision": "GO", + "validated_at": "2026-07-18T09:45:00Z", + "note": "RETRY of a validation pass that crashed silently mid-live-soak ~4 hours earlier with no artifacts/logs (stale empty deploy dir /home/claude/tmp/xce-ha-heartbeat-committed-deploy found and removed at start of this pass; no evidence it was a product bug rather than an environment/harness loss). Prior state: all 11 tasks (1.1-4.1) were [x] in tasks.md, 4.1 already live-verified 2026-07-02, but phase was stuck at \"tasks-generated\" with no validate_impl record (same administrative-oversight pattern as ha-wal-replication / ha-entity-snapshots / ha-commit-durability). This pass ran immediately after two sibling specs: raft-drive-async (NO-GO, bug-1447 ASAN SEGV in netio.c SSL handshake) and ha-wal-replication (GO, clean 10/10 soak) -- extra weight was given to reproducing or ruling out bug-1447's signature in this spec's own live soak.\n\nMECHANICAL: Fresh ASAN build (nix develop + xce-build.sh controller asan), binaries confirmed with session-fresh mtimes (controller_unit_test, xce_test, xce_controller). Standalone controller_unit_test run directly with ASAN_OPTIONS=detect_leaks=0 (authoritative over meson's aggregate line per this project's convention): 972 Tests, 0 Failures, 0 Ignored, OK -- zero \":FAIL:\" occurrences, exit 0. All heartbeat-owned tests explicitly confirmed PASS: test_ha_heartbeat_committed.c's 8 tests (test_observe_advances_on_higher, test_observe_noop_on_equal_or_lower, test_observe_noop_on_zero, test_observe_no_reconcile_no_other_mutation, test_consume_term_valid_advances, test_consume_stale_term_ignored, test_consume_primary_does_not_consume, test_consume_null_or_zero_noop) and test_ha_vote_floor_heartbeat.c's 4 tests (test_heartbeat_raises_vote_floor_to_max, test_candidate_below_raised_floor_rejected, test_candidate_at_or_above_raised_floor_accepted, test_lower_heartbeat_never_lowers_floor). TBD/TODO grep on request.c/request.h/ha.c/ha.h/cluster.c/both test files: only pre-existing, already-flagged-unrelated hits (ha.c:1844 ha-term-safety stub; cluster.c group-building and config-generation TODOs that predate and are untouched by this feature). Secrets grep clean.\n\nINTEGRATION (cross-task wiring confirmed by direct code read against design.md's Architecture + File Structure Plan): producer -- primary_heartbeat_fn (ha.c) calls xce_ha_heartbeat_req(ep, ha->term, persist.committed_lsn, ha->committed_config_tag) (request.h:33 decl, request.c:1257 impl) instead of bare xce_nop_req, sending OP_nop with a heartbeatmsg body; consumer -- cluster.c's OP_nop handler (~line 300-375) checks request.body_which == Request_body_heartbeatmsg and calls ha_consume_heartbeat_committed(c, ...) (ha.c:8090-8170), which: (1) gates out CTRL_PRIMARY nodes (Req 3.4), (2) runs the term-guard via the raft library's real UpdateTerm path (zero-entry ha_consume_append_entries call, not a hand-rolled check) and rejects hb->term < myterm (Req 3.2), (3) calls the reconcile-free monotonic updater ha_observe_heartbeat_committed (ha.c:8033, ha.h:1327) which advances last_known_committed_lsn/_config_tag only on strictly-greater LSN with no WAL truncation and no ha_rejoin_reconcile call (Req 2.1/2.2/3.1/3.3/4.2), and (4) calls ha_standby_advance_committed_and_apply -- a later hardening addition (commit 40f510fb, 2026-07-11, bug-1433/1434 fix) that drives this node's own committed_lsn + apply pump from the heartbeat-learned position; this call is additive to the design's Component boundary, not a violation (design.md's Allowed Dependencies already cover read/write of last_known_committed_* and ha->term). Both _Atomic migration writers confirmed: ha.c:8048 (election path) and ha_convergence.c:368 (cluster_apply_ctrlhello_reply join path), both atomic_store_explicit. ha.h declares both new functions (ha_observe_heartbeat_committed:1327, ha_consume_heartbeat_committed:1346) plus the _Atomic field (ha.h:185). Vote-floor formula (ha.c:2570, max(durable_floor, last_known_committed_lsn)) confirmed unchanged -- this spec only makes its input fresher, per design.md's explicit Out-of-Boundary note. All 6 implementation commits (f4327ce, 282a82f, 8bfa861, 90e7179, d534f83, 3b46562) plus the later hardening commit (40f510fb) confirmed present in git history.\n\nCOVERAGE: all 4 requirement sections map to completed tasks with no gaps -- Req 1 (producer) -> tasks 1.2, 3.1; Req 2 (monotonicity) -> task 2.1; Req 3 (advisory-only/term-guard/no-reconcile) -> tasks 2.1, 2.2, 3.2; Req 4 (liveness/no-regression) -> tasks 2.1, 2.2, 4.1. File Structure Plan MINOR drift (non-blocking): design.md's New Files section only names test_ha_heartbeat_committed.c; the vote-floor safety integration test landed as a second file, test_ha_vote_floor_heartbeat.c (both registered in controller_test.c + controller/meson.build) -- functionally equivalent/additive coverage of the same Req 3.2 property design.md calls for, same category of drift already accepted on the ha-wal-replication sibling. design.md's ha.h path is written as src/controller/include/ha.h; actual file is src/controller/ha.h (doc-only path drift, already flagged in tasks.md's own Implementation Notes as a non-blocking learning, not re-litigated here).\n\nRUNTIME LIVENESS (the blocking check, given the bug-1447 sibling finding and this being a retry of a crashed prior attempt): ran the canonical /xce-test-ha SEQUENTIAL 10-round soak (fresh session binaries) against a real ext4 deploy dir (~/tmp/xce-ha-heartbeat-committed-soak, confirmed ext4 via df -T before the run, cleaned up after). Result: PASS -- 10/10 rounds, 33 jobs submitted, zero committed-job loss, all 3 nodes (ctrl-1, ctrl-2, ctrl-3) served as primary at least once (leadership-coverage maneuver satisfied: ctrl-1 rounds 1-4/6-9, ctrl-2 rounds 1-4/7-10, ctrl-3 rounds 5/6/10). Explicitly grepped the full soak log and all per-node logs for bug-1447's exact signature (AddressSanitizer, DEADLYSIGNAL, SEGV, ssl3_read_bytes, handle_ssl_handshake, io_worker): zero matches across all six markers. Also grepped broadly for any other crash/abort signature (Segmentation fault, core dumped, Aborted, double free, heap-buffer-overflow, use-after-free): zero matches. This does not prove bug-1447 is fixed or absent globally (it is a rare, timing-dependent race in a different subsystem, raft-drive-async's reconnect-offload path, which this spec's boundary does not touch) -- but it does establish that this spec's own heartbeat-borne committed-propagation path runs clean through a full supported failover lifecycle, satisfying this spec's own regression contract (Req 4).\n\nOWNERSHIP: bug-1447 remains raft-drive-async/netio.c territory (see that spec's own NO-GO record); noted here only as a risk actively checked for and ruled out in this run, not adopted as this spec's blocker. No new bugs found or logged during this validation pass; the prior silent crash (~4 hours before this retry) left no diagnosable artifacts and could not be attributed to a specific cause -- treated as an environment/harness loss, not a product regression, since this clean retry reproduced nothing.", + "known_limitations": [ + "bug-1447 (open, NOT-FIXED, owned by raft-drive-async / src/lib/netio.c, not this spec): a timing-dependent ASAN SEGV in handle_ssl_handshake/io_worker triggered by raft-drive-async's own reconnect-offload cadence. Did not reproduce in this spec's clean 10/10 run, but as a shared-codebase race condition it cannot be ruled out for all future runs by one clean pass. No action item for ha-heartbeat-committed-propagation; tracked entirely under raft-drive-async's remediation.", + "The prior validation attempt (~4 hours before this retry) crashed silently mid-soak with no logs or diagnosable artifacts (only an empty stale deploy-dir marker was found). Could not be root-caused or attributed to a product bug; this retry's clean, fully-instrumented 10/10 soak found nothing reproducing it, so it is treated as a one-off environment/harness loss rather than a regression, but it was not positively explained.", + "File Structure Plan drift (non-blocking, doc-only): design.md's New Files section names only test_ha_heartbeat_committed.c; the vote-floor safety integration test landed as a second registered file, test_ha_vote_floor_heartbeat.c. Coverage is equivalent/additive; a future doc pass could update design.md to match.", + "design.md's File Structure Plan lists the ha.h path as src/controller/include/ha.h; the actual file is src/controller/ha.h. Cosmetic path drift already flagged in tasks.md's own Implementation Notes, not a functional gap." + ] } } diff --git a/.kiro/specs/ha-prevote/spec.json b/.kiro/specs/ha-prevote/spec.json index 47cad220..c0d8957c 100644 --- a/.kiro/specs/ha-prevote/spec.json +++ b/.kiro/specs/ha-prevote/spec.json @@ -1,9 +1,9 @@ { "feature_name": "ha-prevote", "created_at": "2026-06-15T23:45:43Z", - "updated_at": "2026-06-16T00:55:00Z", + "updated_at": "2026-07-18T09:55:00Z", "language": "en", - "phase": "tasks-generated", + "phase": "implementation-complete", "approvals": { "requirements": { "generated": true, @@ -18,5 +18,21 @@ "approved": true } }, - "ready_for_implementation": true + "ready_for_implementation": true, + "implementation": { + "completed": true, + "tasks_done": "1.1, 1.2, 2.1, 2.2, 2.3, 3.1, 4.1, 4.2, 5.1, 5.2, 5.3, 6.1 (all 12 leaf tasks; the 6 unchecked tasks.md group-header lines are bookkeeping artifacts, not work -- see research.md's 2026-06-18 Gap Re-Audit)", + "notes": "PreVote phase (Raft dissertation Sec 9.6) added ahead of the failover trigger: raft_begin_pre_vote/raft_on_pre_vote_request/raft_on_pre_vote_reply (raft.c, pure -- no term/voted_for/role mutation, no persist) plus ha_begin_failover/ha_start_prevote_async/pre_vote_reply_dispatch/ha_enter_real_election (ha.c) orchestrating the round and deferring truncate+CTRL_ELECTING to PreVote success. Original implementation landed 2026-06-16 (commits 0d66026..e6b234a, controller suite 563/0 clean); research.md's own RN-1 TLA+/verification-checklist argument and 2026-06-18 gap re-audit closed all 25 MET + 2 preserved-unchanged criteria with zero gaps. Substantial hardening landed on top since: bug-605 (dispatch-gate wiring), bug-606 (per-node failover stagger to avoid lock-step split vote), bug-660 (dead-primary fresh-heartbeat convergence), bug-1421/1424 (PreVote-reply registration on connect/monitor failure), bug-1430 (UAF CAS fix in ha_prevote_round_timeout_fn), bug-1435 (raft_lock serialization), task 5.7 jittered [T,2T] election-timeout backoff, and ha-elect-ctx-lifecycle's reference-counted elect_ctx carrier (commit 6529dcf, replacing the deliberate per-round leak this spec's tasks.md Implementation Notes had carried forward as a known, deferred limitation -- now resolved by that sibling spec, itself pending its own validate-impl pass)." + }, + "validate_impl": { + "decision": "GO", + "validated_at": "2026-07-18T09:55:00Z", + "note": "Cross-task/feature-level validation of a spec whose 18/18 tasks.md checkboxes (12 leaf + 6 group-header) had been complete since 2026-06-16, but whose spec.json phase was never advanced past tasks-generated (same administrative-oversight pattern as several sibling specs validated earlier today: ha-wal-replication, ha-heartbeat-committed-propagation). Ran immediately after raft-drive-async (NO-GO, bug-1447 ASAN SEGV in src/lib/netio.c SSL handshake, round 9/10 of its own sequential soak) and after ha-wal-replication / ha-heartbeat-committed-propagation (both GO, clean 10/10 soaks) -- extra weight was given to reproducing or ruling out bug-1447's signature in this spec's own live soak, especially since round 9 is the exact round bug-1447 hit.\n\nCROSS-TASK CONSISTENCY / DESIGN ALIGNMENT (codegraph-verified against design.md's Architecture, Boundary Commitments, and Requirements Traceability table, plus research.md's own 2026-06-18 gap re-audit which independently mapped all 27 acceptance criteria to file:line evidence): confirmed live on-disk source for raft_begin_pre_vote/raft_on_pre_vote_request/raft_on_pre_vote_reply (src/lib/raft/raft.c) and ha_begin_failover/ha_start_prevote_async/pre_vote_reply_dispatch/ha_enter_real_election (src/controller/ha.c) all still present with the designed pure/side-effect-free shape (zero term/voted_for/role mutation, zero on_persist calls in the three raft-library functions -- verified by direct source read, not just test titles). ha_dispatch_gate (ha.c:3794) admits OPCode_preVoteRequest/preVoteReply in CTRL_INIT/CTRL_ELECTING/CTRL_STANDBY exactly as bug-605's fix requires. Confirmed the dependency direction (capnp/types -> raft library (pure) -> controller adapter -> dispatch/transport) still holds with no upward imports. One legitimate, expected architectural evolution beyond the original design.md snapshot: raft_on_pre_vote_request's signature grew cand_last_log_term/cand_last_log_index parameters and now runs the (lastLogTerm,lastLogIndex) raft_log_ok freshness gate INSIDE the raft library itself, rather than the dispatcher applying freshness before the call as design.md originally specified -- this is the G6 vote-freshness closure from the later raft-commit-safety spec (own boundary, own validation), layered onto this spec's foundation; functionally strictly stronger, not a regression, and does not change any of this spec's own requirements or invariants. research.md documents one further non-blocking, functionally-equivalent routing drift (PreVote replies route via peer_ctrl_msg_handler's role+pre_vote_in_progress state check rather than a dedicated cluster.c OPCode_preVoteReply switch case as task 5.1 literally phrased it) -- already flagged in that file, not re-litigated here.\n\nMECHANICAL: Fresh ASAN build (nix develop + xce-build.sh controller asan) -- zero compiler/linker errors, binaries confirmed with session-fresh mtimes. Both authoritative standalone binaries run directly with ASAN_OPTIONS=detect_leaks=0 (per this project's convention, authoritative over meson's aggregate line): xcelib_test (raft library) 293 Tests, 0 Failures, 0 Ignored, OK, exit 0 -- includes 24 dedicated pre_vote-tagged RUN_TEST entries in test_raft.c (decision/tally purity, quorum WON/LOST/PENDING, duplicate-reply idempotency, out-of-range-peer guard, 5-node quorum, and 4 explicit invariant assertions incl. test_invariant_prevote_sequence_pure_no_persist and test_invariant_prevote_won_grants_no_leadership). controller_unit_test 972 Tests, 0 Failures, 0 Ignored, OK, exit 0 -- includes 28 dedicated PreVote-tagged tests across test_ha_prevote.c (12: start/no-mutation, majority-grant, no-majority-rearm, round-timeout-as-loss, reusable-after-loss, pending-noop, reply-over-real-transport, frozen-rejoin no-op x2, genuine-failover, heartbeat-mid-round-cancel, lost-then-later-elects), test_ha_failover_prevote.c (6: trigger rewire rdhup/liveness->PreVote not election, heartbeat-cancel, WON->real-election, config-less-node guard), test_ha_vote_gate.c (8: primary-role-denies, standby-fresh-heartbeat-denies, catch-up-grants-when-primary-unreachable, catch-up-denies-while-primary-live, stale-freshness-denies, stale-term-denies-no-adoption, granted-when-stickiness-lapsed, dead-primary-grants-despite-fresh-activity -- this last one is bug-660's own regression test), test_ha_role.c (1: dispatch-gate admits PreVote in voter roles), and test_ha_elect_ctx_lifecycle.c (1: 20 PreVote rounds x 2 votes with no carrier leak -- the sibling ha-elect-ctx-lifecycle spec's own regression test, confirming its refcounted-carrier fix holds under repeated PreVote churn). Zero :FAIL anywhere in either log, zero ASAN/crash signatures (AddressSanitizer, heap-buffer-overflow, use-after-free, double-free, SEGV) in either log. TBD/TODO/FIXME/HACK/XXX grep on the PreVote-boundary files (raft.c, raft.h, ha.c, ha.h, cluster.c, controller.capnp, test_ha_prevote.c, test_ha_failover_prevote.c, test_raft.c, test_ha_vote_gate.c): zero PreVote-related matches. Secrets grep: clean.\n\nRUNTIME LIVENESS (the blocking check, given the bug-1447 sibling finding and this spec's own live-election subject matter -- PreVote specifically governs election behavior under partition/unreachability): ran the canonical /xce-test-ha SEQUENTIAL 10-round soak (fresh session binaries) against a real ext4 deploy dir (~/tmp/xce-ha-prevote-soak, confirmed ext4 via df -T before the run, cleaned up after). Result: PASS -- 10/10 rounds, 33 jobs submitted, zero committed-job loss, zero walcompare divergence at any round boundary, all 3 nodes (ctrl-1, ctrl-2, ctrl-3) served as primary at least once (leadership-coverage maneuver satisfied by round 4). Term progressed 2->13 over 10 genuine kills -- climbing by exactly +1 per round in 9 of 10 rounds (clean single-jump elections, no runaway multi-round election-storm churn, exactly the disruption-suppression behavior this spec exists to produce) and +2 in exactly one round (round 9, term 10->12), coinciding with the log's single live PreVote DENIED event (candidate=ctrl-2 term=8 reason=freshness) -- a legitimate freshness-gate denial of a behind candidate, not a fault. Live PreVote activity confirmed directly from the per-node logs (not just inferred from job-loss absence): 13 PreVote round WON events across the three nodes' logs (6+5+2), 1 explicit PreVote DENIED (reason=freshness) log line carrying candidate id + reason exactly as Req 7.2 requires, and 2 'primary heartbeat arrived mid-PreVote -- cancelling the round' events (Req 4.2's mid-round-cancel path) -- meaning this run incidentally live-exercised the frozen-then-resumed re-attach path in addition to the genuine-failover path the sequential kill schedule directly drives, without needing a dedicated SIGSTOP-based scenario (design.md's Testing Strategy already covers the frozen-then-resumed case at the controller integration-test level via test_ha_prevote.c/test_ha_failover_prevote.c, run above; it does not call for a live SIGSTOP soak scenario, and none exists in run-ha-soak.sh's mode set -- confirmed by inspection). Explicitly grepped the full soak driver log AND all three per-node controller logs (11268/12012/11515 lines) for bug-1447's exact signature (AddressSanitizer, DEADLYSIGNAL, SEGV, ssl3_read_bytes, handle_ssl_handshake, io_worker): zero matches anywhere, including at round 9 -- the exact round bug-1447 hit in raft-drive-async's own soak. Also grepped broadly for any other crash/abort signature (Segmentation fault, core dumped, Aborted, double free, heap-buffer-overflow, use-after-free): zero matches. This does not prove bug-1447 is fixed or globally absent (it is a rare, timing-dependent race in raft-drive-async's own reconnect-offload cadence, a different subsystem this spec's boundary does not touch) -- but it does establish that this spec's own PreVote-gated election path runs clean through a full 10-round supported failover lifecycle with live PreVote activity on every round, satisfying this spec's own regression contract (Req 5.4, 4.x).\n\nOWNERSHIP: bug-1447 remains raft-drive-async/netio.c territory (see that spec's own NO-GO record, bug-1447 in buglog.json); noted here only as a risk actively checked for and ruled out in this run, not adopted as this spec's blocker. No new bugs found or fixed during this validation pass -- the tree's PreVote-adjacent bug history (605, 606, 660, 1421, 1424, 1430, 1435) was reviewed and every fix confirmed present and correct in the current source (e.g. bug-1430's CAS-based UAF fix in ha_prevote_round_timeout_fn read directly from ha.c:7216-7240).", + "known_limitations": [ + "bug-1447 (open, NOT-FIXED, owned by raft-drive-async / src/lib/netio.c, not this spec): a timing-dependent ASAN SEGV in handle_ssl_handshake/io_worker triggered by raft-drive-async's own reconnect-offload cadence. Did not reproduce in this spec's clean 10/10 run (including at round 9, the exact round it hit for raft-drive-async), but as a shared-codebase race condition it cannot be ruled out for all future runs by one clean pass. No action item for ha-prevote; tracked entirely under raft-drive-async's remediation.", + "The deliberate per-round ha_elect_ctx_t carrier leak that this spec's own tasks.md Implementation Notes (task 4.2) had explicitly deferred to a follow-up spec is now resolved in the current tree by ha-elect-ctx-lifecycle (commit 6529dcf, reference-counted carrier via atomic_exchange + release-on-last-ref in ha_start_prevote_async) -- confirmed present in the current ha.c source and covered by a live-passing regression test (test_many_prevote_rounds_no_carrier_leak, 20 PreVote rounds x 2 votes, no leak). That sibling spec's own spec.json is still phase=tasks-generated pending its own validate-impl pass (scheduled to run after this one in the same worktree); this note records that ha-prevote's own known-limitation is already satisfied in-tree regardless of that sibling's administrative validation status.", + "design.md's original Service Interface signature for raft_on_pre_vote_request (2 params: from_idx, cand_term) is now stale documentation -- the shipped function additionally takes cand_last_log_term/cand_last_log_index and runs the (lastLogTerm,lastLogIndex) freshness gate inside the raft library itself (the later raft-commit-safety spec's G6 vote-freshness closure, layered onto this spec's foundation). Functionally strictly stronger than the original design, not a regression; a future doc pass could update design.md's Service Interface block to match the shipped signature.", + "research.md's own file already documents one non-blocking, functionally-equivalent routing drift: PreVote replies are routed via peer_ctrl_msg_handler's role+pre_vote_in_progress state check rather than a dedicated cluster.c OPCode_preVoteReply switch case as task 5.1's text literally phrased it. Not re-verified independently in this pass beyond confirming the request-side opcode (preVoteReply @39) is still emitted and the reply still reaches the tally live (confirmed via the soak's 13 live PreVote round WON events)." + ] + } } diff --git a/.kiro/specs/ha-wal-replication/spec.json b/.kiro/specs/ha-wal-replication/spec.json index 2745a539..21415154 100644 --- a/.kiro/specs/ha-wal-replication/spec.json +++ b/.kiro/specs/ha-wal-replication/spec.json @@ -1,9 +1,9 @@ { "feature_name": "ha-wal-replication", "created_at": "2026-06-10T17:00:00Z", - "updated_at": "2026-06-10T17:15:00Z", + "updated_at": "2026-07-18T02:05:00Z", "language": "en", - "phase": "tasks-generated", + "phase": "implementation-complete", "approvals": { "requirements": { "generated": true, @@ -18,5 +18,15 @@ "approved": true } }, - "ready_for_implementation": true + "ready_for_implementation": true, + "validate_impl": { + "decision": "GO", + "validated_at": "2026-07-18T02:05:00Z", + "note": "Prior state: all 20 tasks (1.1-5.5) were [x] in tasks.md but phase was stuck at \"tasks-generated\" with no validate_impl record (same administrative-oversight pattern as ha-entity-snapshots / ha-commit-durability). Real implementation commits exist under the ha-wal-replication tag (18e351e repl_broadcast send, 81dbc4d registration bridge, 566c91c standby ACK reply, 861b746 primary ACK handler, ab270fa/e2fba08 tests, plus two later hardening commits: 708f227 durable-before-ACK, 64ab210 CRC-chain ownership fix, 46bd207 test-seed fix) -- this pass focused on cross-task/feature-level checks per kiro-validate-impl scope, run immediately after a sibling spec (raft-drive-async) validated NO-GO on a shared-codebase ASAN crash (bug-1447), so extra weight was given to reproducing or ruling out that crash signature in this spec's own live soak.\n\nMECHANICAL: Fresh ASAN build (nix develop + xce-build.sh controller asan) clean, zero compiler errors; meson Ok:3 Fail:0. Standalone controller_unit_test binary run directly (authoritative over meson's aggregate line per this project's own convention): 972 Tests, 0 Failures, 0 Ignored, OK -- zero \":FAIL:\" occurrences. WAL-replication-owned tests explicitly confirmed passing: test_repl_add_standby_max, test_repl_add_standby_idempotent, test_repl_registration_lifecycle, test_broadcast_one_standby_sends_entry, test_wal_ack_reply_updates_acked_lsn_and_signals, test_repl_broadcast_rejects_checkpoint, test_pin_repl_broadcast_full_fanout_skips_empty_and_catchup, test_repl_on_ack_wired_updates_watermark_and_commits. TBD/TODO grep on repl.c/ha.c/cluster.c/test_repl_unit.c: only the pre-existing, already-flagged-unrelated ha-term-safety stub at ha.c:1844. Secrets grep clean.\n\nINTEGRATION (cross-task wiring confirmed by direct code read, matching design.md's architecture end-to-end): standby registration on connect (ha.c:1556 raft_cb_become_leader, ha.c:8374 on_standby_connected) -> repl_add_standby; deregistration on disconnect (ha.c:6787 vote_ep_err_handler, ha.c:8422 standby_ep_err_handler_primary, ha.c:8814 eviction path) -> repl_remove_standby; standby applies entry then sends WalAckReply (ha.c:2601 ha_send_wal_ack_reply, wired from ha_apply_wal_catchup_entry, called from cluster.c:192 handle_wal_catchup_entry with ep+c passed through per task 3.1); primary ACK dispatch (ha.c:5928-5940 repl_ack_decode_and_dispatch's RequestReply_data_walackreplymsg arm -> repl_on_ack -> acked_lsn update + ack_cond signal, repl.c:190-226). repl_broadcast (repl.c:236-316) is a real TCP send via xce_wal_batch_req_raw, not a stub -- catch-up skip and MAX_PENDING_ACKS eviction guards both present and exercised by tests. Note: repl_broadcast's live signature now also carries prev_log_term (a later ha-term-safety-era addition for AppendEntries term-safety) beyond what design.md's Component 1 describes -- additive, not a boundary violation; the core WAL-transmission contract this spec committed to is unchanged.\n\nCOVERAGE: all 5 requirement sections map to completed tasks with no gaps -- Req 1 (Registration) -> tasks 2.1-2.3; Req 2 (Broadcast) -> tasks 1.1-1.2; Req 3 (Standby Receipt/ACK) -> tasks 3.1-3.3; Req 4 (Majority Commit) -> tasks 4.1-4.2; Req 5 (Correctness/Observability) -> tasks 5.1-5.5, with 5.1's in-order-apply guarantee satisfied by the adjacent bug-850 contiguity guard in ha_apply_single_entry (ha.c, returns -EAGAIN on an ahead-of-tail entry, silently deferred to catch-up) -- correctly relied upon rather than re-implemented, matching design.md's Out of Boundary note that catch-up streaming is ha-repl-hardening's territory. Minor logging deviation: repl_broadcast's per-send trace uses TRACE not literally \"DEBUG3\" (Req 5.2) -- cosmetic, not a functional gap.\n\nDESIGN: Architecture matches design.md's wire-protocol diagram exactly (repl_broadcast -> xce_wal_batch_req_raw -> standby apply -> WalAckReply -> repl_on_ack -> majority_ack_cond). File Structure Plan MISMATCH (non-blocking): design.md specified CREATE src/controller/testsuite/test_ha_wal_repl.c; the actual implementation consolidated the same test coverage into the pre-existing test_repl_unit.c (plus test_p2_integration.c, test_commit_engine.c, test_convergence_verification.c for cross-spec-integration pins) instead of a new file -- functionally equivalent, all Boundary Commitments (xce_wal_batch_req_raw single-entry mode, existing repl_add_standby/repl_remove_standby API, existing WalAckReply struct, no schema changes) held. Blocked tasks: none.\n\nRUNTIME LIVENESS (the blocking check, given the sibling bug-1447 finding): ran the canonical /xce-test-ha SEQUENTIAL 10-round soak (fresh binaries from this session's build) against a real ext4 deploy dir (/home/claude/tmp/xce-ha-deploy, cleaned up after). Result: PASS -- 10/10 rounds, 33 jobs submitted, zero committed-job loss, walcompare committed-prefix consistent every round, all 3 nodes (ctrl-1/ctrl-2/ctrl-3) served as primary at least once (leadership-coverage maneuver satisfied). Explicitly grepped all per-node logs for AddressSanitizer/DEADLYSIGNAL/SEGV/ssl3_read_bytes/handle_ssl_handshake/io_worker (bug-1447's exact signature): zero matches -- the sibling's netio.c SSL-handshake-race crash did NOT reproduce in this run. This does not prove bug-1447 is fixed or absent (it is a rare, timing-dependent race that took the sibling until round 9/10 to trigger once, and ha-wal-replication's own boundary does not touch the raft-drive-async reconnect-offload path that triggered it) -- but it does establish that ha-wal-replication's own WAL-transmission/ACK/commit path runs clean through a full supported failover lifecycle on the current codebase, which is this spec's own regression contract (Req 4, 5.5).\n\nOWNERSHIP: bug-1447 remains raft-drive-async/netio.c territory (see that spec's own NO-GO record); it is noted here only as a risk that was actively checked for and ruled out in this run, not adopted as this spec's blocker.", + "known_limitations": [ + "bug-1447 (open, NOT-FIXED, owned by raft-drive-async / src/lib/netio.c, not this spec): a timing-dependent ASAN SEGV in handle_ssl_handshake/io_worker triggered once by raft-drive-async's own reconnect-offload cadence at round 9/10 of its sequential soak. Did not reproduce in this spec's clean 10/10 run, but as a shared-codebase race condition it cannot be ruled out for all future runs by one clean pass. No action item for ha-wal-replication; tracked entirely under raft-drive-async's remediation.", + "File Structure Plan drift (non-blocking): design.md specified a new test_ha_wal_repl.c; actual tests landed in test_repl_unit.c (+ cross-spec integration pins in test_p2_integration.c/test_commit_engine.c/test_convergence_verification.c). Coverage is equivalent; a future doc pass could update design.md's File Structure Plan to match, but this is cosmetic.", + "Req 5.2's DEBUG3-level broadcast log is implemented as TRACE level in the current logging framework's naming, not a literal \"DEBUG3\" call site -- functionally satisfies the intent (per-send LSN/opcode visibility at a low log level) but is a naming mismatch against the requirement's literal wording." + ] + } } diff --git a/.kiro/specs/libxce-failover/spec.json b/.kiro/specs/libxce-failover/spec.json index 12421f5f..d03d414b 100644 --- a/.kiro/specs/libxce-failover/spec.json +++ b/.kiro/specs/libxce-failover/spec.json @@ -1,9 +1,9 @@ { "feature_name": "libxce-failover", "created_at": "2026-06-16T10:30:55Z", - "updated_at": "2026-06-16T10:52:00Z", + "updated_at": "2026-07-18T00:35:00Z", "language": "en", - "phase": "tasks-generated", + "phase": "implementation-complete", "approvals": { "requirements": { "generated": true, @@ -18,5 +18,15 @@ "approved": true } }, - "ready_for_implementation": false + "ready_for_implementation": true, + "blocked_on": [], + "validate_impl": { + "decision": "GO", + "validated_at": "2026-07-18T00:35:00Z", + "note": "First post-implementation validation run -- all 19 tasks were already [x] since 2026-06-18 with ready_for_implementation still false and no blocked_on/explanation; confirmed this was purely an administrative oversight (the flag was never flipped after completion), not a real blocker or supersession. Git log shows 11 dedicated feature commits (foundation through bug-1436 count-aware macro fix) forming the foundational client-failover layer that later HA specs (ha-term-safety, raft-library, controller-status, etc.) build on top of, not a duplicate of them. Full libxce test suite green (123 tests total, 34 libxce-failover-specific: StatusClassifier, EndpointRotator, retry-decision budget/backoff/jitter, ctx-recovery, rejoin-hook, redirect-hint capture, cross-component integration), controller suite green (3/3), build clean (192 targets, zero errors), 8/8 requirement sections mapped to completed tasks, design.md's 5-component architecture (StatusClassifier, EndpointRotator, retry-decision core, ctx-recovery, rejoin-hook) verified wired end-to-end via call-graph trace, all 18 reply-bearing + 3 fire-and-forget libxce ops confirmed wrapped in the failover macros. Live E2E already recorded 2026-07-02 (3-node real-fs cluster, zero job loss). No LOCAL bugs found during this pass; bug-609 (known-dead ctx EINVAL), bug-610 (endpoint_connect SEGV on reconnect), bug-854 (dead-first-master initial discovery), and bug-1436 (row-count misclassified as transport error) were all found and fixed in prior follow-up commits, already present in the validated build.", + "known_limitations": [ + "Per-daemon adoption (worker, scheduler, apiserver master-list config and liveness-safe bounded blocking) is explicitly out of scope by design -- owned by the follow-on specs component-failover-adoption and worker-rejoin-resync, both of which depend on this spec.", + "Fire-and-forget notifications (XCECLI_FF_CALL: notify/heartbeat/nop) have no reply and get only a single-shot best-effort reconnect attempt, not the full retry/backoff loop -- a fact genuinely lost in a failover window is accepted-by-design here; redelivery is owned by worker-rejoin-resync." + ] + } } diff --git a/.kiro/specs/raft-sim-conformance/spec.json b/.kiro/specs/raft-sim-conformance/spec.json index 36fef554..83fa1b2e 100644 --- a/.kiro/specs/raft-sim-conformance/spec.json +++ b/.kiro/specs/raft-sim-conformance/spec.json @@ -3,7 +3,7 @@ "created_at": "2026-06-21T13:00:00Z", "updated_at": "2026-06-21T13:00:00Z", "language": "en", - "phase": "tasks-generated", + "phase": "implementation-complete", "approvals": { "requirements": { "generated": true, @@ -19,5 +19,16 @@ } }, "ready_for_implementation": true, - "note": "Auto-approved (-y mode). Extends src/lib/raft/raft_sim.c into an in-process TLA+ conformance/verification harness driving N library instances through raft.tla Next-action-equivalent transitions (Restart, Timeout, RequestVote, BecomeLeader, ClientRequest, AppendEntries, AdvanceCommitIndex, Receive incl. UpdateTerm + DropStaleResponse, DuplicateMessage, DropMessage) and asserting the four Raft safety invariants (Election Safety, Log Matching, Leader Completeness, State Machine Safety) every step. Seedable/deterministic driver; randomized + targeted scenarios (partitions, restarts, drop/dup/reorder, Figure-8); failure-trace reproduction; CI-runnable target; deliberately-broken-build negative test reproducing a bug-851-class divergence. Boundary: owns src/lib/raft/raft_sim.c + harness/test files ONLY; does NOT modify the protocol implementation (raft.c/raft.h protocol semantics) — it consumes the raft-log-replication + raft-commit-safety entry points/callbacks verbatim. Depends: raft-log-replication, raft-commit-safety." + "note": "Auto-approved (-y mode). Extends src/lib/raft/raft_sim.c into an in-process TLA+ conformance/verification harness driving N library instances through raft.tla Next-action-equivalent transitions (Restart, Timeout, RequestVote, BecomeLeader, ClientRequest, AppendEntries, AdvanceCommitIndex, Receive incl. UpdateTerm + DropStaleResponse, DuplicateMessage, DropMessage) and asserting the four Raft safety invariants (Election Safety, Log Matching, Leader Completeness, State Machine Safety) every step. Seedable/deterministic driver; randomized + targeted scenarios (partitions, restarts, drop/dup/reorder, Figure-8); failure-trace reproduction; CI-runnable target; deliberately-broken-build negative test reproducing a bug-851-class divergence. Boundary: owns src/lib/raft/raft_sim.c + harness/test files ONLY; does NOT modify the protocol implementation (raft.c/raft.h protocol semantics) — it consumes the raft-log-replication + raft-commit-safety entry points/callbacks verbatim. Depends: raft-log-replication, raft-commit-safety.", + "updated_at_tasks": "2026-06-24T16:43:39.427Z", + "validate_impl": { + "decision": "GO", + "validated_at": "2026-07-17T18:00:00Z", + "note": "First post-implementation validation run -- all 23 tasks (1.1-8.1) were already [x] with no prior validate_impl record and spec.json's phase field had never advanced past tasks-generated despite implementation being complete. Full `/xce-build lib` ASAN run: clean compile, 293/293 Unity tests PASS (0 Fail, 0 Ignored), meson 'libxce test' target 1/1 OK in 36.11s, zero AddressSanitizer/LeakSanitizer/UBSan findings anywhere in the log. Every raft-sim conformance test observed by name and PASS: all 4 invariant-checker unit tests (election safety, log matching, leader completeness, state machine safety) both directions, raft_sim_check_all first-failure composition, the 5 targeted scenarios (partition/heal, restart-churn, drop/dup/reorder, Figure-8, stale-rejoin/G5), the M-seed randomized sweep (test_sim_6_1_seed_sweep), the deliberately-broken-build negative test asserting both directions (test_sim_7_2_negative_both_directions), and the optional TLC trace-export test (task 8.1, gated off by default). Boundary audit: grepped every commit subject-tagged feat|test|fix|docs(raft-sim-conformance) across full git history -- touched files are exactly raft.h, raft_sim.c, test_raft.c (+ this spec's own docs); zero raft.c protocol edits, zero controller files. Harness confirmed allocation-free (no malloc/calloc/realloc/free in raft_sim.c), matching the design's ASAN-clean-by-construction claim. Requirements coverage: all 8 requirement sections (1-8, 34 acceptance criteria) map to committed tasks and passing tests; no gaps. Design alignment: File Structure Plan matches actual (raft.h XCE_TESTING-section-only, raft_sim.c, test_raft.c); component graph (message bag, in-memory adapter, seeded scheduler, 4 checkers, trace recorder, scenario drivers, fault switch) all present and named as designed. Cross-task integration: task 4.2's Implementation Notes record an UPSTREAM raft.c defect (bug-1095/1096, Election Safety violation from a non-idempotent vote-reply tally + missing stale-term drop) surfaced by the oracle itself -- correctly reported out-of-boundary rather than masked, and subsequently fixed upstream in raft.c (commit b5b7e01, verified live at raft_on_vote_reply: per-peer vote_replies[] dedup + termterm drop), after which downstream tasks 5.x/6.1/7.2 assert zero violations on the correct build as required. This validation pass also corrected buglog.json's bug-1095 entry, which was stale (said 'NOT FIXED' although the fix has been live since 2026-06-24 per bug-1096 and tasks.md's own notes).", + "known_limitations": [ + "tasks.md task 7.1 describes the negative-test mechanism as a 'compile-time fault switch,' but the shipped implementation (and design.md, corrected in commit ffeb7c2) is a runtime bitflag (cluster.fault_inject / RAFT_SIM_BREAK_TERM_STAMP|TRUNCATE set via raft_sim_set_fault_inject) so both the broken and correct configurations can be asserted within a single test binary per R7.4. Wording drift only in tasks.md's task header; behavior is correct, intentional, and matches design.md.", + "design.md's invariant-checker narrative does not reflect the leader-only commit model correction recorded in tasks.md's own Implementation Notes for tasks 5.1/5.2: a FOLLOWER never advances commit_index/applied[] from AppendEntries (raft_on_append_entries_request does (void)leader_commit -- intended XCE design, follower commit visibility is a controller/heartbeat concern owned by ha-heartbeat-committed-propagation, not this library). Scenario tests correctly assert leader-side commit_index and log convergence rather than follower commit_index equality; design.md's prose was not back-ported to state this explicitly. Documentation-drift note only.", + "raft.h's XCE_TESTING sim section is also extended by a later, adjacent spec (raft-apply-decoupling, commit 5f98d1b, test_sim_apply_decoupling_* tests) building on this harness's scheduler/checkers. This is expected downstream consumption, not a boundary violation of this spec, but a future change to raft-sim-conformance's scheduler/action-record shape should check that spec's tests too." + ] + } }