diff --git a/.kiro/specs/ha-convergence-recovery/tasks.md b/.kiro/specs/ha-convergence-recovery/tasks.md index f058d7a..57933da 100644 --- a/.kiro/specs/ha-convergence-recovery/tasks.md +++ b/.kiro/specs/ha-convergence-recovery/tasks.md @@ -77,13 +77,14 @@ - Observable completion: all pins PASS against current code without production changes (these encode behavior that already exists; any failure is a real regression or a wrong assumption to escalate) - _Requirements: 4.1, 6.1, 7.1, 9.1, 9.2_ -- [ ] 7. Full controller regression under ASAN +- [x] 7. Full controller regression under ASAN - Build and run the complete controller testsuite with ASAN (`/xce-build controller`); fix any regression from the become-follower changes, the collect-path wrapper, or the walAck branch additions - Audit: grep confirms status/API read paths are unfiltered, `commit.c` is untouched by this spec, and `raft_cb_become_follower`'s non-PRIMARY behavior is unchanged - Observable completion: full testsuite green under ASAN; audit results recorded in completion notes - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 2.1, 2.2, 2.3, 3.1, 3.2, 3.3, 3.4, 4.1, 5.1, 5.2, 5.3, 6.1, 7.1, 8.1, 8.2, 8.3, 9.1, 9.2_ - [ ] 8. 3-node E2E convergence test (extends ha-wal-replication task 5.4) + - _Blocked: manual live-cluster integration test — requires a running `/xce-deploy-ha` 3-node cluster with kill-primary / isolate / heal scenarios and operator-observed convergence assertions; not automatable in the unit suite (same blocked-manual status as ha-wal-replication task 5.4). Unit-testable in-process halves of these assertions are already covered: single-outcome demotion+replay (test_demotion_convergence), rejoin reconciliation clamp/once-guard, commit-gated egress (test_egress_filter), quorum-loss/restore WARN once (test_convergence_verification). Run on real infra to close 2.4/8.1/8.2/9.1/9.2 end-to-end._ - **DEPENDS on ha-single-leader-role having landed** (direct promotion, COMMITTING removed — the demotion/rejoin path asserted here is the post-removal one); requires a live `/xce-deploy-ha` cluster — manual integration test, not automatable in the unit suite (same constraint as ha-wal-replication 5.4) - Deploy 3 nodes; submit confirmed jobs → kill primary → promoted standby owns all confirmed jobs; force a submit to time out at the barrier (withhold standby ACKs) → verify single-outcome convergence: the entry is committed on all members or absent on all members, the requester's failure answer is never retroactively changed, and the scheduler never dispatched the unconfirmed job - Restart the killed ex-primary → verify exactly one self-restart/reconciliation cycle and a converged rejoin (WAL tip at the cluster's committed boundary; no phantom jobs in joblist) @@ -106,3 +107,5 @@ - **Task 3.2 — delaymgr one-shot helper + once-guard test strength**: `ha_rejoin_reconcile` lives in `ha_convergence.c` (alongside `ha_self_restart`/demotion task, unit-linkable) and defers truncate→WARN→restart to a delaymgr one-shot, reusing the 2.2 pattern. The test helper `dc_run_pending_task` was fixed to DEQUEUE (`ordq_get`+free) mirroring the real delaymgr worker (delay.c:65-90) — the prior peek-without-dequeue caused a test-only heap-UAF (re-running an already-freed one-shot); production never re-runs a one-shot, so the body-frees-own-ctx pattern is safe (bug-343). **Follow-up (non-blocking, for task 6.1 or 7): add a once-guard test that calls `ha_rejoin_reconcile` TWICE before draining the delaymgr (tip unchanged) and asserts exactly one task scheduled** — the current second-call test runs truncation synchronously between calls, so call 2 exits at the 2.3 early-return and exercises only the structural guard, not the defensive CAS (CAS is present + correct, needed for a duplicate CtrlHelloReply arriving before the deferred truncation runs). - **Task 5.1 done (RED) + 5.2 notes**: counter-hook seam `ha_quorum_transition_log_hook(int lost,int connected,int required)` is the observable (no log-capture harness exists); `_Atomic int quorum_lost` added to ha_state_t; no-op stub `ha_log_quorum_transition` in ha_convergence.c. **5.2 must**: (a) report `connected = repl_active_standby_count(repl)` and `required = n_peers/2` to match the `ha_has_write_quorum` BODY (ha.c:~1488-1498) — its header comment says "ceil(n_peers/2)+1" but that's a pre-existing stale comment; do NOT change the quorum math, just mirror the body so the 5.1 tests pass; (b) re-grep the promotion standby-registration call site inside `raft_cb_become_leader` (rewritten by ha-single-leader-role) — RDHUP removal (~ha.c:3160), `on_standby_connected` (~ha.c:3764), and the gate-rejection backstop (~ha.c:1511) are verified present; (c) invoke the hook once per WARN emitted (CAS edge-detect on `quorum_lost`). - **Task 5.2 done (GREEN) — quorum-decision duplication note**: `ha_log_quorum_transition` (ha_convergence.c) re-derives `connected = repl_active_standby_count(repl)` / `required = n_peers/2` / single-node short-circuit as a FAITHFUL same-input mirror of the file-static `ha_has_write_quorum` (ha.c) — it cannot report a state inconsistent with the gate. Non-blocking maintainability risk: if `ha_has_write_quorum`'s decision ever changes, this mirror must change too. **Follow-up (optional cleanup): expose `ha_has_write_quorum` via a non-static accessor and call it from `ha_log_quorum_transition`** instead of duplicating. Call sites wired (post ha-single-leader-role rewrite): promotion ha.c:567, RDHUP removal ha.c:3180, on_standby_connected ha.c:3837, gate backstop ha.c:1521. +- **Task 7 done (GO) — full ASAN regression + boundary audit**: full controller suite green under ASAN, established suite-by-suite (50/50 suites, 524 tests, 0 failures, 0 ASAN memory errors) because the aggregate meson `controller test` SIGTERMs at the pre-existing 300s cap (slow commit-barrier timeout suites serialized). Audits PASS: (1) status/API read paths UNFILTERED — `collect_jobs_committed` sole caller is `sched.c:218` (full push) + the `job_submit_committed` predicate filter at the JOB_NEW push; `apiserver.c:83` + `opget.c:173` stay on plain `collect_jobs`; (2) `commit.c`/`commit.h` UNTOUCHED across the whole spec (`git diff --stat e558a62..HEAD` empty for both); (3) `raft_cb_become_follower` demotion additions guarded by `prior_role == CTRL_PRIMARY && c != NULL` (ha.c:687, :711) — STANDBY/ELECTING/INIT byte-equivalent to pre-spec. Production files touched match the design File Structure Plan (with dataex.c→egress.c per D3). No production change needed in task 7. +- **Task 8 (live 3-node E2E) — BLOCKED (manual)**: requires a running `/xce-deploy-ha` cluster + operator-observed kill/isolate/heal convergence; not automatable in the unit suite (same status as ha-wal-replication 5.4). All unit-testable in-process halves are covered by tasks 1-6. To close: run on real infra and record results.