From 3609f468c5302cc30b159f619ca2457aba5a1c21 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Sat, 27 Jun 2026 03:16:17 +0000 Subject: [PATCH] chore(raft-ha-core-safety): init spec scaffold (HA-core failover-safety gate) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New spec for the pre-existing HA-core consistency/convergence failures the xce-test-ha soak reproduces under chaotic failover — committed-prefix divergence (bug-851, bug-795/828), standby-rejoin/reconverge failure (bug-1042), and round-N election stall — all baseline-confirmed at commit 21e26a1 (independent of raft-drive-async). This spec is the gate that must close before raft-drive-async flips its dispatch flags ON in production. Scaffold only (spec.json + requirements.md project description); requirements generation is the next phase. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2 --- .../specs/raft-ha-core-safety/requirements.md | 42 +++++++++++++++++++ .kiro/specs/raft-ha-core-safety/spec.json | 22 ++++++++++ .wolf/memory.md | 5 +++ 3 files changed, 69 insertions(+) create mode 100644 .kiro/specs/raft-ha-core-safety/requirements.md create mode 100644 .kiro/specs/raft-ha-core-safety/spec.json diff --git a/.kiro/specs/raft-ha-core-safety/requirements.md b/.kiro/specs/raft-ha-core-safety/requirements.md new file mode 100644 index 0000000..f8904f2 --- /dev/null +++ b/.kiro/specs/raft-ha-core-safety/requirements.md @@ -0,0 +1,42 @@ +# Requirements Document + +## Project Description (Input) + +**raft-ha-core-safety** — Fix the pre-existing HA-core consistency and convergence failures that the live failover soak (`xce-test-ha`) reproduces under repeated/chaotic primary failover. This work is **independent of, and a gate that must close before, the raft-drive-async production flag-flip**. + +### Who has the problem +Operators and clients of an XCE controller HA cluster relying on Raft failover to keep committed jobs safe and the cluster available. Under repeated/chaotic primary churn the cluster can silently diverge committed state and/or fail to recover a healthy quorum. + +### Current situation (all baseline-reproduced) +All three symptoms reproduce on the **pre-raft-drive-async baseline (commit `21e26a1`)** — confirmed by a baseline-vs-current failover-soak comparison — so they are **NOT** caused by the raft-drive-async async-drive/offload work (which is complete and dormant). They are pre-existing HA-core bugs: + +1. **Committed-prefix divergence (bug-851, bug-795/828 family).** Under chaotic failover, `walcompare` reports `FAIL: committed-prefix divergence at LSN N` — one member is **MISSING a committed WAL entry that the majority holds**, and keeps a divergent committed prefix silently. Contributing root causes already identified: WAL `hdr.term` left `0` (entry term never set from the writing leader's term), two-leaders-per-term windows, and no per-entry `(term,lsn)` identity or reconcile so a member cannot detect/repair a divergent committed prefix. The bug-795/828 family (committed-job/prefix loss under repeated and total-outage failover) is part of this cluster. + +2. **Standby-rejoin / reconvergence failure (bug-1042 family).** A restarted, previously-downed node intermittently fails to bring the cluster back to a healthy **3-of-3** within bound — even when it reaches the leader's log tip at the raft `match` level — due to post-recovery `log_len`/applied-state desync and catch-up apply gaps. + +3. **Round-N election stall.** After several rapid kill/restart rounds the cluster intermittently fails to elect a new primary after a primary kill (no `CTRL_PRIMARY` at `term > prev` within the failover window) — a liveness regression under churn. + +### What should change +Make repeated/chaotic failover **provably safe and live**: + +- **Safety — committed entries never diverge across members.** Enforce a real per-entry `(term, lsn)` commit identity plus a reconcile path so a member can **detect and repair** a divergent committed prefix rather than silently keeping it. Ensure the WAL entry term is always the writing leader's term (never `0`). Close two-leaders-per-term windows. +- **Convergence — a restarted node always reconverges to 3-of-3.** Deterministic post-recovery `log_len`/applied sync and catch-up apply completeness. +- **Liveness — elections always complete under churn.** No permanent election stall after repeated rapid failover. + +### Out of scope +The raft-drive-async async-drive/offload work (SlowLane, peer-control vtask dispatch, snapshot/config-push offload, endpoint-quiescence) — separate, complete, and dormant. This spec must not change its behavior. + +### Deliverables +- A **deterministic reproduction**: extend the `xce-test-ha` soak and add a unit/integration harness that forces the committed-prefix divergence and the reconverge / election-stall paths (RED before fix). +- The **`(term, lsn)` commit-identity + reconcile** mechanism. +- The **recovery/convergence** fixes (post-recovery log/applied sync, catch-up apply completeness) and the **election-liveness-under-churn** fix. +- A **green 6+round chaotic-failover soak on a real fs (release build)** with **zero committed-prefix divergence and zero job loss**. + +### Acceptance gate +This spec is the gate that must close before raft-drive-async flips its dispatch flags ON in production. + +### Verification reference +Verify all changes against `raft/raft.tla` and the controller TLA+ safety-invariant checklist in `src/controller/CLAUDE.md` (Election Safety, Leader Completeness, State Machine Safety, Commit Ordering/Durability). + +## Requirements + diff --git a/.kiro/specs/raft-ha-core-safety/spec.json b/.kiro/specs/raft-ha-core-safety/spec.json new file mode 100644 index 0000000..02e9854 --- /dev/null +++ b/.kiro/specs/raft-ha-core-safety/spec.json @@ -0,0 +1,22 @@ +{ + "feature_name": "raft-ha-core-safety", + "created_at": "2026-06-27T03:15:23Z", + "updated_at": "2026-06-27T03:15:23Z", + "language": "en", + "phase": "initialized", + "approvals": { + "requirements": { + "generated": false, + "approved": false + }, + "design": { + "generated": false, + "approved": false + }, + "tasks": { + "generated": false, + "approved": false + } + }, + "ready_for_implementation": false +} diff --git a/.wolf/memory.md b/.wolf/memory.md index f8729cf..4b86bf7 100644 --- a/.wolf/memory.md +++ b/.wolf/memory.md @@ -3551,3 +3551,8 @@ | 19:59 | Session end: 1 writes across 1 files (run-ha-soak.sh) | 1 reads | ~6083 tok | | 18:25 | 4.2 BASELINE comparison: pre-Phase-3 (21e26a1) release soak ALSO fails round 4 (election stall) → raft-drive-async = NO new regression; round-4 reconverge/divergence (LSN9) = pre-existing bug-851/795 family; zero committed-JOB loss both builds | ~/tmp/xce-ha-{rel,bl} | 4.2 regression sweep PASSES for spec; chaotic-soak instability pre-exists | ~50k | | 20:04 | Edited .kiro/specs/raft-drive-async/tasks.md | modified soak() | ~287 | +| 20:04 | Session end: 2 writes across 2 files (run-ha-soak.sh, tasks.md) | 2 reads | ~12938 tok | +| 20:11 | Session end: 2 writes across 2 files (run-ha-soak.sh, tasks.md) | 2 reads | ~12938 tok | +| 20:15 | Created .kiro/specs/raft-ha-core-safety/spec.json | — | ~130 | +| 20:16 | Created .kiro/specs/raft-ha-core-safety/requirements.md | — | ~1116 | +| 18:40 | spun new spec raft-ha-core-safety (gate before raft-drive-async prod flip): bug-851 committed-prefix divergence + bug-1042 reconverge + election stall under chaotic failover, all baseline-reproduced | .kiro/specs/raft-ha-core-safety/{spec.json,requirements.md} | scaffold initialized | ~8k |