forked from lei.chen/myxce
chore(raft-ha-core-safety): init spec scaffold (HA-core failover-safety gate)
New spec for the pre-existing HA-core consistency/convergence failures the
xce-test-ha soak reproduces under chaotic failover — committed-prefix
divergence (bug-851, bug-795/828), standby-rejoin/reconverge failure
(bug-1042), and round-N election stall — all baseline-confirmed at commit
21e26a1 (independent of raft-drive-async). This spec is the gate that must
close before raft-drive-async flips its dispatch flags ON in production.
Scaffold only (spec.json + requirements.md project description); requirements
generation is the next phase.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
This commit is contained in:
parent
294b9ce8eb
commit
3609f468c5
3 changed files with 69 additions and 0 deletions
42
.kiro/specs/raft-ha-core-safety/requirements.md
Normal file
42
.kiro/specs/raft-ha-core-safety/requirements.md
Normal file
|
|
@ -0,0 +1,42 @@
|
|||
# Requirements Document
|
||||
|
||||
## Project Description (Input)
|
||||
|
||||
**raft-ha-core-safety** — Fix the pre-existing HA-core consistency and convergence failures that the live failover soak (`xce-test-ha`) reproduces under repeated/chaotic primary failover. This work is **independent of, and a gate that must close before, the raft-drive-async production flag-flip**.
|
||||
|
||||
### Who has the problem
|
||||
Operators and clients of an XCE controller HA cluster relying on Raft failover to keep committed jobs safe and the cluster available. Under repeated/chaotic primary churn the cluster can silently diverge committed state and/or fail to recover a healthy quorum.
|
||||
|
||||
### Current situation (all baseline-reproduced)
|
||||
All three symptoms reproduce on the **pre-raft-drive-async baseline (commit `21e26a1`)** — confirmed by a baseline-vs-current failover-soak comparison — so they are **NOT** caused by the raft-drive-async async-drive/offload work (which is complete and dormant). They are pre-existing HA-core bugs:
|
||||
|
||||
1. **Committed-prefix divergence (bug-851, bug-795/828 family).** Under chaotic failover, `walcompare` reports `FAIL: committed-prefix divergence at LSN N` — one member is **MISSING a committed WAL entry that the majority holds**, and keeps a divergent committed prefix silently. Contributing root causes already identified: WAL `hdr.term` left `0` (entry term never set from the writing leader's term), two-leaders-per-term windows, and no per-entry `(term,lsn)` identity or reconcile so a member cannot detect/repair a divergent committed prefix. The bug-795/828 family (committed-job/prefix loss under repeated and total-outage failover) is part of this cluster.
|
||||
|
||||
2. **Standby-rejoin / reconvergence failure (bug-1042 family).** A restarted, previously-downed node intermittently fails to bring the cluster back to a healthy **3-of-3** within bound — even when it reaches the leader's log tip at the raft `match` level — due to post-recovery `log_len`/applied-state desync and catch-up apply gaps.
|
||||
|
||||
3. **Round-N election stall.** After several rapid kill/restart rounds the cluster intermittently fails to elect a new primary after a primary kill (no `CTRL_PRIMARY` at `term > prev` within the failover window) — a liveness regression under churn.
|
||||
|
||||
### What should change
|
||||
Make repeated/chaotic failover **provably safe and live**:
|
||||
|
||||
- **Safety — committed entries never diverge across members.** Enforce a real per-entry `(term, lsn)` commit identity plus a reconcile path so a member can **detect and repair** a divergent committed prefix rather than silently keeping it. Ensure the WAL entry term is always the writing leader's term (never `0`). Close two-leaders-per-term windows.
|
||||
- **Convergence — a restarted node always reconverges to 3-of-3.** Deterministic post-recovery `log_len`/applied sync and catch-up apply completeness.
|
||||
- **Liveness — elections always complete under churn.** No permanent election stall after repeated rapid failover.
|
||||
|
||||
### Out of scope
|
||||
The raft-drive-async async-drive/offload work (SlowLane, peer-control vtask dispatch, snapshot/config-push offload, endpoint-quiescence) — separate, complete, and dormant. This spec must not change its behavior.
|
||||
|
||||
### Deliverables
|
||||
- A **deterministic reproduction**: extend the `xce-test-ha` soak and add a unit/integration harness that forces the committed-prefix divergence and the reconverge / election-stall paths (RED before fix).
|
||||
- The **`(term, lsn)` commit-identity + reconcile** mechanism.
|
||||
- The **recovery/convergence** fixes (post-recovery log/applied sync, catch-up apply completeness) and the **election-liveness-under-churn** fix.
|
||||
- A **green 6+round chaotic-failover soak on a real fs (release build)** with **zero committed-prefix divergence and zero job loss**.
|
||||
|
||||
### Acceptance gate
|
||||
This spec is the gate that must close before raft-drive-async flips its dispatch flags ON in production.
|
||||
|
||||
### Verification reference
|
||||
Verify all changes against `raft/raft.tla` and the controller TLA+ safety-invariant checklist in `src/controller/CLAUDE.md` (Election Safety, Leader Completeness, State Machine Safety, Commit Ordering/Durability).
|
||||
|
||||
## Requirements
|
||||
<!-- Will be generated in /kiro-spec-requirements phase -->
|
||||
22
.kiro/specs/raft-ha-core-safety/spec.json
Normal file
22
.kiro/specs/raft-ha-core-safety/spec.json
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
{
|
||||
"feature_name": "raft-ha-core-safety",
|
||||
"created_at": "2026-06-27T03:15:23Z",
|
||||
"updated_at": "2026-06-27T03:15:23Z",
|
||||
"language": "en",
|
||||
"phase": "initialized",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
},
|
||||
"design": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
},
|
||||
"tasks": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": false
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue