chore(raft-ha-core-safety): init spec scaffold (HA-core failover-safety gate)

New spec for the pre-existing HA-core consistency/convergence failures the
xce-test-ha soak reproduces under chaotic failover — committed-prefix
divergence (bug-851, bug-795/828), standby-rejoin/reconverge failure
(bug-1042), and round-N election stall — all baseline-confirmed at commit
21e26a1 (independent of raft-drive-async). This spec is the gate that must
close before raft-drive-async flips its dispatch flags ON in production.

Scaffold only (spec.json + requirements.md project description); requirements
generation is the next phase.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
This commit is contained in:
Rongsong Shen 2026-06-27 03:16:17 +00:00
parent 294b9ce8eb
commit 3609f468c5
3 changed files with 69 additions and 0 deletions

View file

@ -0,0 +1,42 @@
# Requirements Document
## Project Description (Input)
**raft-ha-core-safety** — Fix the pre-existing HA-core consistency and convergence failures that the live failover soak (`xce-test-ha`) reproduces under repeated/chaotic primary failover. This work is **independent of, and a gate that must close before, the raft-drive-async production flag-flip**.
### Who has the problem
Operators and clients of an XCE controller HA cluster relying on Raft failover to keep committed jobs safe and the cluster available. Under repeated/chaotic primary churn the cluster can silently diverge committed state and/or fail to recover a healthy quorum.
### Current situation (all baseline-reproduced)
All three symptoms reproduce on the **pre-raft-drive-async baseline (commit `21e26a1`)** — confirmed by a baseline-vs-current failover-soak comparison — so they are **NOT** caused by the raft-drive-async async-drive/offload work (which is complete and dormant). They are pre-existing HA-core bugs:
1. **Committed-prefix divergence (bug-851, bug-795/828 family).** Under chaotic failover, `walcompare` reports `FAIL: committed-prefix divergence at LSN N` — one member is **MISSING a committed WAL entry that the majority holds**, and keeps a divergent committed prefix silently. Contributing root causes already identified: WAL `hdr.term` left `0` (entry term never set from the writing leader's term), two-leaders-per-term windows, and no per-entry `(term,lsn)` identity or reconcile so a member cannot detect/repair a divergent committed prefix. The bug-795/828 family (committed-job/prefix loss under repeated and total-outage failover) is part of this cluster.
2. **Standby-rejoin / reconvergence failure (bug-1042 family).** A restarted, previously-downed node intermittently fails to bring the cluster back to a healthy **3-of-3** within bound — even when it reaches the leader's log tip at the raft `match` level — due to post-recovery `log_len`/applied-state desync and catch-up apply gaps.
3. **Round-N election stall.** After several rapid kill/restart rounds the cluster intermittently fails to elect a new primary after a primary kill (no `CTRL_PRIMARY` at `term > prev` within the failover window) — a liveness regression under churn.
### What should change
Make repeated/chaotic failover **provably safe and live**:
- **Safety — committed entries never diverge across members.** Enforce a real per-entry `(term, lsn)` commit identity plus a reconcile path so a member can **detect and repair** a divergent committed prefix rather than silently keeping it. Ensure the WAL entry term is always the writing leader's term (never `0`). Close two-leaders-per-term windows.
- **Convergence — a restarted node always reconverges to 3-of-3.** Deterministic post-recovery `log_len`/applied sync and catch-up apply completeness.
- **Liveness — elections always complete under churn.** No permanent election stall after repeated rapid failover.
### Out of scope
The raft-drive-async async-drive/offload work (SlowLane, peer-control vtask dispatch, snapshot/config-push offload, endpoint-quiescence) — separate, complete, and dormant. This spec must not change its behavior.
### Deliverables
- A **deterministic reproduction**: extend the `xce-test-ha` soak and add a unit/integration harness that forces the committed-prefix divergence and the reconverge / election-stall paths (RED before fix).
- The **`(term, lsn)` commit-identity + reconcile** mechanism.
- The **recovery/convergence** fixes (post-recovery log/applied sync, catch-up apply completeness) and the **election-liveness-under-churn** fix.
- A **green 6+round chaotic-failover soak on a real fs (release build)** with **zero committed-prefix divergence and zero job loss**.
### Acceptance gate
This spec is the gate that must close before raft-drive-async flips its dispatch flags ON in production.
### Verification reference
Verify all changes against `raft/raft.tla` and the controller TLA+ safety-invariant checklist in `src/controller/CLAUDE.md` (Election Safety, Leader Completeness, State Machine Safety, Commit Ordering/Durability).
## Requirements
<!-- Will be generated in /kiro-spec-requirements phase -->

View file

@ -0,0 +1,22 @@
{
"feature_name": "raft-ha-core-safety",
"created_at": "2026-06-27T03:15:23Z",
"updated_at": "2026-06-27T03:15:23Z",
"language": "en",
"phase": "initialized",
"approvals": {
"requirements": {
"generated": false,
"approved": false
},
"design": {
"generated": false,
"approved": false
},
"tasks": {
"generated": false,
"approved": false
}
},
"ready_for_implementation": false
}