wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity
myxce/.kiro/specs/raft-ha-core-safety/requirements.md
Rongsong Shen 3609f468c5 chore(raft-ha-core-safety): init spec scaffold (HA-core failover-safety gate) 
New spec for the pre-existing HA-core consistency/convergence failures the
xce-test-ha soak reproduces under chaotic failover — committed-prefix
divergence (bug-851, bug-795/828), standby-rejoin/reconverge failure
(bug-1042), and round-N election stall — all baseline-confirmed at commit
21e26a1 (independent of raft-drive-async). This spec is the gate that must
close before raft-drive-async flips its dispatch flags ON in production.

Scaffold only (spec.json + requirements.md project description); requirements
generation is the next phase.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
2026-06-27 03:16:17 +00:00

4.1 KiB
Raw Blame History

Requirements Document

Project Description (Input)

raft-ha-core-safety — Fix the pre-existing HA-core consistency and convergence failures that the live failover soak (xce-test-ha) reproduces under repeated/chaotic primary failover. This work is independent of, and a gate that must close before, the raft-drive-async production flag-flip.

Who has the problem

Operators and clients of an XCE controller HA cluster relying on Raft failover to keep committed jobs safe and the cluster available. Under repeated/chaotic primary churn the cluster can silently diverge committed state and/or fail to recover a healthy quorum.

Current situation (all baseline-reproduced)

All three symptoms reproduce on the pre-raft-drive-async baseline (commit 21e26a1) — confirmed by a baseline-vs-current failover-soak comparison — so they are NOT caused by the raft-drive-async async-drive/offload work (which is complete and dormant). They are pre-existing HA-core bugs:

  1. Committed-prefix divergence (bug-851, bug-795/828 family). Under chaotic failover, walcompare reports FAIL: committed-prefix divergence at LSN N — one member is MISSING a committed WAL entry that the majority holds, and keeps a divergent committed prefix silently. Contributing root causes already identified: WAL hdr.term left 0 (entry term never set from the writing leader's term), two-leaders-per-term windows, and no per-entry (term,lsn) identity or reconcile so a member cannot detect/repair a divergent committed prefix. The bug-795/828 family (committed-job/prefix loss under repeated and total-outage failover) is part of this cluster.

  2. Standby-rejoin / reconvergence failure (bug-1042 family). A restarted, previously-downed node intermittently fails to bring the cluster back to a healthy 3-of-3 within bound — even when it reaches the leader's log tip at the raft match level — due to post-recovery log_len/applied-state desync and catch-up apply gaps.

  3. Round-N election stall. After several rapid kill/restart rounds the cluster intermittently fails to elect a new primary after a primary kill (no CTRL_PRIMARY at term > prev within the failover window) — a liveness regression under churn.

What should change

Make repeated/chaotic failover provably safe and live:

  • Safety — committed entries never diverge across members. Enforce a real per-entry (term, lsn) commit identity plus a reconcile path so a member can detect and repair a divergent committed prefix rather than silently keeping it. Ensure the WAL entry term is always the writing leader's term (never 0). Close two-leaders-per-term windows.
  • Convergence — a restarted node always reconverges to 3-of-3. Deterministic post-recovery log_len/applied sync and catch-up apply completeness.
  • Liveness — elections always complete under churn. No permanent election stall after repeated rapid failover.

Out of scope

The raft-drive-async async-drive/offload work (SlowLane, peer-control vtask dispatch, snapshot/config-push offload, endpoint-quiescence) — separate, complete, and dormant. This spec must not change its behavior.

Deliverables

  • A deterministic reproduction: extend the xce-test-ha soak and add a unit/integration harness that forces the committed-prefix divergence and the reconverge / election-stall paths (RED before fix).
  • The (term, lsn) commit-identity + reconcile mechanism.
  • The recovery/convergence fixes (post-recovery log/applied sync, catch-up apply completeness) and the election-liveness-under-churn fix.
  • A green 6+round chaotic-failover soak on a real fs (release build) with zero committed-prefix divergence and zero job loss.

Acceptance gate

This spec is the gate that must close before raft-drive-async flips its dispatch flags ON in production.

Verification reference

Verify all changes against raft/raft.tla and the controller TLA+ safety-invariant checklist in src/controller/CLAUDE.md (Election Safety, Leader Completeness, State Machine Safety, Commit Ordering/Durability).

Requirements