myxce/docs/superpowers/plans/2026-05-07-persistence-ha-overview.md

ARCHIVED — superseded. This phase map planned dual-zarray COW checkpointing, LSF-style priority-list election, and single-standby-ACK durability for HA. What actually shipped instead: capnp-frame checkpoints (ha-entity-snapshots), full quorum-based Raft consensus (raft-library, raft-ha-core-safety, ha-prevote, election-iomgr-refactor, raft-log-replication, raft-commit-safety, ha-term-safety, ha-single-leader-role), and majority-quorum commit (ha-commit-engine, ha-commit-durability, ha-commit-barrier-rollout) — all closed .kiro/specs/. See the SUPERSEDED notes in src/docs/persistence-and-ha-design.md (Parts 3, 5, and the durability section) for details.

XCE Persistence & HA — Implementation Overview

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Make the XCE controller survive restarts and primary failures with no job/worker/queue state loss.

Architecture: WAL + periodic checkpoint (dual-zarray COW) for single-controller persistence; LSF-style priority election + fan-out TCP replication for HA. libxce smart discovery handles client auto-reconnect after failover.

Tech Stack: C (controller/lib), Cap'n Proto (serialization), Nix flakes + meson + ninja (build), Go (API server discovery bridge)

---

Phase Map

Each phase is an independently shippable increment. Create the next phase plan only after the current phase passes its gate tests.

Phase	Plan file	Goal	Gate
P0	2026-05-07-p0-single-controller-persistence.md	Controller survives restart	P0-G smoke tests green
P1	create after P0 ships	Standby takes over when primary fails	Election + promotion tests green
P2	create after P1 ships	Primary replicates to standbys in real time	Replication round-trip + catch-up tests green
P2.5	create alongside P2	Workers/schedulers auto-reconnect after failover	Discovery + reconnect tests green
P3	create after P2 ships	End-to-end failover + hardening + metrics	Full failover test + benchmark passing
P4	create after P3 ships	Docs, runbook, stale-ref cleanup	Runbook review complete

Out of Scope (all phases)

• Admin CLI for manual failover (xce hostcontrol --failover)
• WAL compaction beyond checkpoint trimming
• Multi-DC replication
• Runtime config changes without restart
• Snapshot compression
• config_version hash in heartbeat (deferred to P3 tooling)
• Degraded-mode restricted-state timeout (repl_degraded_timeout_s)
• MAX_HA_CANDIDATES > 4 / MAX_STANDBYS > 3

Design Document

src/docs/persistence-and-ha-design.md — all implementation detail lives here. Read the relevant Part before starting each task.