41 lines
2.9 KiB
Markdown
41 lines
2.9 KiB
Markdown
> **ARCHIVED — superseded.** This phase map planned dual-zarray COW checkpointing, LSF-style priority-list election, and single-standby-ACK durability for HA. What actually shipped instead: capnp-frame checkpoints (`ha-entity-snapshots`), full quorum-based Raft consensus (`raft-library`, `raft-ha-core-safety`, `ha-prevote`, `election-iomgr-refactor`, `raft-log-replication`, `raft-commit-safety`, `ha-term-safety`, `ha-single-leader-role`), and majority-quorum commit (`ha-commit-engine`, `ha-commit-durability`, `ha-commit-barrier-rollout`) — all closed `.kiro/specs/`. See the SUPERSEDED notes in `src/docs/persistence-and-ha-design.md` (Parts 3, 5, and the durability section) for details.
|
|
|
|
# XCE Persistence & HA — Implementation Overview
|
|
|
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
|
|
**Goal:** Make the XCE controller survive restarts and primary failures with no job/worker/queue state loss.
|
|
|
|
**Architecture:** WAL + periodic checkpoint (dual-zarray COW) for single-controller persistence; LSF-style priority election + fan-out TCP replication for HA. libxce smart discovery handles client auto-reconnect after failover.
|
|
|
|
**Tech Stack:** C (controller/lib), Cap'n Proto (serialization), Nix flakes + meson + ninja (build), Go (API server discovery bridge)
|
|
|
|
---
|
|
|
|
## Phase Map
|
|
|
|
Each phase is an independently shippable increment. Create the next phase plan only after the current phase passes its gate tests.
|
|
|
|
| Phase | Plan file | Goal | Gate |
|
|
|-------|-----------|------|------|
|
|
| **P0** | `2026-05-07-p0-single-controller-persistence.md` | Controller survives restart | `P0-G` smoke tests green |
|
|
| **P1** | _create after P0 ships_ | Standby takes over when primary fails | Election + promotion tests green |
|
|
| **P2** | _create after P1 ships_ | Primary replicates to standbys in real time | Replication round-trip + catch-up tests green |
|
|
| **P2.5** | _create alongside P2_ | Workers/schedulers auto-reconnect after failover | Discovery + reconnect tests green |
|
|
| **P3** | _create after P2 ships_ | End-to-end failover + hardening + metrics | Full failover test + benchmark passing |
|
|
| **P4** | _create after P3 ships_ | Docs, runbook, stale-ref cleanup | Runbook review complete |
|
|
|
|
## Out of Scope (all phases)
|
|
|
|
- Admin CLI for manual failover (`xce hostcontrol --failover`)
|
|
- WAL compaction beyond checkpoint trimming
|
|
- Multi-DC replication
|
|
- Runtime config changes without restart
|
|
- Snapshot compression
|
|
- `config_version` hash in heartbeat (deferred to P3 tooling)
|
|
- Degraded-mode restricted-state timeout (`repl_degraded_timeout_s`)
|
|
- `MAX_HA_CANDIDATES > 4` / `MAX_STANDBYS > 3`
|
|
|
|
## Design Document
|
|
|
|
`src/docs/persistence-and-ha-design.md` — all implementation detail lives here. Read the relevant Part before starting each task.
|