Rongsong Shen
d8d1901901
- ha-async-io.md: document the same-batch endpoint lifetime hazard (bug-1447) and its ref-whole-batch-before-dispatch fix pattern - xce-audit-invariants SKILL.md: add object-lifetime-within-a-dispatch-batch as a 6th audit dimension, so a future pass checks other batch loops (vtaskmgr drain, peer-list iteration) for the same shape - product.md: replace the stale "HA consistency family is the current focus" framing with the actual 2026-07-18 status (umbrella complete, 9-spec validate-impl batch closed, job-array-single-wal-record shipped) - roadmap.md: correct 5 entries that were stale relative to actual spec status (ha-consistency-invariants/ha-wal-replication/libxce-failover pulled out of "In Flight", ha-term-safety and raft-drive-async marked implemented+GO, raft-sim-conformance and job-array-single-wal-record given closure annotations) -- struck through, not deleted, per this file's own additive-update convention - testing.md: document that meson's aggregate Fail:0 can mask Unity sub-group failures -- always check the standalone binary's own summary - src/controller/CLAUDE.md: fix a stale line-citation for the votedFor atomicity invariant (had drifted onto unrelated post-election code)
68 lines
5.6 KiB
Markdown
68 lines
5.6 KiB
Markdown
# Product Overview
|
||
|
||
XCE is an HPC (High-Performance Computing) cluster management system providing job scheduling, queue management, and cluster resource control. It targets organizations running batch workloads on on-premises or air-gapped compute clusters who need deterministic scheduling, multi-tenant access control, and zero external database dependencies.
|
||
|
||
## Core Capabilities
|
||
|
||
1. **Job lifecycle management** — submit, start, stop, restart, kill, and monitor jobs including job arrays. Resource usage data is collected and exposed via API.
|
||
2. **Queue and scheduling** — plugin-based scheduling engine for job-to-node placement with queue control (enable/disable) and expression-based resource filtering.
|
||
3. **Cluster and host management** — node status monitoring (online/offline/unavailable), cluster topology tracking, and host control operations.
|
||
4. **Multi-tenant RBAC + pluggable auth** — domain-based `action:scope` permission model (e.g., `job:view:self`). Auth backends: configfile, LDAP, OAuth2, PAM. Built-in IP whitelist and rate limiting.
|
||
5. **Controller persistence and HA** — WAL + checkpoint-based single-controller persistence with LSF-style priority election for standby failover and live replication over TCP (Cap'n Proto serialization).
|
||
|
||
## Target Use Cases
|
||
|
||
- **Local HPC batch computing** — long-running scientific/engineering workloads requiring deterministic scheduling, queue priority, and resource isolation.
|
||
- **Air-gapped / secure environments** — standalone deployments with file-based auth + SSL, no external cloud dependencies.
|
||
- **Multi-team shared clusters** — multiple teams sharing a cluster through RBAC with JWT token authentication.
|
||
- **No-shared-storage HPC** — controller uses local filesystem (ext4/xfs) for storage; job files transferred via API upload/download rather than network shares.
|
||
|
||
## Value Proposition
|
||
|
||
XCE provides a **fully integrated single-vendor stack** (no database setup, unified build system) with an open, buildable codebase (Nix flakes, AppImage packaging). Compared to Slurm/PBS, it eliminates the external database requirement. Compared to commercial LSF, it is open source and reproducibly buildable.
|
||
|
||
**Key architectural trade-off**: single-writer model throughout — only the primary writes. HA follows Raft semantics with CP guarantees: consensus election (RequestVote, term-based voting, freshness gates) via the `src/lib/raft/` shared module (`raft.c`/`raft.h`, consumed by the controller through `ha_init_raft`), and majority-ACK log commitment for state-changing writes (WAL entries ≈ Raft log, `committed_lsn` ≈ commitIndex). Operations that cannot reach majority are rejected, never locally degraded. The single-writer constraint is XCE's simplification of Raft — no concurrent client proposals — while commitment and election inherit Raft's safety rules. All controller state is in-memory; persistence is WAL + checkpoint, not a traditional database.
|
||
|
||
## Active Development Context
|
||
|
||
The project is in a **persistence and HA transformation** (branch `persistent_ha`). Phases:
|
||
|
||
| Phase | Goal | Status |
|
||
|-------|------|--------|
|
||
| P0 | Restart survival (WAL + checkpoint) | Complete |
|
||
| P1 | Standby election and takeover (Raft-style) | Complete |
|
||
| P2 | Live replication primary → standby (config + WAL) | Complete |
|
||
| P2.5 | Worker/scheduler auto-reconnect on failover | In progress |
|
||
| P3–P4 | Hardening, metrics, docs | In progress |
|
||
|
||
## Spec-Driven Development
|
||
|
||
Features are designed and implemented via the kiro spec system (`.kiro/specs/`).
|
||
Each feature follows a 3-phase approval workflow: Requirements → Design → Tasks → Implementation.
|
||
|
||
**Status as of 2026-07-18**: the ha-consistency-invariants umbrella (ha-commit-engine,
|
||
ha-commit-barrier-rollout, ha-single-leader-role, ha-convergence-recovery) is
|
||
COMPLETE (closed 2026-07-13). The Raft TLA+ Conformance family (raft-library,
|
||
raft-log-replication, raft-commit-safety, raft-sim-conformance) and its
|
||
XCE-integration umbrella ha-term-safety are implemented + validate-impl GO.
|
||
job-array-single-wal-record shipped complete (2026-07-17, including a
|
||
user-directed residual-gap closure pass). A backlog sweep (2026-07-18) ran
|
||
`/kiro-validate-impl` on 9 specs whose tasks were 100% done but had never
|
||
been formally closed — raft-drive-async, ha-wal-replication,
|
||
ha-heartbeat-committed-propagation, ha-prevote, ha-elect-ctx-lifecycle,
|
||
ha-entity-snapshots, ha-commit-durability, libxce-failover,
|
||
raft-sim-conformance — all now GO. That sweep surfaced one real bug
|
||
(bug-1447: same-batch endpoint use-after-free in `io_worker`'s epoll
|
||
dispatch loop, see `ha-async-io.md`), root-caused and fixed the same day.
|
||
|
||
Remaining backlog (requirements-only or design-pending, not yet implemented):
|
||
admin-config-tool (spec finalized), scheduler-framework, config-upgrade-snapshot
|
||
(blocked on entity-bearing snapshots). See `roadmap.md` for dependency order
|
||
and status.
|
||
|
||
Key design decisions already locked:
|
||
- No `fork()` for snapshots (mutex inheritance deadlock in multithreaded controller)
|
||
- Raft + CP adopted (2026-06-10, supersedes earlier "No Raft"): Raft is the consensus model for both election and log commitment — majority-ACK commit barrier on all state-changing writes, reject (never degrade) when quorum is unavailable. Single-writer log retained as XCE's simplification: the controller is the sole log writer, avoiding concurrent-proposal complexity while inheriting Raft's safety rules (see `.kiro/specs/ha-consistency-invariants/brief.md`)
|
||
- No BoltDB (using dual zarray COW for application-level checkpointing)
|
||
- No application-layer encryption (trust OS boundary: chmod 0700, LUKS/dm-crypt)
|
||
- No WAL logging for job deletion (deferred to checkpoint time to simplify replay)
|