Product Overview

XCE is an HPC (High-Performance Computing) cluster management system providing job scheduling, queue management, and cluster resource control. It targets organizations running batch workloads on on-premises or air-gapped compute clusters who need deterministic scheduling, multi-tenant access control, and zero external database dependencies.

Core Capabilities

Job lifecycle management — submit, start, stop, restart, kill, and monitor jobs including job arrays. Resource usage data is collected and exposed via API.
Queue and scheduling — plugin-based scheduling engine for job-to-node placement with queue control (enable/disable) and expression-based resource filtering.
Cluster and host management — node status monitoring (OK, UNAVAIL, LOCKEDU, LOCKEDW, BUSY, CLOSED, CLOSED_BY_ADMIN), cluster topology tracking, and host control operations.
Multi-tenant RBAC + pluggable auth — two-part action:operation permission IDs (e.g., job:view) with a separately-keyed domain scope per role (e.g., type: self, type: self_ug, type: all). Auth backends: configfile, LDAP, OAuth2, PAM. Built-in IP whitelist and rate limiting.
Controller persistence and HA — WAL + checkpoint-based single-controller persistence with Raft-style quorum election (RequestVote, term-based voting, freshness gates) for standby failover and live replication over TCP (Cap'n Proto serialization).

Target Use Cases

Local HPC batch computing — long-running scientific/engineering workloads requiring deterministic scheduling, queue priority, and resource isolation.
Air-gapped / secure environments — standalone deployments with file-based auth + SSL, no external cloud dependencies.
Multi-team shared clusters — multiple teams sharing a cluster through RBAC with JWT token authentication.
No-shared-storage HPC — controller uses local filesystem (ext4/xfs) for storage; job files transferred via API upload/download rather than network shares.

Value Proposition

XCE provides a fully integrated single-vendor stack (no database setup, unified build system) with an open, buildable codebase (Nix flakes, AppImage packaging). Compared to Slurm/PBS, it eliminates the external database requirement. Compared to commercial LSF, it is open source and reproducibly buildable.

Key architectural trade-off: single-writer model throughout — only the primary writes. HA follows Raft semantics with CP guarantees: consensus election (RequestVote, term-based voting, freshness gates) via the src/lib/raft/ shared module (raft.c/raft.h, consumed by the controller through ha_init_raft), and majority-ACK log commitment for state-changing writes (WAL entries ≈ Raft log, committed_lsn ≈ commitIndex). Operations that cannot reach majority are rejected, never locally degraded. The single-writer constraint is XCE's simplification of Raft — no concurrent client proposals — while commitment and election inherit Raft's safety rules. All controller state is in-memory; persistence is WAL + checkpoint, not a traditional database.

Active Development Context

The project is in a persistence and HA transformation (branch persistent_ha). Phases:

Phase	Goal	Status
P0	Restart survival (WAL + checkpoint)	Complete
P1	Standby election and takeover (Raft-style)	Complete
P2	Live replication primary → standby (config + WAL)	Complete
P2.5	Worker/scheduler auto-reconnect on failover	In progress
P3–P4	Hardening, metrics, docs	In progress

Spec-Driven Development

Features are designed and implemented via the kiro spec system (.kiro/specs/). Each feature follows a 3-phase approval workflow: Requirements → Design → Tasks → Implementation.

Status as of 2026-07-18: the ha-consistency-invariants umbrella (ha-commit-engine, ha-commit-barrier-rollout, ha-single-leader-role, ha-convergence-recovery) is COMPLETE (closed 2026-07-13). The Raft TLA+ Conformance family (raft-library, raft-log-replication, raft-commit-safety, raft-sim-conformance) and its XCE-integration umbrella ha-term-safety are implemented + validate-impl GO. job-array-single-wal-record shipped complete (2026-07-17, including a user-directed residual-gap closure pass). A backlog sweep (2026-07-18) ran /kiro-validate-impl on 9 specs whose tasks were 100% done but had never been formally closed — raft-drive-async, ha-wal-replication, ha-heartbeat-committed-propagation, ha-prevote, ha-elect-ctx-lifecycle, ha-entity-snapshots, ha-commit-durability, libxce-failover, raft-sim-conformance — all now GO. That sweep surfaced one real bug (bug-1447: same-batch endpoint use-after-free in io_worker's epoll dispatch loop, see ha-async-io.md), root-caused and fixed the same day.

Remaining backlog (requirements-only, design-pending, or partially implemented): admin-config-tool (spec finalized), config-upgrade-snapshot (tasks generated, unblocked, implementation nearly complete). scheduler-framework was decomposed into scheduler-core (implementation-complete), scheduler-allocation, and scheduler-preemption (both active). See roadmap.md for dependency order and status.

Key design decisions already locked:

No fork() for snapshots (mutex inheritance deadlock in multithreaded controller)
Raft + CP adopted (2026-06-10, supersedes earlier "No Raft"): Raft is the consensus model for both election and log commitment — majority-ACK commit barrier on all state-changing writes, reject (never degrade) when quorum is unavailable. Single-writer log retained as XCE's simplification: the controller is the sole log writer, avoiding concurrent-proposal complexity while inheriting Raft's safety rules (see .kiro/specs/ha-consistency-invariants/brief.md)
No BoltDB (using dual zarray COW for application-level checkpointing)
No application-layer encryption (trust OS boundary: chmod 0700, LUKS/dm-crypt)
No WAL logging for job deletion (deferred to checkpoint time to simplify replay)

5.9 KiB Raw Blame History Unescape Escape