commit missed

.kiro/steering/product.md

@@ -21,7 +21,7 @@ XCE is an HPC (High-Performance Computing) cluster management system providing j
 
 XCE provides a **fully integrated single-vendor stack** (no database setup, unified build system) with an open, buildable codebase (Nix flakes, AppImage packaging). Compared to Slurm/PBS, it eliminates the external database requirement. Compared to commercial LSF, it is open source and reproducibly buildable.
 
-**Key architectural trade-off**: single-writer model throughout — only the primary writes. HA follows Raft semantics with CP guarantees: consensus election (RequestVote, term-based voting, freshness gates) via `raft.c`/`raft.h`, and majority-ACK log commitment for state-changing writes (WAL entries ≈ Raft log, `committed_lsn` ≈ commitIndex). Operations that cannot reach majority are rejected, never locally degraded. The single-writer constraint is XCE's simplification of Raft — no concurrent client proposals — while commitment and election inherit Raft's safety rules. All controller state is in-memory; persistence is WAL + checkpoint, not a traditional database.
+**Key architectural trade-off**: single-writer model throughout — only the primary writes. HA follows Raft semantics with CP guarantees: consensus election (RequestVote, term-based voting, freshness gates) via the `src/lib/raft/` shared module (`raft.c`/`raft.h`, consumed by the controller through `ha_init_raft`), and majority-ACK log commitment for state-changing writes (WAL entries ≈ Raft log, `committed_lsn` ≈ commitIndex). Operations that cannot reach majority are rejected, never locally degraded. The single-writer constraint is XCE's simplification of Raft — no concurrent client proposals — while commitment and election inherit Raft's safety rules. All controller state is in-memory; persistence is WAL + checkpoint, not a traditional database.
 
 ## Active Development Context
 
@@ -39,10 +39,13 @@ The project is in a **persistence and HA transformation** (branch `persistent_ha
 
 Features are designed and implemented via the kiro spec system (`.kiro/specs/`).
 Each feature follows a 3-phase approval workflow: Requirements → Design → Tasks → Implementation.
-Active specs (10 total) include: ha-config-bootstrap, ha-heartbeat-liveness,
+Active specs (15 total). The HA consistency family is the current focus:
+ha-consistency-invariants (umbrella contract) decomposed into ha-commit-engine,
+ha-commit-barrier-rollout, ha-single-leader-role, and ha-convergence-recovery.
+Supporting/foundational specs: ha-config-bootstrap, ha-heartbeat-liveness,
 ha-peer-connection-reuse, ha-repl-hardening, ha-wal-replication, raft-library,
 election-iomgr-refactor, scheduler-controller-msg-gap, worker-join-refactor,
-resval-id-decoupling.
+resval-id-decoupling. See `roadmap.md` for dependency order and status.
 
 Key design decisions already locked:
 - No `fork()` for snapshots (mutex inheritance deadlock in multithreaded controller)

.kiro/steering/roadmap.md

@@ -36,6 +36,8 @@ Smaller items that likely don't warrant a full spec; pick up opportunistically o
 - [ ] lost-and-found drain command — `xcecli lost-and-found drain`: reassign or cancel sidelined jobs, reset `lost_and_found_job_count`. Automates operator cleanup after config upgrade. _(FUTURE_WORK.md §4.)_
 - [ ] OP_upgradeConfig standby redirect verification — test that a standby returns the primary's address to an admin tool and the tool retries against the primary. Note: redirect semantics change under ha-consistency-invariants Req 1 (no transitional-state redirect); verify against the new behavior. _(FUTURE_WORK.md §5.)_
 - [x] steering sync — `ha-async-io.md` synchronized to the three-state model + own-term gate (transitional-state row/role removed; heartbeat re-arm PRIMARY-only; PRIMARY no-quorum fact allowlist noted). Done as part of ha-single-leader-role task 6.1.
+- [ ] xce_test shell command support — add `!<command>` prefix to `xce_test` test scripts to execute arbitrary shell commands inline during test runs. Enables test scripts to drive environment setup/teardown (e.g., kill a node, write a file, run a helper) without calling out to a separate harness layer. Scope: parser change in the xce_test runner to detect `!`-prefixed lines, fork/exec the remainder as a shell command, fail the test on non-zero exit, capture stdout/stderr to the test log.
+- [ ] scheduler framework enhancement — strengthen the scheduler subsystem: (a) improve job scheduling policy extensibility (pluggable queue disciplines beyond FIFO), (b) add priority-aware dispatch, (c) improve resource-aware scheduling (CPU/memory/GPU slot tracking per host), (d) wire scheduler decisions through the commit barrier so scheduling state is replicated and survives failover. Dependencies: ha-consistency-invariants (commit barrier). Spec advised before implementation; promote via `/kiro-spec-init "scheduler-framework"` when ready.
 
 ## Superseded / Absorbed Items
 

.kiro/steering/structure.md

@@ -7,6 +7,8 @@
 ```
 src/
 ├── lib/           C — shared library (libxce): data structures, IPC, utilities
+│   ├── raft/      C — reusable Raft consensus module (raft.c/raft.h)
+│   └── testsuite/ C — libxce unit tests (test_*.c)
 ├── controller/    C — cluster controller daemon
 ├── worker/        C — per-node job execution daemon (xcecd)
 ├── scheduler/     C — job scheduling engine (plugin architecture)

.kiro/steering/tech.md

@@ -97,4 +97,4 @@ Early return on every error. Log with zap fields before returning.
 3. **CGO bridge** — `backend/` package wraps C library calls; unit tests run `CGO_ENABLED=0`, backend tests need `CGO_ENABLED=1` inside nix dev shell.
 4. **Swagger-first API docs** — `swaggo/swag` annotations on handlers; docs served at runtime by Gin.
 5. **Structured logging** — every log entry carries context via `zap.String()`, `zap.Error()`. Never use `fmt.Println` for operational output.
-6. **Single-writer HA** — controller is always one active primary; standby election via Raft-style consensus (term-based voting, freshness gates) in `raft.c`, with the single-writer constraint avoiding Raft log replication complexity.
+6. **Single-writer HA** — controller is always one active primary; standby election via Raft-style consensus (term-based voting, freshness gates) in the `src/lib/raft/` shared module (`raft.c`/`raft.h`), with the single-writer constraint avoiding Raft log replication complexity.