docs(steering): resource decision and question log (PR-1..3 proposals, R/P/N items)

Persistent log of the resource-design discussion: the user's proposals
(generic supply map incl. provisioned supply; a resource provider proxy
between the controller and upstream providers; datasets in the map with
staging as data provisioning) with analysis, the user's 2026-10-11 input
(duration/decay must-have but future; RBAC in the apiserver, discuss later;
OCI containers via application profiles; CSI compatibility preferred;
GPU/topology scheduler-first and extensible), the open R1-R9/P1-P9 items
and new questions N1-N4, C3. resource-model.md, the provisioner research,
roadmap and CLAUDE.md point at it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2026-10-12 04:00:47 +00:00

88 KiB

Raw Blame History

Roadmap

Overview

XCE is in the persistence + HA transformation (branch persistent_ha). Governing model: Raft consensus + CP (see product.md key decisions). Active spec work is tracked per-spec in .kiro/specs/; this file is the single consolidated future-work document — the forward-looking queue, long-deferred items, and hardening backlog. It absorbs the former src/docs/FUTURE_WORK.md (2026-05-12, config-replication-design review), which now redirects here.

Long-term direction (user, 2026-10-10): XCE is heading toward a unified platform for applications such as LLM training and deployment (serving), not only classic HPC batch. Features are therefore drawn from IBM LSF and Kubernetes: consumables, rusage, elim, licenses and application profiles from LSF; extended resources, device plugins, CSI volumes, Dynamic Resource Allocation claims and services with replicas from Kubernetes. Designs for resources, providers, job lifecycle and APIs should serve both batch jobs and long-running services. Resource vocabulary: resource-model.md.

In Flight

scheduler-framework — NEW 2026-07-06 (/kiro-discovery). Iterator-driven allocation architecture: lazy plan iterator + host iterator, combinatorial parallel job allocation, configurable job picker, session-level unsatisfiable resreq cache. Brief: .kiro/specs/scheduler-framework/brief.md. DECOMPOSED 2026-07-27 into three child specs (.kiro/specs/scheduler-framework/spec.json child_specs): scheduler-core — core/driver separation, pluggable job picker interface, and simulation mode; IMPLEMENTATION-COMPLETE, GO/NO-GO validation 2026-08-07/08 (2 non-blocking follow-ups: (a) tlog logging-integration built but never activated, (b) fcfs.c/builtin_fcfs.c doc+traceability gap — design.md text was corrected same-day, but tasks 2.1-2.6 in tasks.md still carry no _Verified: lines, still open), 45/45 sub-tasks done as of that validation. Updated 2026-08-19: task 12.5 (REPL live-operator-testing fixes, Req 5.22 amendment) was added after that validation from live-operator testing gaps (bug-1593/1594/1595/1596) and is now also done, independently re-verified with fresh evidence (/kiro-verify-completion) — see tasks.md's own _Verified: line on that task; this closes follow-up (a) too (the tlog gap is now activated — see design.md's SchedulerCore component and research.md). A separate retrospective design review (2026-08-19, unrelated to the original 2026-08-07 follow-ups) additionally found and fixed an OPCode_schedAdmin blocking-wait admission gap and extended core.c's isolation-boundary test coverage to its core_events.c/core_session.c/core_clock.c siblings — see research.md's "SUPERSEDES" notes. scheduler-allocation — lazy iterator-driven allocation architecture, parallel job support, session-level unsatisfiable resreq cache; requirements generated, blocked_on: [scheduler-core] — now unblocked. scheduler-preemption — SIGSTP-based victim selection + controller→worker pipeline; requirements generated, blocked_on: [scheduler-core] — now unblocked. The parent scheduler-framework spec is retained only for discovery-context provenance and is no longer in flight itself. Next: /kiro-spec-design scheduler-allocation and /kiro-spec-design scheduler-preemption.
ha-consistency-invariants — UMBRELLA CONTRACT (requirements + design + research approved/complete). Implementation decomposed 2026-06-11 into the four sub-specs below; the umbrella generates NO tasks itself. Each sub-spec brief references the umbrella docs as authoritative. COMPLETE 2026-07-13 — all four sub-specs (ha-commit-engine, ha-commit-barrier-rollout, ha-single-leader-role, ha-convergence-recovery) implemented; Criterion 2.9 (Facts-Apply invariant) satisfied via raft-apply-decoupling. No longer in flight.
~~ha-wal-replication — implemented except task 5.4 (live 3-node integration test; needs /xce-deploy-ha cluster); sub-spec 4's E2E extends it~~ task 5.4 CLOSED 2026-07-18 via /kiro-validate-impl's live 10-round sequential soak (972/972 tests, zero loss). Spec is GO, no longer in flight.
libxce-failover — CLIENT-SIDE transparent controller failover (requirements generated 2026-06-16). Wires all libxce production request paths through the (already-built-but-orphaned) xce_request_with_retry() primitive + adds multi-master rotation; classifies on the real controller status taxonomy E_XCE_REDIRECT / E_XCE_NOMASTER / E_XCE_NOT_READY / E_XCE_STANDBY (errnox.h:22-25); bounded retry → distinct cluster-not-ready (no more opaque EINVAL); includes one small controller change so redirect replies carry the primary host/port (schema fields already exist, controller.capnp:598-599, just unpopulated in join.c). Verified E2E via xce_test (the original reproduction). Closes the integration gap left by ha-config-bootstrap Req 7 (tasks 10/10.1/10.2 shipped only the standalone primitive). See bug-608. Dependencies: ha-config-bootstrap (primitive + error codes), ha-commit-barrier-rollout (write idempotency makes write-retry safe). IMPLEMENTED, validate-impl GO 2026-07-18 (all 19 tasks were done since 2026-06-18; the ready_for_implementation flag was just never flipped — administrative staleness, not a real block). No longer in flight. component-failover-adoption (backlog, below) is the next layer on top of this.

Raft TLA+ Conformance — raft-library enhancement series (NEW 2026-06-21)

Goal: evolve the election-only src/lib/raft/raft.{c,h} into a STRICT implementation of raft/raft.tla (Ongaro canonical) covering the log + commit half, keeping the pure callback-driven design so it stays independently verifiable via the deterministic simulator (raft_sim.c) against the TLA+ invariants (Election Safety, Log Matching, Leader Completeness, State Machine Safety). Motivation: bug-851 — the log protocol is currently hand-rolled across persist.c/repl.c/ha.c/ha_convergence.c and violates the TLA+ log invariants. Gap analysis (six gaps G1–G6): .kiro/specs/ha-term-safety/TLA-gap-analysis.md. Implement in dependency order; each spec lands with simulator coverage. The library core stays pure/synchronous/deterministic — all I/O (async vtask, fan-out replication, the sync commit barrier, snapshot transport) lives in controller-side callback adapters. Preserve as policy (do NOT canonicalize away): PreVote, priority leadership, synchronous commit-barrier + persist-before-ACK durability, "controller never self-terminates".

raft-log-replication — (specs generated + auto-approved 2026-06-21: 8 reqs, 24 sub-tasks, cross-spec review PASSED. Ready for /kiro-impl.) Extend the library to OWN the replicated log. TLA+ actions: ClientRequest, AppendEntries, HandleAppendEntriesRequest, UpdateTerm-on-AppendEntries. Per-entry term in the library log model (closes G1); AppendEntries logOk = prevLogTerm match (G2); conflict branch deletes the divergent (term,lsn) suffix (G3); ANY higher-term RPC incl. AppendEntries/heartbeat advances currentTerm and steps down (G5). New pure callbacks: on_persist_entry, on_get_entry_term(lsn), on_send_append_entries(peer, prevLogIdx, prevLogTerm, entries, commitIdx), on_truncate_from(lsn); controller adapters implement them over the existing WAL/repl transport. Simulator: drive AppendEntries across partitions/restarts, assert Log Matching. Dependencies: raft-library (election core). Closes bug-851 G1/G2/G3/G5.
raft-commit-safety — (specs generated + auto-approved 2026-06-21: 6 reqs, 18 sub-tasks, cross-spec review PASSED.) AdvanceCommitIndex with the Figure-8 current-term rule (commit Max(agreeIndexes) only if log[Max].term == currentTerm) (G4); RequestVote/PreVote freshness becomes (lastLogTerm, lastLogIndex) instead of LSN-only (G6). New callback on_commit_advanced(lsn) feeds the existing synchronous commit barrier / durability (persist-before-ACK preserved as policy). Simulator: assert Leader Completeness. Dependencies: raft-log-replication. Closes bug-851 G4/G6.
raft-sim-conformance — (specs generated + auto-approved 2026-06-21: 8 reqs, 23 sub-tasks, cross-spec review PASSED.) Extend raft_sim.c into a conformance/verification harness driving the library through raft.tla-action-equivalent transitions (randomized + targeted) and asserting the four Raft safety invariants; optionally trace-check against TLC output of raft.tla. The "use the verified spec to help" payoff: a regression oracle for the consensus core. Dependencies: raft-log-replication, raft-commit-safety. IMPLEMENTED, validate-impl GO 2026-07-18 (23/23 tasks; 293/293 lib tests, zero ASAN findings; this is the spec this worktree/branch — raft_tla — is named for).

ha-term-safety (.kiro/specs/ha-term-safety/) is the XCE-side INTEGRATION + REGRESSION umbrella for this family: it wires the controller WAL / commit-barrier into the new raft callbacks (thin adapter — protocol delegated to the library), owns the config-change marker (R7 — config changes as (term,lsn) log markers; bytes via the existing out-of-band config-push transport, hash-verified against the marker), and holds the live bug-851 chaotic-failover regression. Its protocol invariants R1–R6 are realized by the library specs above (not re-implemented in the controller). Scope: R1–R7 only. The former R8 (config-upgrade full-snapshot consistency, bug-857) was descoped 2026-06-21 to the independent config-upgrade-snapshot spec (below), because it requires entity-bearing snapshots that do not exist yet. IMPLEMENTED, validate-impl GO 2026-07-17 (20/20 tasks; closed bug-1445, a first-boot promotion-check race with job-array-single-wal-record's Live Apply Resource Fence, as an out-of-boundary follow-up fix).

raft-adapter-conformance (.kiro/specs/raft-adapter-conformance/) — NEW 2026-07-29, sibling audit to the library-side conformance work above: a systematic, action-by-action pass over the controller's Raft adapter (the wire-level send/receive/reply code in ha.c/repl.c/config_repl.c) against raft.tla's actions — distinct from raft-sim-conformance's in-process library harness. Motivated by a live-soak finding that the controller's WalAckReply constructor hardcoded success (no failing reply ever reached the primary) plus a standby catch-up streaming-range self-deadlock, both already fixed (bug-1479) before this spec was seeded. This pass found and fixed two more instances of the same defect class: configuration-push rejects previously sent no reply at all (bug-1480), and a vote/PreVote endpoint failure arriving after connect routed through a legacy tally path instead of the production one (extending bug-1423's connect-time fix to that remaining timing window). Also records, as a single explicit decision, which AppendEntries mechanism is production-authoritative (repl_broadcast + ha_stream_wal_to_standby; the per-peer index-driven raft_send_append_entries stays dormant by documented design, not oversight). Requirements/design/tasks generated and approved, all 15 leaf tasks (1–7.3) checked; src/controller/CLAUDE.md's Protocol Verification Checklist marks the corresponding invariants "post raft-adapter-conformance, landed 2026-07-29" — spec.json phase (tasks-generated) has not been flipped to reflect this, same administrative-staleness pattern seen elsewhere in this doc (cf. the former libxce-failover entry above). Dependencies: raft-log-replication, raft-commit-safety, ha-term-safety, raft-sim-conformance (adjacent/informational — this spec fixes the adapter layer those validate underneath).

Raft Apply-Ordering & Hook Decoupling — NEW 2026-06-24 (`/kiro-discovery`); IMPLEMENTED — see `raft-apply-decoupling` below

Goal (historical — achieved): make the controller follow canonical Raft ordering — pre-hooks → raft commit → apply → notify — with the raft protocol step lifted OUT of the notification hook fan-out. At the time this family was opened, persist_job_hook_fun (an AFTER_EVENT hook) performed the WAL append + synchronous majority-commit barrier inline, and create_job() mutated in-memory state BEFORE that commit — the structural root of the bug-795/828/851/1060 divergence family. A failed/timed-out commit did not cleanly revert the optimistic mutation. Decision (user, 2026-06-24): apply-after-commit (canonical) — a single shared apply_committed_entry() seam mutates the state machine only after commit, reused by restart replay so online-apply and replay-apply are identical. Hooks demote to notification-only. Decomposed via Path E (2 new specs + touchpoints), delivered as raft-apply-decoupling (IMPLEMENTED 2026-07-07, see below). Current code reflects the fix: persist_job_hook_fun (persist.c ~6509-6553) now explicitly returns early for WAL_MODE_WAIT entries, with a comment noting job-submit durability is invoked directly by handle_jobsubmit — persist_submit_job_payloads_and_commit(), called strictly before any job object is constructed — never from this hook; create_job() is now defined at job.c:48.

Async-drive prerequisites (surfaced 2026-06-25 by blocking audit; thread-role contract now in `ha-async-io.md`)

The protocol drive is NOT fully event-driven. Root cause: peer_ctrl_msg_handler runs inline on iomgr instead of dispatching to the vtask pool, and delaymgr (the heartbeat driver) was the documented blocking sink. Corrected thread-role contract is now steering (ha-async-io.md, 2026-06-25; SUPERSEDES "delaymgr for blocking ops = Permitted"). Two separable prerequisites (full evidence: .kiro/specs/raft-apply-decoupling/facts-in-raft.md § Keystone + "Two distinct prerequisites" + P2 offload mechanism):

P1 — commit-barrier async (the commit drive). Convert repl_wait_majority_ack() from blocking pthread_cond_timedwait (commit.c:359) to event-driven resolve(lsn,outcome) on the existing commit engine. Dissolves apply-self-deadlock (B), enables reply continuations (C), removes vtask-pool exhaustion. Owned within raft-apply-decoupling as its keystone (sequenced before its Phase A).
P2 — drive hardening (the transport/timer drive). Now its own spec raft-drive-async (brief: .kiro/specs/raft-drive-async/brief.md) — promoted because it threatens liveness independently. See the Specs list below.

Specs (dependency order — Raft Drive & Apply Async family)

raft-drive-async — IMPLEMENTED, validate-impl GO 2026-07-18 (21/21 tasks; the live sequential soak this validation ran surfaced a real crash, bug-1447 — a same-batch endpoint use-after-free in io_worker's epoll dispatch loop, exposed by this spec's own faster reconnect cadence, see ha-async-io.md; root-caused, fixed, re-soaked clean, re-validated GO same day). Original description retained below for provenance. (was: START FIRST — threatens failover liveness TODAY) — bring the code into compliance with the corrected ha-async-io.md thread-role contract: dispatch peer_ctrl_msg_handler to the vtask pool (root-cause fix); snapshot send + config-push off iomgr → admission-capped slow lane (per-peer dedup + WAL coordination); delaymgr peer connect → bounded non-blocking + offloaded (reuse xce_probe_config_tag pattern); add a general non-IO vtask-submit path + admission reservation (extend COMMIT_WAIT_RESERVED). Verify with a partition (unreachable-not-killed) soak: heartbeats to survivors continue, no spurious election. Likely root of the bug-851 chaotic-failover family. Dependencies: none hard (builds on existing commit engine + vtask + the steering contract); election-iomgr-refactor is completed upstream (provides delay_task_ms/on_connected/non-blocking vote fan-out) — STANDALONE per 2026-06-25 scope-check (ADJACENT-SAME-AREA: it fixed startup blocking, this fixes runtime drive blocking). Brief + requirements + research + design + tasks done; phase=tasks-generated (tasks approved: false). /kiro-validate-design GO 2026-06-25 — three review findings folded into design+tasks: (1) deadlock-freedom + LivenessAdmission — the documented invariant reversal (ack/commit_on_ack moves off iomgr to the vtask pool, commit.c:16) co-locates commit-barrier wakers+waiters; review found COMMIT_WAIT_RESERVED is insufficient because it counts only commit-waits while peer_ctrl also puts vote-grant fsync + config-ack waits on the pool (invisible to it). Fix: generalize the reservation to a pool-level task-class admission in vtaskmgr (VTASK_LIVENESS/LIVE_BLOCK/REQ_BLOCK; liveness never gated; COMMIT_WAIT_RESERVED folded in) — new Phase-1 task 1.6 (dormant infra), hard prerequisite of 3.1; task 3.4 is the admission gate. Transient until raft-apply-decoupling P1 makes the barrier async; (2) WAL-trim cursor-bound guard added to task 3.2 (bug-1147, see Hardening below — moving checkpoint/trim onto a concurrent lane worsens the existing latent trim-vs-cursor race); (3) in-process partition primitive in xce_test as the portable primary for R7 (firewall DROP secondary) so the soak actually runs unprivileged. Next: approve tasks (/kiro-spec-tasks raft-drive-async) → /kiro-impl; ship Phase 1 (bounded connect) first.
raft-apply-decoupling — decouple the raft protocol step from the hook fan-out (item #1); canonical pre-hooks → commit → apply → notify ordering (item #2); single apply_committed_entry() seam shared by online commit + restart replay; post-restart validation that committed log was correctly applied to materialized state (item #4); requirement + regression tests (item #5). Contains P1 (commit-barrier async) as its keystone. Commands (WAL_MODE_WAIT) apply-after-commit; facts (WAL_MODE_RETAIN) routed through commit-then-apply too — D1 RESOLVED (block-but-durable under quorum loss); formalized as ha-consistency-invariants criterion 2.9. Dependencies: raft-drive-async, raft-log-replication + raft-commit-safety (DONE/implemented), worker-rejoin-resync; coordinate with ha-term-safety. Briefs: .kiro/specs/raft-apply-decoupling/brief.md + facts-in-raft.md. IMPLEMENTED 2026-07-07: 32/32 tasks, validate-impl GO — 79/79 controller_unit_test groups (926 tests) green, live 3-node smoke boot verified, zero blocked tasks. Task 7.2 fixed bug-1416 (boot-time TOCTOU false APPLY_DIVERGENT fence). Task 7.4's own throughput benchmark found and fixed a real O(n²) apply-pump regression (56% throughput drop, superlinear scaling) via a segment-read cache on persist_t, 2 adversarial review rounds. Residual, non-blocking: design.md has 3 sections not yet reconciled with implementation deviations already self-documented in tasks.md's Implementation Notes (repl.c's planned repl_submit_majority_commit() was never built; apply pump wired via commit_resolve_pending() not ha.c's on_commit_advanced; fact hooks collapse to generic HOOK_DETAIL_JOB_UPDATE not per-type); task 5.3 touched ha.c without a formal _Boundary:_ exception tag (substantively justified, process-inconsistent only). Two out-of-scope findings flagged for follow-up: an intermittent client hang in the synchronous jobsub path (resembles but not confirmed as bug-1416's class), and trim_wal_segments's retention floor has no awareness of the local apply pump's own lag (same family as an already-tracked standby-cursor gap).
worker-rejoin-resync — PROMOTED to spec 2026-08-10 (/kiro-discovery; brief at .kiro/specs/worker-rejoin-resync/brief.md). Elevated to HARD prereq for the facts half of raft-apply-decoupling 2026-06-25 (was backlog "designated backstop"). Worker re-reports running/finished job state on rejoin so facts lost in a failover window are recovered; re-reports flow through commit-then-apply idempotently (criterion 2.9). Today workers send only hostattrs on join (bug-396). Scoped narrowly to worker job-execution-fact resync only — apiserver/scheduler resync-on-reconnect is a different data shape (cluster/queue/job list state, not job execution facts) and stays out of scope (see the deferred "lightweight rejoin re-attach" backlog item below, still YAGNI). Dependencies: ha-consistency-invariants (commit barrier + idempotency guards) — both implemented, unblocked. Next: /kiro-spec-requirements worker-rejoin-resync.
job-array-single-wal-record — PROMOTED from backlog 2026-06-24. Persist/replicate/push a job-array submission as ONE Raft record (base spec + index range/count + per-index overrides) committed atomically through a single barrier; expand on apply (online + replay) and on the scheduler side; preserve per-element identity/addressing; idempotent re-apply (item #3). Dependencies: raft-apply-decoupling (HARD — builds on the shared apply seam), ha-commit-engine, ha-commit-barrier-rollout, ha-consistency-invariants. Brief: .kiro/specs/job-array-single-wal-record/brief.md.

Existing Spec Updates (Raft Apply-Ordering phase)

raft-sim-conformance — add apply-after-commit regression assertions (item #5): no apply on a failed/uncommitted entry; online-apply ≡ replay-apply for the same committed log; ordering invariant pre-hooks → commit → apply → notify. Revisit with /kiro-spec-requirements raft-sim-conformance once raft-apply-decoupling requirements land.
ha-term-safety — coordinate: raft-apply-decoupling restructures the same controller submit→commit path this umbrella wires. Sequence the apply-decoupling edits against ha-term-safety's adapter work (disjoint-but-adjacent regions) to avoid conflicting refactors.

Direct Implementation Candidates (Raft Apply-Ordering phase)

xce_test waldump/walcompare/snapdump inspectors (already a Hardening item below) double as the offline evidence for the item #4 restart-validation requirement — wire their nonzero-exit-on-divergence into the decoupling spec's E2E rather than as a separate spec.

Specs (dependency order)

[x] = spec generated (requirements/design/tasks approved, ready for implementation). Implementation tracked via each spec's tasks.md; implement in roadmap order (cross-spec review 2026-06-11: four specs touch ha_dispatch_gate / raft_cb_become_follower in disjoint-but-adjacent regions).

ha-commit-engine — commit.c per-LSN majority engine + quorum accounting fixes, wired behind job submit. Dependencies: none. (specs done 2026-06-11; IMPLEMENTED 2026-06-12: 12/12 tasks, validate-impl GO, 467 tests green; deferred follow-ups in its tasks.md §Implementation Notes)
ha-commit-barrier-rollout — classification table, commit-first-then-reply sweep, RETAIN facts, config barrier, idempotency guards. Dependencies: ha-commit-engine. (specs done 2026-06-11; IMPLEMENTED 2026-06-13: 16/16 tasks, validate gate GO, 508 controller tests green under ASAN. Fixed bug-447 (key_find found-rc leak — queue/host control was silently unpersisted+unreplicated in production) and bug-396 (worker fact loss on quorum-loss gate rejection); task 5.2 needed one remediation round (wired the missing production config-ACK ingestion — would have bricked multi-member config upgrades — + kept the barrier off the iomgr thread). Logged not-fixed: bug-448 (job_ctrl ref leak), bug-451 (RUSAGE va_list OOB read). Barrier-before-state-removal precondition for ha-single-leader-role satisfied)
ha-single-leader-role — promotion no-op + own-term gate, transitional-state (the former fourth role) removal. Dependencies: ha-commit-engine, ha-commit-barrier-rollout (HARD: barrier before state removal). (specs done 2026-06-11: 2 reqs / 7 components / 11 sub-tasks)
ha-convergence-recovery — demotion truncate+self-restart, rejoin reconciliation, egress watermark filter, ACK-driven catch-up, quorum logging. Dependencies: ha-commit-engine (E2E additionally needs ha-single-leader-role). (specs done 2026-06-11: 9 reqs / 7 components / 13 sub-tasks)
ha-commit-durability — closes bug-661 (acknowledgment did not imply durability; committed jobs lost across failover). Makes acknowledged ⇒ durable on a majority: the primary counts its own copy toward the commit majority only at its durable LSN (majority_candidate_locked caps self at persist_durable_lsn, not the appended tip); fsync_worker repurposed timer→demand-driven group-commit flusher (one fdatasync over pending appends, advance durable_lsn, wake commit-eval) — absorbs the former "WAL group-commit batching" backlog item (see below); recovery integrity verdict (clean vs corruption-truncated WAL) + durable-backed freshness floor (candidate advertises bare persist_durable_floor, voter floor = max(durable_floor, learned_committed)) + pre-promotion eligibility fence. Establishes the controllers-never-self-terminate invariant: a recovery-truncated/behind member that cannot catch up is a LOUD PERMANENT FENCE (non-promotable, stays alive, keeps retrying — never exit/abort), self-heals on peer return (see ha-async-io.md process-lifecycle invariant + src/controller/CLAUDE.md). Dependencies: ha-commit-engine (consumes commit-position semantics), ha-wal-replication (standby ACK/catch-up), ha-commit-barrier-rollout (reply gate, unchanged). (specs done 2026-06-18; IMPLEMENTED 2026-06-18: 17/17 tasks (1.1–5.3), validate-impl GO, controller suite 0 failures under ASAN; deterministic ack-before-fdatasync E2E RED→GREEN + rolling-SIGKILL soak zero acked-job loss. Verified by tests + Protocol Verification Checklist, NOT by raft.tla (which assumes a durable log). Fixed bug-760 (adjacent test_fact_hook_append raced the async flusher — test made durable-deterministic via persist_flush). raft.tla durability-model extension deferred unless explicitly opted in.)
scheduler-allocator-runner — NEW 2026-09-28 (/kiro-discovery, Path C). The framework owns job_alloc_fn through a new xce_sched_allocator_iface_t (accepts guard + plan iterator) and a single sched_allocator_run(); both my_alloc()s are deleted (seqjob/parjob), and alloc_iter.h init is retyped to xce_sched_core_t*. Behaviour must stay identical (bug-1882/1767/1768). Closes issues/new_requirements.md item 4. Brief: .kiro/specs/scheduler-allocator-runner/brief.md. Dependencies: scheduler-allocation, scheduler-core (both done). IMPLEMENTED 2026-09-30 (11/11 tasks, e2266582..28cff179; validate-impl GO; scheduler ASAN suite 96/96): runner in src/scheduler/src/alloc_runner.c; decided along the way: the runner clears alloc_req.host before every candidate so every short host gets pend reasons (both allocators), and fixes bug-1893 (shared reasons were never recorded). Follow-ups: publish pend reasons only when changed; sim-mode parjob allocation parity fixture; bug-1892; bug-1897.
scheduler-alter-pend-reasons — NEW 2026-09-28 (/kiro-discovery, Path E). An ALTER-resreq job keeps a pend reason for EVERY alternative instead of only the last (scheduler.c:424 FIXME). Wire shape (user decision): additive alternatives @3: List(JobPendReason) on JobPendReason, per-alternative attribution; carried end-to-end through controller job record, WAL preason fact, recordings and job_preason_eq. Also fixes the confirmed obj->reason overwrite leak (scheduler.c:388). Brief: .kiro/specs/scheduler-alter-pend-reasons/brief.md. Dependencies: libxce-alloc-preason-equality (HARD: extends its job_preason_eq); sequence after scheduler-allocator-runner (its "no pend-reason content change" rule).
application-profile — NEW 2026-09-29 (/kiro-discovery, Job Sandbox Execution initiative, Path D; reshaped from job-execution-mode by user direction: execution mode is a job attribute determined by the APPLICATION). Admin-defined named application profiles (LSF -app precedent) in cluster config: execution mode + options (direct|sandbox, room for container/vm), launch type, env + pre/post exec, resource defaults/limits (extensible). Users pick a profile at submit (--app), cluster default profile otherwise; queues (scheduling policy) and profiles (application config) are orthogonal — neither restricts the other; only profile-marked overridable attributes may be overridden; access by RBAC permission (new app:use/app:view with app domain type, folded into this spec); apiserver checks profile existence; controller resolves at submit and persists effective values on the job (wire additive, WAL/snapshot field-completeness). apiserver never checks node accounts; worker fails a direct job with a clear reason if the exec user is missing. Worker capability advertisement + scheduler placement. Brief: .kiro/specs/application-profile/brief.md. Dependencies: none.
job-sandbox-execution — NEW 2026-09-29 (Job Sandbox Execution). Worker: pluggable execution-backend interface (direct refactored + systemd-sandbox; non-systemd backends later). Sandbox = existing transient unit + DynamicUser with fresh per-job name (never the submitter's), ProtectSystem=strict/ProtectHome/PrivateTmp/NoNewPrivileges, Bind(ReadOnly)Paths for job I/O dirs or worker-created temp workspace (handed to the uid via SO_PEERCRED), StandardOutput=file: output, helper auth by unit/cgroup, MANDATORY post-job cleanup/ownership normalization (dynamic uid reuse), systemd ≥ 242 + nss-systemd capability probe; single-node incl. single-node MPI. Viability-checked 2026-09-29. Brief: .kiro/specs/job-sandbox-execution/brief.md. Dependencies: application-profile; remote-task-root fix (Direct-Implementation).
job-sandbox-multinode — NEW 2026-09-29 (Job Sandbox Execution). Multi-node sandbox jobs: cluster-consistent ephemeral uid (expected: controller-allocated, WAL-replicated uid lease from a reserved range), worker-driven remote/MPI launch (no ssh; PMIx/prted via worker), shared-FS ownership normalization, one unit per job per node. Brief: .kiro/specs/job-sandbox-multinode/brief.md. Dependencies: job-sandbox-execution, application-profile.
rbac-persistence — NEW 2026-09-29 (/kiro-discovery, raised during application-profile discovery). Closes the deferred security.md gap: runtime RBAC mutations (roles, permissions, user-role assignments, Logon auto-assignment) are in-memory only and silently revert to rbac.yaml on apiserver restart — including revocations. Durable-or-rejected mutations, audit trail, defaults-vs-runtime reconciliation, multi-apiserver consistency; storage approach (rbac.yaml write-through / controller-replicated entity / embedded store / read-only) decided in design. Brief: .kiro/specs/rbac-persistence/brief.md. Dependencies: none (independent of the sandbox initiative; application-profile's API-made app:use grants rely on it for durability).
job-time-window — NEW 2026-09-29 (/kiro-discovery, Path C). Job begin time (scheduler gate in sched_job_is_ready() via xce_core_now, pend reason BEGIN_TIME_NOT_REACHED) and end time (WORKER enforces running jobs: helper SIGTERM→grace→SIGKILL, exit reason END_TIME_REACHED persisted, unit RuntimeMaxUSec backstop; PEND jobs whose end time can't be met stay PEND with END_TIME_UNMEETABLE until user kills/modifies — no controller sweep, primary not involved). Adds runtime estimate (-We/--estimated_run_time, scheduling hint only, never enforced; estimate > run limit rejected); unmeetable rule now + (estimate ‖ run_limit ‖ 0) > end. Times/estimate modifiable only while PEND (new xce jobs modify + harness jobmod). Both xcecli and harness jobsub gain all options (-b/-t/-W/estimate). Also fixes the silent drop of xcecli -b/-t/-W and REST run_time_estimation (apiserver modelJobToSubmit never sets them nor exdata). Brief: .kiro/specs/job-time-window/brief.md. Dependencies: none (new reason codes need text in job-status-detail-display).
job-dependency — NEW 2026-09-29 (/kiro-discovery, Path D; ref LSF 10.1 Admin Guide "Job dependency and job priority" pp. 459-465 — dependency only, priority deferred). -w expressions by job ID: done/ended/exit(id[,op code])/started/post_done/post_err (PDONE/PERR) + bare id, && || ! (). Shared libxce grammar (Go via cgo); apiserver early validation (syntax, parent exists, submitter has job:view); controller authoritative re-validation, stores expression (additive JobSubmitData field) + in-memory reverse index rebuilt on replay (implements controller-job-cleanup's job_has_dependents()); scheduler evaluates in sched_job_is_ready() (pend reasons DEP_NOT_SATISFIED / DEP_NEVER_SATISFIED; orphans stay PEND). Fixes scheduler exit-code field-completeness gap (xce_sched_job_t/SchedJob/snapshot lack exit code). xcecli -w, modify -w/-wn; harness jobsub -w/jobmod. Brief: .kiro/specs/job-dependency/brief.md. Dependencies: job-time-window (shared gate + modify path) — coordinate.
job-dependency-insight — NEW 2026-09-29. bjdepinfo-style query (parents/children/-p blocking/-r levels/-l verdicts, RBAC-filtered) via REST/xcecli/harness, dependency info in job details, optional orphan termination (per-job -ti + cluster grace period, default off; EXIT reason ORPHAN; failover-safe persisted grace start; minimal primary periodic work). Brief: .kiro/specs/job-dependency-insight/brief.md. Dependencies: job-dependency, job-time-window (exit-reason field).
controller-job-cleanup — NEW 2026-09-29 (/kiro-discovery, Path C). Primary-only periodic sweep (configurable job_cleanup_period, default 300 s) removes DONE/EXIT jobs finished longer than configurable job_retention (default 3600 s, 0 = off) with no dependents (stub; XCE has no job dependencies yet), via a NEW replicated WAL job-removal fact applied on every node (first-ever jobmgr delete; refcount-safe xkvmgr delete), propagated to scheduler and apiserver (OnJobDeleted). Retires the node-local snapshot-only job_should_retain finished-job filter (hard-coded 3600 s; causes snapshot/memory job-set drift). HA/persistence → full 3-phase workflow. Brief: .kiro/specs/controller-job-cleanup/brief.md. Dependencies: none (extends ha-entity-snapshots).
job-status-detail-display — NEW 2026-09-28 (/kiro-discovery, Path E). End users (REST/xcecli) see NO pend reason for ANY job today: the apiserver cgo conversion (backend/convert.go) never maps xce_job_t.pend_reasons, so pending_reason is always empty. Maps it to a structured JSON (including alternatives from day one) using libxce's code→text mapping, and renders it in xcecli job detail. Open for requirements: also map run_usage → JobRusage (same gap; reserved vs. used for RUN jobs). Brief: .kiro/specs/job-status-detail-display/brief.md. Dependencies: none for flat reasons; scheduler-alter-pend-reasons for per-alternative rendering.
job-config-merge — NEW 2026-10-09 (/kiro-discovery, Path E "configuration layering"; ref LSF 10.1 Admin Guide pp. 314-316, 435-437, 561-563, 643, 660). One merge function in src/lib turning the job's own values + application profile + queue policy + project into effective values (resreq incl. span, task range, limits, pre/post-exec, env). One owner per attribute (user decision; .kiro/steering/job-config-layering.md): job overrides a profile attribute only if it is default (for select only by adding ANDed terms); queue ceilings and project quotas only check (reject if never satisfiable, pend if full now). Live merge (user decision): WAL keeps inputs + names; controller/apiserver validate at submit; scheduler computes effective values and re-evaluates pending jobs on config change. Queue is policy only: queue res_req and pre_exec/post_exec deprecated (pre/post-exec moves to the profile), dropped queue limits (cpulimit/memlimit/runlimit/task limit) wired. Brief: .kiro/specs/job-config-merge/brief.md. Dependencies: none.
project-config — NEW 2026-10-09 (configuration layering). Projects control resource ownership, v1 = quota: project definitions in cluster config (members, priority, cluster default), default-project resolution, submit access check, scheduler pgmgr loaded from config so PROJECTS/PER_PROJECT limits work end to end, per-project usage display. Brief: .kiro/specs/project-config/brief.md. Dependencies: none.
host-failure-job-recovery — NEW 2026-10-10 (/kiro-discovery, "Long-running services" phase 1; also a batch gap: RUN jobs on a dead host stay RUN forever, controller/sched.c:631 FIXME). After a grace period, jobs on a lost host go to an unknown state, then are requeued (rerunnable, job option or profile default) or end EXIT "host lost"; multi-host jobs resolved as a whole; attempt numbers fence a stale copy when the host returns; manual requeue (REST/xcecli). Brief: .kiro/specs/host-failure-job-recovery/brief.md. Dependencies: none.
service-supervision — NEW 2026-10-10 (Long-running services phase 1). Application profile kind service: restart policy (always/on-failure/never, backoff, give-up), liveness and readiness probes (command/TCP/HTTP) run by each replica's own worker (no special replica), in-place restart, change-only health reports, display. Brief: .kiro/specs/service-supervision/brief.md. Dependencies: application-profile, host-failure-job-recovery.
service-replica-set — NEW 2026-10-10 (Long-running services phase 2; user decision: no head agent, no promotion). Controller-held service object owning independent replica jobs; event-driven rules on commit (replace ended replicas with backoff, scale, rolling update with max-unavailable/surge, pause/resume/rollback); spread rule; REST/xcecli services. Brief: .kiro/specs/service-replica-set/brief.md. Dependencies: service-supervision, host-failure-job-recovery, application-profile.
service-endpoint — NEW 2026-10-10 (Long-running services phase 2). Named ports allocated per replica as host resources, host addresses reported by workers, endpoint = host:port of ready replicas from readiness changes, REST/xcecli queries; DNS/proxy later. Brief: .kiro/specs/service-endpoint/brief.md. Dependencies: service-supervision, service-replica-set.
provisioning-demand — NEW 2026-10-10 (/kiro-discovery, "Auto resource provisioning"; LSF resource-connector precedent). Scheduler knows a provisionable catalog (host templates, license headroom, supply in flight), tries it after real resources fail (templates as virtual hosts, headroom shortfall), finalizes demand and idle supply in the end-of-session handler (preferred design) and publishes it to the controller. Simulator support is a hard requirement (same demand as live, REPL display, recordings, simulated provider responses). Brief: .kiro/specs/provisioning-demand/brief.md. Dependencies: none (licenses need resource-distribution-mapping's shared pools).
resource-provisioner — NEW 2026-10-10 (Auto resource provisioning). New daemon (src/autoscaler, an LSF RC-style port, is prior art only, not a required basis): several instances, each owning providers via controller leases (cloud and license provisioners may differ); providers describe capacity, cost, priority, lead time, granularity, hold time; script contract v2 (hosts and counts); automatic policy (no approval), budgets, cooldowns, retries, idle reclaim; request/instance state as controller facts; orphan cleanup; REST/xcecli/audit; simulated provider. Brief: .kiro/specs/resource-provisioner/brief.md. Dependencies: provisioning-demand.
elastic-hosts — NEW 2026-10-10 (Auto resource provisioning). Host templates in config, expected-host records matched at join, provisioned/draining/decommissioned states, drain-then-release, decommission control, spot loss as host loss. Brief: .kiro/specs/elastic-hosts/brief.md. Dependencies: resource-provisioner, host-failure-job-recovery.
elastic-shared-resources — NEW 2026-10-10 (Auto resource provisioning). Elastic shared resources (e.g. EDA licenses): base + borrowed total updated by the provisioner role through the controller, unused borrowed amounts returned, display. Brief: .kiro/specs/elastic-shared-resources/brief.md. Dependencies: resource-provisioner, resource-distribution-mapping.
custom-resource-reporting — NEW 2026-10-11 (/kiro-discovery, "Unified resource model" phase 2; resource-distribution-mapping option 3 split out, pending user decision 1 in resource-model.md). Runtime custom resource values: an elim-like script reporter on workers (posix_spawn) and/or an external reporter role (decision 2), staleness and interval enforced, a string-set kind for dataset locality, instance-inventory publishing (DRA ResourceSlice-like). Brief: .kiro/specs/custom-resource-reporting/brief.md. Dependencies: resource-distribution-mapping, peer-authentication-hardening.
resource-instances — NEW 2026-10-11 (Unified resource model phase 3). Identified instances (GPU, MIG slice, NIC, ports) with attributes; topology-aware allocator plugin driving the unused affinity[] (NVLink domain, GPU-NIC affinity; switch block via same[]/span[]); durable per-job instance assignment (WAL/snapshot field-complete); worker binding (device cgroup, CUDA_VISIBLE_DEVICES, port env). Brief: .kiro/specs/resource-instances/brief.md. Dependencies: custom-resource-reporting, scheduler-framework.
resource-claims — NEW 2026-10-11 (Unified resource model phase 4). Data and volumes as lifecycle resources: template claims (per job/replica: scratch, staging, KV cache) and standalone claims (K8s PVC/ResourceClaim-like: checkpoint volumes, model weights, dataset caches; access modes RWO/ROX/RWX/RWOP; retain/delete), claim WAL entity, dispatch readiness gate (stage-in/out), storage provider contract (reused from resource-provisioner), worker publish into the sandbox, orphan handling. Absorbs the sandbox workspace and transfer_files staging (bug-1937 deferral). Brief: .kiro/specs/resource-claims/brief.md. Dependencies: resource-provisioner, resource-instances, job-sandbox-execution.
gang-groups — NEW 2026-10-11 (Unified resource model phase 5, LLM training). All-or-nothing scheduling of a group of jobs (launcher + workers, JobSet-like), topology packing across the gang, start barrier until every member is placed and its claims are READY, group failure policy. Brief: .kiro/specs/gang-groups/brief.md. Dependencies: resource-instances, resource-claims, parjob-placement.

Existing Spec Updates (2026-10-11 unified resource model discovery)

Direction: .kiro/steering/resource-model.md. Decisions, proposals (PR-1 generic supply map incl. provisioned supply, PR-2 resource provider proxy, PR-3 datasets in the map with staging as provisioning) and open questions: .kiro/steering/resource-questions.md. Duration/decay: must-have, future spec (R8).

resource-distribution-mapping — phase 1: shared pools (option 2) plus the resource type descriptor fields (scope, unit, reporter, layer, instances, provider); option 3 moves to custom-resource-reporting unless the user keeps it here (decision 1).
service-endpoint — ports as identified instances of resource-instances instead of a separate port model.
job-sandbox-execution — the per-job workspace becomes the first template claim of resource-claims; publish (mounts) through the claim's worker step. Where data staging lives is decision 9.
application-profile — default claims (model weights, checkpoint) and GPU instance requests in profiles; container execution mode decides volume publishing (decision 6).
job-resize — re-bind a job's standalone claims to the new host set; resize a gang group.
service-replica-set — replicas share one read-many weights claim; rollout attaches version N+1 while N stays attached; per-replica template claims.
resource-provisioner — its provider contract, lease, epoch and orphan machinery are shared with storage/claim providers.
provisioning-demand — claim shortfall as provisionable demand.
peer-authentication-hardening — reporter and claim-provider roles on the per-role opcode allowlist.
scheduler-framework — the deferred GPU/affinity allocator plugin is delivered by resource-instances; honour rusage duration/decay, parsed but ignored today (alloc_runner.c:429; decision 8, licenses need it).

Existing Spec Updates (2026-10-10 auto resource provisioning discovery)

resource-distribution-mapping — choose its shared-pool scope (option 2) so a cluster-wide pool (licenses) can be declared in resources.yaml; prerequisite of elastic-shared-resources.
peer-authentication-hardening — a provisioner role with its allowed operations (provisioning facts, shared-total updates, expected hosts).
xce-node-manager — revise requirements (approved 2026-09-03) for the cross-spec interactions recorded 2026-10-10 in its requirements.md: shared supervision/probe module with service-supervision, a lease-based provisioner role, worker stop on decommission and role declaration for provisioned hosts, grace period vs worker restart, future service proxy under reverse-proxy.

Existing Spec Updates (2026-10-10 long-running services discovery)

application-profile — profile kind service section: restart policy, probes, ports, default replicas, spread rule, rollout settings (consumed by service-supervision, service-replica-set, service-endpoint).
job-config-merge — service attributes follow the ownership rule (profile owns them; the job may override only default ones, e.g. replica count).

Existing Spec Updates (2026-10-09 configuration layering discovery)

application-profile — the profile carries the job's resource requirement (select/rusage/span/same) and task limits (min/default/max); "resolved and frozen at submit" replaced by the live merge of job-config-merge (brief updated). Dependencies: job-config-merge.
parjob-placement — span read from the effective resreq; span constraints in the fixed-N planner; range search bisects over the planner's allowed counts (block: multiples; ptile: multiples + max); LSF 10.1 findings recorded in its requirements.md notes; span[stripe] candidate. Dependencies: job-config-merge (for span from profiles; job-level -R span can land first).

Existing Spec Updates (2026-09-28 `info job` discovery)

scheduler-sim-diagnostics — info job <jobid> enhancement: a PEND job shows its pend reason set (per alternative once scheduler-alter-pend-reasons lands; shows the flat reason until then); a RUN job shows per-host/per-resource reserved (job->allocs) vs. real used (job->rusages), and when no rusage has been reported yet says "resource usage not reported yet" (no fabricated zeros; reserved still shown). Byte units per its Req 6. Read-only. Next: /kiro-spec-requirements scheduler-sim-diagnostics. Dependencies: none (alternatives display depends on scheduler-alter-pend-reasons).

Future Specs / Backlog (dependency order)

project-guarantee — NEW 2026-10-09 (configuration layering, backlog). Project-owned guarantees (resource amount or host pool), borrowing of idle guaranteed capacity, reclaim through preemption. Brief: .kiro/specs/project-guarantee/brief.md. Dependencies: project-config, scheduler-preemption.
job-resize — NEW 2026-10-10 (from long-running services discovery; not on the services path). Grow/shrink a running parallel job's allocation (LSF resizable jobs, bresize) for elastic applications. Brief: .kiro/specs/job-resize/brief.md. Dependencies: parjob-placement, host-failure-job-recovery.
project-delegated-membership — NEW 2026-10-10 (from project-config requirements, user decision: future). Project managers edit their project's (and subtree's) members at runtime without an admin config push; needs a durable store that coexists with xceadmin config pushes (e.g. a separate delegated member list) and audit. Dependencies: project-config, rbac-persistence.
lsf-config-migration — NEW 2026-10-09 (user idea, backlog; no brief yet). A tool (likely an xceadmin subcommand) that converts LSF configuration (lsb.queues, lsb.applications, lsb.resources Limit sections, lsb.users/lsb.hosts groups, lsb.params defaults, project settings) into XCE configs/*.yaml, with a report of what does not map: queue RES_REQ (XCE queues are policy only; move into a profile or drop), LSF's full per-section merge rules vs XCE's simplified ones, compound/alternative resreqs, unsupported parameters. Dependencies: job-config-merge, application-profile, project-config, parjob-placement (span), admin-config-tool. Promote via /kiro-discovery when those land.
xce-harness-component — PROMOTED to spec 2026-09-13 (/kiro-discovery; brief at .kiro/specs/xce-harness-component/brief.md). Elevate xce_test (the REPL-driven wire-protocol test client used by HA soak / multi-host / resource-value verification) out of src/controller/testsuite/ into its own first-class src/harness/ component — own meson.build, CLAUDE.md, and unit-test binary (carved out of controller_unit_test, which today recompiles ~14 of xce_test's own files a second time paired with white-box tests) — fully decoupled from /xce-build controller (today's implicit "controller always builds xce_test" coupling, build-deployment.md:24). Verified finding: none of xce_test's 15 source files include any controller-private header — only libxce public headers + generated capnp schema — so this is a pure relocation + build-graph split, not a dependency untangling problem, and no xce_test behavior changes. Touches /xce-test-ha, /xce-deploy-ha, /xce-test-multihost, /xce-test-resources (each must build harness explicitly once decoupled) and is adjacent to (not overlapping) the three existing behavior specs xce-test-cli-ux/xce-test-e2e-primitives/xce-test-rc-file, whose file-path references go stale after the move. Next: /kiro-spec-init xce-harness-component.
config-upgrade-snapshot — descoped from ha-term-safety R8 on 2026-06-21 (requirements generated; .kiro/specs/config-upgrade-snapshot/). Closes bug-857: a configuration upgrade can leave members with DIVERGENT materialized state because today it does config-push + restart + per-node WAL replay under the new config (config_repl.c do_upgrade_config_with_bytes ~1001-1163), with no snapshot pinning an identical post-upgrade state (replay is config-dependent — lost_and_found re-homes orphaned entities). Approach: an upgrade materializes a full entity-bearing state snapshot at the config-change marker's (term,lsn) boundary, embeds the new config, and installs it on every member via the snapshot_transfer_s transport — no per-node re-derivation; supersedes ha-term-safety R7's interim config-push byte delivery for upgrades. Former HARD dependency entity-bearing-snapshots is CLEARED — it shipped as spec ha-entity-snapshots (IMPLEMENTATION-COMPLETE, validate-impl GO 2026-07-18; see Future Specs/Backlog below), and ha-term-safety (the config-change marker it anchors on) is also implemented. config-upgrade-snapshot's own spec.json now has blocked_on: []. Verify by reproducing bug-857 (upgrade during chaotic multi-kill, diff materialized state). Ready to promote to design.
component-failover-adoption — recorded 2026-06-16; depends on libxce-failover (HARD). Make the C daemons and the Go apiserver actually exercise libxce's transparent failover. The mechanism reaches them for free (worker/scheduler/controller link libxce_dep; apiserver uses it via cgo, src/frontend/apiserver/backend/clib.go), but two per-component integration items remain: (1) each component must hand libxce the full master list instead of a single endpoint (today even the CLI uses only conf->masters[0], client.c:122); (2) bounded retry makes the already-blocking xce_request block longer (seconds of backoff) — for the async C daemons (worker controller_connector.c, scheduler handler.c) this must be verified liveness-safe, i.e. it must not stall a thread that also drives heartbeats (the "never block the iomgr thread" rule). Scope: per-component master-list config + retry policy + liveness-safe blocking + per-component E2E failover for scheduler, worker, apiserver. (xce_test is NOT here — it is the in-spec verification client of libxce-failover.) Promote via /kiro-spec-init component-failover-adoption after libxce-failover lands.
~~worker-rejoin-resync~~ — PROMOTED to spec 2026-08-10, see In Flight (Raft Apply-Ordering family) above for current status and brief. No longer tracked here.
admin-config-tool — PROMOTED from backlog to spec 2026-07-06 (/kiro-discovery; brief at .kiro/specs/admin-config-tool/brief.md). NEW: xceadmin Python CLI for cluster operators — builds and deploys cluster configuration via direct TCP+TLS capnp wire to controller (NOT through apiserver). Two subcommands: (a) config build — directory scan: parse configs/{hosts,queues,resources,users}.yaml, validate against JSON schemas (configs/schema/*.schema.json), serialize ClusterConfig capnp struct to cluster.bin with SHA-256; (b) config push — zero-config dev (default localhost:9991, same as xce_test), single --controller flag for production, auto-discovers peers + primary from controller reply, sends OP_upgradeConfig, displays {versionTag,ackedCount,error}. Single dependency: controller.capnp. Pure Python (pycapnp + yaml + jsonschema + click), open-source ready (pip install). The tool is the client-side trigger for config-upgrade-snapshot's controller-side path. MUST validate against reserved name "lost_and_found" for queue/host (defence-in-depth). xcecli comparison: xcecli = user tool (HTTP/HTTPS → apiserver → CGO → controller, JWT auth); xceadmin = operator tool (direct TCP+TLS → controller, zero-config, same trust model as controller peers). Shared future: xcecommon Python package with capnp schema + wire protocol reusable by both tools. Dependencies: controller.capnp schema only. Blocks: nothing — ships independently. Controller-side upgrade snapshot path (config-upgrade-snapshot) was blocked on entity-bearing-snapshots; that blocker is now cleared (see Future Specs/Backlog above — ha-entity-snapshots implemented, config-upgrade-snapshot blocked_on: []). Next: /kiro-spec-requirements admin-config-tool.
component-rejoin-resync — PROMOTED to spec 2026-08-10 (/kiro-discovery; brief at .kiro/specs/component-rejoin-resync/brief.md) from the "lightweight rejoin re-attach" item below. apiserver + scheduler reconnect/rejoin resync efficiency — today each does a full re-join and receives full state again (whole job list, queue/resource/config state), not a delta; for a large cluster this is sizable but not yet a measured problem. Sibling of worker-rejoin-resync — disjoint data (list/config state vs. job-execution facts), NOT merged into that spec. Not started: first step is investigation (is reconnect cost actually a measured problem at realistic cluster sizes?) before committing to any delta/session-token mechanism — original YAGNI discipline still applies, this spec is the promoted vehicle for that investigation, not a rubber-stamped design. Original decision (libxce-failover design review) retained below for provenance. Dependencies: libxce-failover (rejoin hook — implemented), component-failover-adoption (daemon rejoin content — in progress, sequence alongside/after). Next: /kiro-spec-requirements component-rejoin-resync when ready to begin (not immediate).
~~lightweight rejoin re-attach (failover optimization)~~ — recorded 2026-06-16; deferred (YAGNI); apiserver/scheduler half PROMOTED to component-rejoin-resync 2026-08-10 (see above); worker half already promoted to worker-rejoin-resync. No longer tracked as a bare backlog item — provenance only. Original text: on controller failover a reconnecting daemon does a full re-join and apiserver/scheduler receive full state (e.g. the whole job list), not a delta; for a large cluster this is sizable but rare. Decision (libxce-failover design review): accept full-state rejoin — it is simple, correct, and still cheaper than a full-history replace with no snapshot (cf. IBM LSF). Future optimization: a session-token "re-attach" hello that lets the new primary skip re-pushing already-replicated state, or a delta-only state transfer keyed on the consumer's last-known committed LSN. Delta reconciliation is non-trivial (the primary must know what each consumer already holds). Promote only if failover-rejoin latency/bandwidth is measured as a problem at scale.
job-array-single-wal-record — IMPLEMENTED, validate-impl GO 2026-07-17 (reopened once 2026-07-16 for three user-directed residual gaps — bug-1443 bulk-insert partial-visibility, task 4.2's boot-replay fence gap, bug-1444 replay_wal array-blind tally — all closed, re-validated fresh GO). PROMOTED to a spec 2026-06-24 (/kiro-discovery; brief at .kiro/specs/job-array-single-wal-record/brief.md; now tracked under the Raft Apply-Ordering & Hook Decoupling phase above, sequenced after raft-apply-decoupling). Original note retained for provenance: Enhance controller & scheduler so a job-array submission persists, replicates, and pushes to the scheduler as a single WAL record rather than one record per array element. Today a job array fans out to N per-job state changes, so a 10k-element array means 10k WAL appends + 10k replicated commits + 10k scheduler pushes — amplifying commit-engine load, replication bandwidth, and failover replay time linearly in array size. Scope sketch: (a) controller writes one WAL entry describing the array (base spec + index range/count + per-index overrides) committed atomically through the commit barrier; (b) replication carries the single record so standbys reconstruct the whole array from one committed LSN; (c) scheduler receives the array as one push and expands locally. Must preserve per-element identity/state addressing after expansion (status, cancel, completion still operate per task id). Dependencies: ha-commit-engine + ha-commit-barrier-rollout (single atomic commit), ha-consistency-invariants (replicated scheduling state). Spec advised before implementation; promote via /kiro-spec-init "job-array-single-wal-record" when ready.
job-array-one-based-index — NEW 2026-09-30 (spec .kiro/specs/job-array-one-based-index/); IMPLEMENTED 2026-10-01, 30/30 tasks; validate-impl GO after one remediation round (full suite, HA array soak, multihost, live REST flow). LSF-compatible array numbering: xcecli jobs submit -J 'name[1-N]' / 'name[1]' create elements numbered 1..N (previously 0..N-1, and the indices in -J were ignored — bug-1896); submit reports the array identity and element count; elements are shown and addressed as <array-id>[k], control commands also take <array-id>[a-b]; unsupported LSF forms (other starts, index lists, steps, %N, interactive arrays) are rejected with specific messages; API-server max_array_size default 1000, clamped to 32767; running elements see XCE_JOBINDEX (0 for ordinary jobs); harness jobsub -a N numbers 1..N and $jobid[k] names elements; the controller rejects multi-element arrays starting at 0 from un-upgraded clients (E_XCE_ARRAY_START0). Supersedes job-array-single-wal-record Requirement 5 (identities unchanged, numbered from 0) and the xce-test-e2e-primitives $jobid/expect-job definitions (superseded notes added in those specs). User documentation: src/docs/xce.md "Job arrays" section, including the LSF features not supported. Bugs: bug-1896 (indices ignored, numbered from 0), bug-1898 ([..] filters never evaluated; restart path fixed afterwards in c73708b8), bug-1899, bug-1900, bug-1901 fixed; found and fixed during validation: bug-1903, bug-1904, bug-1906, bug-1907. Open item: a live XCE_JOBINDEX check on a host with systemd (unit-tested only).
bulk job-array query/cancel — recorded 2026-07-13, deferred out of job-array-single-wal-record scope by design. A single client operation to query or cancel an entire array (not one element at a time) is explicitly out of scope for job-array-single-wal-record — see that spec's Boundary Context. Reason: array elements can be in heterogeneous states at request time (done/exited elements need no action; running elements need a worker kill signal; pending elements need a direct state transition to cancelled without ever running), which is a materially different bulk-operation problem from atomic array submission and needs its own requirements (partial-failure semantics, atomicity vs best-effort per element, etc.) rather than being folded into the submission spec. Dependencies: job-array-single-wal-record (per-element identity/addressing this would operate over). Promote via /kiro-discovery once job-array-single-wal-record ships.
~~WAL group-commit batching~~ — recorded 2026-06-17 as deferred; ABSORBED INTO ha-commit-durability 2026-06-18 (design-review reversal). The /kiro-validate-design review surfaced that the highest-volume WAL traffic is WAL_MODE_RETAIN facts (job start/finish/fail/rusage) — not array-collapsible and not covered by job-array-single-wal-record — so a pure per-entry synchronous design would penalize the dominant write path. Decision (with user): do group-commit now — repurpose fsync_worker from a timer into a demand-driven flusher (batch one fdatasync across pending appends, advance durable_lsn, wake commit-eval) as part of ha-commit-durability rather than deferring. No longer a separate backlog item.
primary step-down-on-quorum-loss — optional CP follow-on noted in ha-consistency-invariants brief (writes already blocked on quorum loss; step-down is an availability/clarity optimization).
read consistency / leader leases — Raft §8 read-only optimization; status queries may currently observe stale or provisional state (documented deviation in ha-consistency-invariants).
entity-bearing snapshots ("Task 11" format) — PROMOTED to spec ha-entity-snapshots (created 2026-07-01); IMPLEMENTED, validate-impl GO 2026-07-18 (21/21 tasks 1.1-8.1; 962 controller tests green including all 9 test_persist_snapshot_entities.c cases; live --force-snapshot soak: committed_divergence=0, committed_job_loss=0, confirmed real "HA: snapshot" install log line). write_snapshot() (persist.c ~2304-2372) now collects and serializes job/queue/host entity frames (collect_job_frames/collect_queue_frames/collect_host_frames) ahead of the header — no longer a bare 16-byte LSN sentinel. Original backlog note retained for provenance: real entity serialization unblocks snapshot-based state transfer and faster recovery. Cheaper than it looks (verified 2026-06-11): the dual-zarray COW machinery it needs was already built and unit-tested but dormant — xzarray_freeze/xzarray_thaw in the vtable (zarray.c:695-757, O(1) freeze, thaw with should_retain GC filter, test_zarray_cow.c) with zero production callers (kvmgr.c:297 vfreeze = NULL /* wired in P0-2 */). Note: single-generation COW — the snapshot consumer owns freeze/thaw; do not add second concurrent consumers. Slot-level only (snapshots array structure, not object contents — serialize item state under per-object locks during the freeze window). Also strengthens ha-consistency-invariants §C eviction option (iii) and faster rebuild-on-demotion. Downstream: config-upgrade-snapshot's former HARD dependency on this is now cleared (blocked_on: []) — see that item above. Also strategy B for catch-up of a far-behind / empty-rejoin standby (cursor=0) is now live: the standby-side snapshot-install/apply path is wired (ha.c ~3112-3147 wire-receive path task 6.1; persist.c ~4277-4285 local boot recovery task 8.1), with a decision path that requests a snapshot transfer when a standby's cursor falls below the leader's oldest retained WAL LSN (ha.c:4413). The bug-1147 WAL-trim cursor-bound guard (see Hardening below) is implemented in production, no longer just a stopgap.

Hardening / Direct-Implementation Items

Smaller items that likely don't warrant a full spec; pick up opportunistically or bundle into adjacent spec work.

remote-task units run as root (SECURITY, found 2026-09-29 during job-sandbox discovery) — job_start_remote_task path allocates task_ctx (src/worker/src/controller_connector.c:1204) and submits it (:1388) without ever setting task_ctx->limits; start_systemd_job() only appends User= when limits != NULL (xcecd_cgroups.c:296-312), so remote tasks of multi-host jobs run in a system unit as root. Confirmed by code reading (not yet live-reproduced). Fix: build limits (at least user_name = taskhost execuser) on the remote-task path; refuse to start with an empty user. Log via buglog before fixing. HARD prereq for job-sandbox-execution / job-sandbox-multinode.
~~eliminate src/frontend/Makefile, consolidate Go builds onto meson~~ — PROMOTED to spec 2026-08-27 (/kiro-discovery; brief at .kiro/specs/eliminate-frontend-makefile/brief.md). Investigation (recorded 2026-08-21, from dirty-content-fingerprint spec's task 4.3) confirmed the risk is real and independently corroborated: build-server is separately confirmed BROKEN today (bug-1597, bare go build with no CGO flags, fatal error: client.h: No such file or directory; src/docs/deployment.md:243-247 already tells developers to use /xce-build apiserver instead). Blast radius is narrower than originally framed: /xce-build already builds both binaries via meson exclusively (build-cli/build-server are dead weight from that path today); only 2 live call sites (xce-build.sh:240,287) depend on make test, with an already-proven direct-go test replacement 3 lines away in the same file. One genuine gap found: no meson test() exists for the CGO backend suite (test-backend) — that must be added before the Makefile can be safely removed, or backend CGO coverage is lost, not just inconvenienced. Next: /kiro-spec-init eliminate-frontend-makefile.
job_ctrl direct-mutate path — migrate to commit-then-apply — recorded 2026-07-13, investigated 2026-08-19 (finding: real risk, wider scope than originally framed). raft-apply-decoupling deliberately scoped job_ctrl's mark_job_update/mark_job_fail (admin SIGCONT/SIGSTOP) OUT-OF-BOUNDARY (task 4.1/6.1 Implementation Notes). The 2026-08-19 investigation found the original framing wrong in two ways: (1) NOT the only remaining call site — on_job_stop/on_job_resume/on_job_ckpt (job.c:2067-2222) share the identical unconverted mutate-then-notify-then-WAL-append pattern, fed by ordinary worker/scheduler traffic, not "rare, admin-only" as originally described; (2) blast radius (a) is confirmed real, not theoretical — the pre-commit mutation is synchronously pushed live to BOTH the scheduler (sched_job_update_fun) AND every connected apiserver/client (apiserver_job_update_fun) before the commit barrier resolves; on barrier timeout the admin gets a failure reply but job->job.state is never rolled back and no retraction is ever sent — matching the bug-795/828/851/1060 shape. Live-apply vs replay-apply (b) is narrower than feared for job_ctrl specifically (the allocs != NULL gate mutually excludes it from the converted deferred-apply fact types), but the investigation surfaced a SEPARATE bug underpinning that safety argument: job_ctrl (job.c:999) holds only a read lock while mutating job state — every other mutator in this file takes a write lock — so two concurrent admin commands on the same job aren't mutually excluded, undermining the "same lock, no staleness window" claim the replay-safety argument depends on (already flagged by a raft-apply-decoupling task 2.1 reviewer, never fixed). Recommendation: promote to a small follow-up spec covering all four call sites together (bounded, well-precedented fix — same shape as the fact-receipt conversion raft-apply-decoupling already built for job submit); the lock-type bug is independently fixable and low-risk regardless of when/whether the larger migration happens. Not yet promoted — user decision 2026-08-19 to correct scope/framing here first, promote later.
WAL-trim cursor-bound guard (bug-1147; surfaced 2026-06-25 by raft-drive-async design review) — IMPLEMENTED as part of raft-drive-async task 3.2 (raft-drive-async is IMPLEMENTED, validate-impl GO 2026-07-18; see above). trim_wal_segments (persist.c ~2597-2644) now computes floor = repl_trim_lsn(&p->repl, active_seg_lsn, p->config.snap_interval_s) — folding in active-standby acked_lsn and disconnected-slot last_disconnected_lsn — and only unlinks segments below that floor; the code comment at persist.c:2634 explicitly labels this the "(bug-1147 guard)". Far-behind/empty join (cursor=0) is additionally covered now that entity-bearing snapshots (strategy B) has landed as ha-entity-snapshots (see above), with a live decision path (ha.c:4413, "< oldest_wal_lsn → requesting snapshot transfer"). Related: bug-795, bug-828, bug-1060.
OP_heartbeat peers cross-validation — include a hash of the local --peers string in heartbeat requests; log ERROR on mismatch between candidates. Detects misconfigured nodes at election time instead of silently risking dual-primary split-brain. (FUTURE_WORK.md §3.)
peer-authentication-hardening — NEW 2026-07-06 from admin-config-tool discovery (/kiro-discovery). XCE currently operates under a trusted-LAN model — any CA-signed SSL peer can connect to the controller and perform ANY operation. Three gaps identified: (a) Admin→Controller: validate_host() is a stub returning success for all peers (roadmap § below). OP_upgradeConfig has no real authorization gate today. Same gap confirmed 2026-08-02 for OPCode_schedAdmin (scheduler-core design, cluster.c's case OPCode_upgradeConfig was checked as the closest precedent and has only a primary/standby routing gate, no RBAC/auth check) — scheduler-core ships OPCode_schedAdmin at parity with that existing gap (project owner decision: not a scheduler-core blocker), so this item now covers both opcodes; when validate_host() is implemented, wire it to both OP_upgradeConfig and OPCode_schedAdmin in the same pass. (b) Worker/Scheduler→Controller: workers and schedulers connect via controller_connector.c using the same TLS context as controller peers, with no role differentiation — a compromised worker could send OP_upgradeConfig or read full config via OP_replConfig. (c) Cross-component trust: no distinction between controller peers, admin tools, workers, and schedulers — they share the same X.509 identity pool. (d) xceadmin login (future): xceadmin currently uses zero-config TLS (same as xce_test). Production deployments likely need an admin login requirement — token-based session (xceadmin login → token → config push) or password-per-command. X.509 CN alone proves identity but not intent; a login step provides audit trail, session expiry, and operator accountability. Needs design: role-based X.509 CN validation (controller/admin/worker/scheduler roles), per-role OP code allowlists (now including OPCode_schedAdmin), admin login/session model, audit logging for admin operations. Separate from the existing validate_host() hardening (below) which is the minimal gate for OP_upgradeConfig/OPCode_schedAdmin specifically. Discuss + design before implementing; likely promotes to a spec via /kiro-discovery.
validate_host() real authorization — currently a stub returning success for all peers (join.c:13-16, return 1; unconditionally). CONF_ENABLE_UPGRADE_CONFIG is already 1 in the standard build (config_repl.h:27-28, no build-system override found) — the precondition this item was originally framed to guard against has already been crossed; OP_upgradeConfig (and OPCode_schedAdmin, added 2026-08-02, scheduler-core) are live and unauthenticated TODAY, not a future risk to prevent. Minimum fix: verify peer X.509 CN against a configured allowlist (--admin-cn flag or admin_cn_list config), reject non-allowlisted OP_upgradeConfig and OPCode_schedAdmin with clear error, WARN-log rejected attempts with CN+IP. Future: IP allowlist, role-based access, admin-specific CA. (FUTURE_WORK.md §6; security-critical.) Part of the broader peer-authentication-hardening item above.
OP_replConfig access control hardening — currently serves the full config blob to any CA-signed SSL peer (trusted-LAN model). Consider: verify requester is a known --peers controller, rate-limit per peer, minimum_role check (controllers only). (FUTURE_WORK.md §7.) Part of the broader peer-authentication-hardening item above.
lost-and-found drain command — xcecli lost-and-found drain: reassign or cancel sidelined jobs, reset lost_and_found_job_count. Automates operator cleanup after config upgrade. (FUTURE_WORK.md §4.)
~~OP_upgradeConfig standby redirect verification~~ — RESOLVED 2026-08-19, then the underlying gap itself CLOSED 2026-08-26 (bug-1700/1701). As of 2026-08-19, a standby did NOT return the primary's address for OPCode_upgradeConfig — ha_dispatch_gate rejected it before the handler's own dead defensive check ever ran, replying flat status=-XCE_ERR_STANDBY, data_which=nodata with no redirect fields; fill_redirect_reply() (join.c:27) was never called for this opcode, a documented, deliberate feature gap (admin-config-tool design.md's "Revalidation Triggers"). That gap is now closed: cluster.c's generic dispatch-gate rejection path calls fill_redirect_reply() on every -XCE_ERR_STANDBY rejection (bug-1700/1701, gate-level fix, not opcode-specific), and a standby pushing config push now redirects to the primary and succeeds. Coverage: client-side redirect-following remains unit-tested against mocks (test_discovery.py, test_config_push_mock.py::test_push_config_follows_one_redirect_to_primary); the real server-side behavior is live-verified against a real 3-node cluster (src/xceadmin/tests/test_e2e_live_ha_push.py::test_live_config_push_against_standby_redirects_and_succeeds, renamed from ..._returns_standby_rejection_not_redirect to match the now-correct outcome).
steering sync — ha-async-io.md synchronized to the three-state model + own-term gate (transitional-state row/role removed; heartbeat re-arm PRIMARY-only; PRIMARY no-quorum fact allowlist noted). Done as part of ha-single-leader-role task 6.1.
xce_test WAL/snapshot inspection tooling — extend the offline inspector family in xce_test (sibling of the shipped waldump, src/controller/testsuite/waldump.{c,h} + cmd_waldump). Two remaining commands: (a) snapdump <snap_dir|snap_file> — decode the XCESNAP\0 header (8-byte LSN; persist.c:1266 load_snapshot) and report whether the snapshot carries job/queue entities or is an LSN-only stub. This directly settles Open Fork #3 of the failover job-loss investigation (.kiro/specs/ha-commit-durability/INVESTIGATION-failover-job-loss.md): if snapshots are entity-less, any empty-rejoin/WAL-prune below committed is unrecoverable BY DESIGN. (b) walcompare <dirA> <dirB> — diff two nodes' WALs by lsn → crc/jobid to flag divergence/prefix-gaps between members instantly. Reuse the waldump.c reader/decoder; same zero-alloc capnp jobid decode + c_capnproto_dep/zlib_dep wiring already added to xce_test_deps. Nonzero exit on entity-less snapshot / divergence so both double as E2E assertions. (recorded 2026-06-20; waldump already done.)
~~xce_test shell command support~~ — IMPLEMENTED (task 2.1, spec xce-test-e2e-primitives, closed 2026-06-19; stale entry corrected 2026-08-19). preprocess_and_run() (src/controller/testsuite/test.c:950) detects a leading ! and runs the remainder via system(); the shell command's exit status becomes g_last_status (the $? auto-variable) and a non-zero status sets g_session_failed, aborting the script the same way any other failing command does — system()'s child inherits the parent's stdout/stderr fds, so output lands in whatever the invoking xce_test session's own output is already redirected to. Works in every REPL context (no leave_ctx()/disconnect), including mid-session inside worker/client contexts — used this session to fix bug-1579 (resource-value-verification's nested controller-check pattern). No parser/harness-layer change remains to do.
scheduler framework enhancement — PROMOTED to spec 2026-07-06, then DECOMPOSED 2026-07-27 into scheduler-core (IMPLEMENTATION-COMPLETE), scheduler-allocation, and scheduler-preemption — see In Flight above for current status. Scope notes from the original brief still apply: commit barrier wiring deferred (separate hardening item); GPU/affinity allocation deferred (pluggable via new allocator interface); new job selection policies deferred (framework enables them, ships separately). No longer tracked here.
[→] perf-tuning-extraction — PROMOTED to spec 2026-07-20 (clusterconfig-schema-split, .kiro/specs/clusterconfig-schema-split/). Originally recorded 2026-07-06 from admin-config-tool discovery as a documentation/audit item; the audit (2026-07-20) found the fields are load-bearing enough to need real code changes, not just a doc pass — candidates/vcpus/hash-tuning fields (jobmgr/workermgr/qmgr/resmgr/xstrings) are already dead weight in the ClusterConfig capnp blob (superseded by --peers/--vcpus CLI flags or never read into runtime state — see src/docs/cluster-config-split-analysis.md). port is NOT dead — a second review pass (2026-07-20) found cluster_open_listener (cluster.c:1216-1258) falls back to the blob's port field, and single-node mode (no --peers flag at all, main.c:293-294) depends on that fallback entirely; the spec now requires a replacement port-sourcing mechanism before the field can be removed (clusterconfig-schema-split Requirement 1.7), not a bare deletion. heartbeatInterval/inactiveHeartbeatCount (controller→worker liveness, NOT the HA/Raft election heartbeat) were investigated and deliberately kept IN ClusterConfig for operational uniformity despite passing the independence rule — see .kiro/steering/config-design.md. fetchJobInterval/netioTimeout left unresolved, flagged out-of-scope in the new spec for a future pass.
Raft/HA heartbeat interval+timeout as CLI parameters — NEW 2026-07-20 from clusterconfig-schema-split discovery. The controller-to-controller HA/Raft election heartbeat (ha->config.heartbeat_interval_ms/heartbeat_timeout_ms — the one where a cross-node mismatch risks spurious failover) is 100% hardcoded today (HA_DEFAULT_HEARTBEAT_INTERVAL_MS/HA_DEFAULT_HEARTBEAT_TIMEOUT_MS, cluster.c:1131) — there is no ClusterConfig field for it, and per user decision (2026-07-20) there should never be one: these values must be available to a controller BEFORE it has validated/received cluster configuration (a controller needs its own election-timing parameters to even participate in the election that delivers config in the first place — a bootstrap/chicken-egg constraint, not an independence/frequency judgment call). Target: --heartbeat-interval-ms/--heartbeat-timeout-ms CLI flags following the exact --vcpus pattern (OPT_* + struct arguments + myconf_t field, 0 sentinel = use hardcoded default). Kept hardcoded for now; this is a future item, not blocking clusterconfig-schema-split.
resource-distribution-mapping — created 2026-07-23, never added here (found 2026-08-19, itself a small doc gap). Physical resources have no "configured total vs. currently-consumed" concept anywhere in this codebase, live or dead — resources.yaml's resourcemap field (which expresses exactly this: a value shared across N hosts) is deliberately dropped by xceadmin config build (build.py:41-43), and the matching controller.capnp fields are commented out (controller.capnp:208-209). mark_host_res() (worker.c) actively rejects any resource flagged XCE_RESF_SHARED today, so "shared pool" and "per-host value" are mutually exclusive by construction in the live system right now. A working total/current pooling mechanism DOES exist in the scheduler (resource.c's sched_check_shared_satisfy_()), but it exclusively serves job-limit virtual resources (queue MAX_RUNNING-style), never physical shared resources — architecturally reusable, never connected. This spec (research.md done, 3 scoped options, requirements.md never generated — sitting at the decision point) is the existing owner of closing this gap; surfaced again 2026-08-19 when scheduler-core's REPL info resource command couldn't show a total for a GLOBAL/SHARED resource because there genuinely isn't one to show. Next: pick a scope option in research.md, then /kiro-spec-requirements resource-distribution-mapping.
scheduler-sim-diagnostics — NEW 2026-08-10 (/kiro-discovery). Open-ended, continuously-extended spec for scheduler simulation-mode debug/troubleshoot capability — added to over time as new investigation needs surface from discussion or from other in-flight scheduler specs (scheduler-allocation, scheduler-preemption). Extends scheduler-core's REPL (info job/queue/worker/resource/summary, breakpoints, snapshots). Two seed capabilities identified so far: (1) limit inspection — xce_limit_t/xce_vres_t config + live usage currently live only in the limit plugin's private kvmgr tables, invisible to the REPL; needs a new info limit <name|all> command + accessor plumbing. (2) decision-reasoning trace — xce_pendreason_t is captured on pend decisions but never printed by the REPL, and dispatch decisions have no reasoning trail at all (no record of which limit/resource checks passed) — needs new instrumentation in the allocation path (resource.c's sched_try_alloc_on_host_, plugin.c's limit_prepare_allocation) plus a REPL command to surface it. Explicitly simulation-mode only (no live/production controller-connected diagnostics) and read-only (no state mutation) per user decision 2026-08-10. Brief: .kiro/specs/scheduler-sim-diagnostics/brief.md. Not started — requirements to be added per-capability, not fully speced upfront. Next: /kiro-spec-init scheduler-sim-diagnostics when ready to begin.
xcecd_virtual concurrent-job support (MXJ > 1) — NEW 2026-08-19, recorded during bug-1589 investigation (.wolf/buglog.json). Live testing found that when a job array (multiple elements) is dispatched to a single xcecd_virtual worker in quick succession, only the first element's start ever gets confirmed back to the controller (on_job_start) — the rest are dispatched but never confirmed. Root-caused (or being root-caused — see bug-1589 for the confirmed mechanism once the in-flight investigation lands) to xcecd_virtual's job-simulation path: unlike the real xcecd binary, which launches an actual dbus-managed worker process per job (tracked by real pid, real IPC status reports), xcecd_virtual (src/worker/src/xcecd.c's xcecd_process_command(), VIRTUAL_CLUSTER branch) never launches a real process at all — it assigns a bare incrementing fake pid (static int virtual_pid, xcecd.c:27) and simulates job execution via a sleep-timer mechanism (xcecd_worker_utils.h's VIRTUAL_CLUSTER-only xcecd_worker_t fields start_time/sleep_time/killed_signal, check_worker_sleep_time_elapsed()). The immediate bug-1589 fix (once landed) addresses the specific reported symptom; this item is the broader follow-up: properly support N concurrent simulated jobs per xcecd_virtual worker, since MXJ (max jobs per host) > 1 is a legitimate, common configuration to want to exercise in virtual-cluster-mode testing/CI (this project's own dev workflow relies on xcecd_virtual for HA soak tests, resource-value verification, and now scheduler-core's live validation — several .wolf/buglog.json entries this session alone used it), not just a single-job-at-a-time edge case. Scope sketch (to refine once bug-1589's root cause is fully confirmed): audit every VIRTUAL_CLUSTER-only code path in src/worker/ (xcecd.c, main.c, xcecd_worker_utils.h/.c, process_worker_manager.c) for single-job assumptions (a lone poll/early-return that only processes one pending simulated job per tick, a shared mutable scratch struct reused across concurrent jobs, etc.) and generalize each to iterate the full set of in-flight simulated jobs, matching the pattern the real-process path (process_manager.c's process_info_t manager->processes[], a proper per-pid array) already establishes correctly. Dependencies: bug-1589 (must land first — its root-cause trace is the concrete map of what needs generalizing). Promote via /kiro-discovery or /kiro-spec-init once bug-1589 is closed and the exact mechanism is confirmed.

Superseded / Absorbed Items

~~Quorum-ACK gate before config restart~~ (FUTURE_WORK.md §2) — largely superseded by ha-consistency-invariants Requirement 5: configuration changes now require majority acknowledgment before being confirmed/committed, and fail to the requester on timeout (no silent rollback when standbys are offline). Residual to verify in that spec's design: restart sequencing after a committed config change (don't restart the primary before the commit barrier passes).
~~Lost-and-found keyword validation~~ (FUTURE_WORK.md §8) — folded into the admin-config-tool spec (brief: .kiro/specs/admin-config-tool/brief.md).
~~admin-config-tool backlog item~~ (FUTURE_WORK.md §1 + §8, 2026-05-12) — absorbed into admin-config-tool spec 2026-07-06 (/kiro-discovery). The one-paragraph backlog sketch is superseded by the full brief with architecture decisions, decomposition, and boundary contracts at .kiro/specs/admin-config-tool/brief.md.

Notes

Backlog items here are seeds for /kiro-discovery / /kiro-spec-init; promote an item by creating its spec under .kiro/specs/.
Provenance: items marked (FUTURE_WORK.md §N) originate from the 2026-05-12 config-replication-design.md review, merged here 2026-06-10.
2026-10-11 unified resource model discovery (data/volumes as resources, custom resources, one interface; context: the LLM training + serving platform direction in the Overview). Key decisions:
- One resource type descriptor and one provider contract over three layers: L1 attribute, L2 counter, L3 instance/claim. Not one mechanism for everything.
- The scheduler only selects from its cached snapshot; providers run after dispatch.
- Batch stays LSF-style.
- Claims come in two forms, template and standalone (Kubernetes PVC/ResourceClaim-like), as WAL entities with the resource-provisioner discipline.
- Adopt Kubernetes DRA/CSI concepts, not (yet) their APIs.
Phases: resource-distribution-mapping (shared pools) → custom-resource-reporting → resource-instances → resource-claims → gang-groups. Nine user decisions are pending, listed in resource-model.md.

2026-07-06 admin-config-tool discovery (/kiro-discovery, Path E mixed decomposition):

#	Workstream	Type	Language	Dependencies
1	admin-config-tool	NEW spec	Python	`controller.capnp` only
2	config-upgrade-snapshot	EXISTING spec extension	C	entity-bearing-snapshots (blocked upstream)
3	perf-tuning-extraction	DIRECT work	doc	None

Key architecture decisions: Python (future open-source, shared capnp infra), pure Python wire (no CGO/libxce), direct TCP+TLS to controller (not apiserver), single controller.capnp dependency, xceadmin separate from xcecli (user=REST, admin=TLS), auto primary detection via peer list + redirect. Full details: .kiro/specs/admin-config-tool/brief.md.

2026-07-06 scheduler-framework discovery (/kiro-discovery, Path C single-scope):

Key architecture decisions: Iterator-driven allocation (lazy plan iterator + host iterator composition), configurable job picker (plugin name from config, not hardcoded), combinatorial parallel job allocation (1host×N → 2hosts×⌈N/2⌉ → … → Nhhosts×1 sorted by preference), session-level unsatisfiable resreq fingerprint cache. Scope narrowed from original backlog seed: commit barrier wiring deferred, new job selection policies deferred (framework enables them), GPU/affinity allocation deferred (pluggable via allocator interface). Alternatives documented in brief.md (Approach A: incremental/rejected, Approach C: full framework/deferred). Full details: .kiro/specs/scheduler-framework/brief.md.

88 KiB Raw Blame History Unescape Escape