docs(scheduler-sim-repl-verbosity): add approved spec (requirements/design/tasks)

Full spec scaffold (requirements/design/research/tasks all approved,
ready_for_implementation=true) plus a roadmap backlog item recorded from
the dirty-content-fingerprint spec's task 4.3 findings (Makefile/meson
fingerprint-computation duplication risk).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

2026-08-22 02:45:52 +00:00

61 KiB

Raw Blame History

Roadmap

Overview

XCE is in the persistence + HA transformation (branch persistent_ha). Governing model: Raft consensus + CP (see product.md key decisions). Active spec work is tracked per-spec in .kiro/specs/; this file is the single consolidated future-work document — the forward-looking queue, long-deferred items, and hardening backlog. It absorbs the former src/docs/FUTURE_WORK.md (2026-05-12, config-replication-design review), which now redirects here.

In Flight

scheduler-framework — NEW 2026-07-06 (/kiro-discovery). Iterator-driven allocation architecture: lazy plan iterator + host iterator, combinatorial parallel job allocation, configurable job picker, session-level unsatisfiable resreq cache. Brief: .kiro/specs/scheduler-framework/brief.md. DECOMPOSED 2026-07-27 into three child specs (.kiro/specs/scheduler-framework/spec.json child_specs): scheduler-core — core/driver separation, pluggable job picker interface, and simulation mode; IMPLEMENTATION-COMPLETE, GO/NO-GO validation 2026-08-07/08 (2 non-blocking follow-ups: (a) tlog logging-integration built but never activated, (b) fcfs.c/builtin_fcfs.c doc+traceability gap — design.md text was corrected same-day, but tasks 2.1-2.6 in tasks.md still carry no _Verified: lines, still open), 45/45 sub-tasks done as of that validation. Updated 2026-08-19: task 12.5 (REPL live-operator-testing fixes, Req 5.22 amendment) was added after that validation from live-operator testing gaps (bug-1593/1594/1595/1596) and is now also done, independently re-verified with fresh evidence (/kiro-verify-completion) — see tasks.md's own _Verified: line on that task; this closes follow-up (a) too (the tlog gap is now activated — see design.md's SchedulerCore component and research.md). A separate retrospective design review (2026-08-19, unrelated to the original 2026-08-07 follow-ups) additionally found and fixed an OPCode_schedAdmin blocking-wait admission gap and extended core.c's isolation-boundary test coverage to its core_events.c/core_session.c/core_clock.c siblings — see research.md's "SUPERSEDES" notes. scheduler-allocation — lazy iterator-driven allocation architecture, parallel job support, session-level unsatisfiable resreq cache; requirements generated, blocked_on: [scheduler-core] — now unblocked. scheduler-preemption — SIGSTP-based victim selection + controller→worker pipeline; requirements generated, blocked_on: [scheduler-core] — now unblocked. The parent scheduler-framework spec is retained only for discovery-context provenance and is no longer in flight itself. Next: /kiro-spec-design scheduler-allocation and /kiro-spec-design scheduler-preemption.
ha-consistency-invariants — UMBRELLA CONTRACT (requirements + design + research approved/complete). Implementation decomposed 2026-06-11 into the four sub-specs below; the umbrella generates NO tasks itself. Each sub-spec brief references the umbrella docs as authoritative. COMPLETE 2026-07-13 — all four sub-specs (ha-commit-engine, ha-commit-barrier-rollout, ha-single-leader-role, ha-convergence-recovery) implemented; Criterion 2.9 (Facts-Apply invariant) satisfied via raft-apply-decoupling. No longer in flight.
~~ha-wal-replication — implemented except task 5.4 (live 3-node integration test; needs /xce-deploy-ha cluster); sub-spec 4's E2E extends it~~ task 5.4 CLOSED 2026-07-18 via /kiro-validate-impl's live 10-round sequential soak (972/972 tests, zero loss). Spec is GO, no longer in flight.
libxce-failover — CLIENT-SIDE transparent controller failover (requirements generated 2026-06-16). Wires all libxce production request paths through the (already-built-but-orphaned) xce_request_with_retry() primitive + adds multi-master rotation; classifies on the real controller status taxonomy E_XCE_REDIRECT / E_XCE_NOMASTER / E_XCE_NOT_READY / E_XCE_STANDBY (errnox.h:22-25); bounded retry → distinct cluster-not-ready (no more opaque EINVAL); includes one small controller change so redirect replies carry the primary host/port (schema fields already exist, controller.capnp:598-599, just unpopulated in join.c). Verified E2E via xce_test (the original reproduction). Closes the integration gap left by ha-config-bootstrap Req 7 (tasks 10/10.1/10.2 shipped only the standalone primitive). See bug-608. Dependencies: ha-config-bootstrap (primitive + error codes), ha-commit-barrier-rollout (write idempotency makes write-retry safe). IMPLEMENTED, validate-impl GO 2026-07-18 (all 19 tasks were done since 2026-06-18; the ready_for_implementation flag was just never flipped — administrative staleness, not a real block). No longer in flight. component-failover-adoption (backlog, below) is the next layer on top of this.

Raft TLA+ Conformance — raft-library enhancement series (NEW 2026-06-21)

Goal: evolve the election-only src/lib/raft/raft.{c,h} into a STRICT implementation of raft/raft.tla (Ongaro canonical) covering the log + commit half, keeping the pure callback-driven design so it stays independently verifiable via the deterministic simulator (raft_sim.c) against the TLA+ invariants (Election Safety, Log Matching, Leader Completeness, State Machine Safety). Motivation: bug-851 — the log protocol is currently hand-rolled across persist.c/repl.c/ha.c/ha_convergence.c and violates the TLA+ log invariants. Gap analysis (six gaps G1–G6): .kiro/specs/ha-term-safety/TLA-gap-analysis.md. Implement in dependency order; each spec lands with simulator coverage. The library core stays pure/synchronous/deterministic — all I/O (async vtask, fan-out replication, the sync commit barrier, snapshot transport) lives in controller-side callback adapters. Preserve as policy (do NOT canonicalize away): PreVote, priority leadership, synchronous commit-barrier + persist-before-ACK durability, "controller never self-terminates".

raft-log-replication — (specs generated + auto-approved 2026-06-21: 8 reqs, 24 sub-tasks, cross-spec review PASSED. Ready for /kiro-impl.) Extend the library to OWN the replicated log. TLA+ actions: ClientRequest, AppendEntries, HandleAppendEntriesRequest, UpdateTerm-on-AppendEntries. Per-entry term in the library log model (closes G1); AppendEntries logOk = prevLogTerm match (G2); conflict branch deletes the divergent (term,lsn) suffix (G3); ANY higher-term RPC incl. AppendEntries/heartbeat advances currentTerm and steps down (G5). New pure callbacks: on_persist_entry, on_get_entry_term(lsn), on_send_append_entries(peer, prevLogIdx, prevLogTerm, entries, commitIdx), on_truncate_from(lsn); controller adapters implement them over the existing WAL/repl transport. Simulator: drive AppendEntries across partitions/restarts, assert Log Matching. Dependencies: raft-library (election core). Closes bug-851 G1/G2/G3/G5.
raft-commit-safety — (specs generated + auto-approved 2026-06-21: 6 reqs, 18 sub-tasks, cross-spec review PASSED.) AdvanceCommitIndex with the Figure-8 current-term rule (commit Max(agreeIndexes) only if log[Max].term == currentTerm) (G4); RequestVote/PreVote freshness becomes (lastLogTerm, lastLogIndex) instead of LSN-only (G6). New callback on_commit_advanced(lsn) feeds the existing synchronous commit barrier / durability (persist-before-ACK preserved as policy). Simulator: assert Leader Completeness. Dependencies: raft-log-replication. Closes bug-851 G4/G6.
raft-sim-conformance — (specs generated + auto-approved 2026-06-21: 8 reqs, 23 sub-tasks, cross-spec review PASSED.) Extend raft_sim.c into a conformance/verification harness driving the library through raft.tla-action-equivalent transitions (randomized + targeted) and asserting the four Raft safety invariants; optionally trace-check against TLC output of raft.tla. The "use the verified spec to help" payoff: a regression oracle for the consensus core. Dependencies: raft-log-replication, raft-commit-safety. IMPLEMENTED, validate-impl GO 2026-07-18 (23/23 tasks; 293/293 lib tests, zero ASAN findings; this is the spec this worktree/branch — raft_tla — is named for).

ha-term-safety (.kiro/specs/ha-term-safety/) is the XCE-side INTEGRATION + REGRESSION umbrella for this family: it wires the controller WAL / commit-barrier into the new raft callbacks (thin adapter — protocol delegated to the library), owns the config-change marker (R7 — config changes as (term,lsn) log markers; bytes via the existing out-of-band config-push transport, hash-verified against the marker), and holds the live bug-851 chaotic-failover regression. Its protocol invariants R1–R6 are realized by the library specs above (not re-implemented in the controller). Scope: R1–R7 only. The former R8 (config-upgrade full-snapshot consistency, bug-857) was descoped 2026-06-21 to the independent config-upgrade-snapshot spec (below), because it requires entity-bearing snapshots that do not exist yet. IMPLEMENTED, validate-impl GO 2026-07-17 (20/20 tasks; closed bug-1445, a first-boot promotion-check race with job-array-single-wal-record's Live Apply Resource Fence, as an out-of-boundary follow-up fix).

raft-adapter-conformance (.kiro/specs/raft-adapter-conformance/) — NEW 2026-07-29, sibling audit to the library-side conformance work above: a systematic, action-by-action pass over the controller's Raft adapter (the wire-level send/receive/reply code in ha.c/repl.c/config_repl.c) against raft.tla's actions — distinct from raft-sim-conformance's in-process library harness. Motivated by a live-soak finding that the controller's WalAckReply constructor hardcoded success (no failing reply ever reached the primary) plus a standby catch-up streaming-range self-deadlock, both already fixed (bug-1479) before this spec was seeded. This pass found and fixed two more instances of the same defect class: configuration-push rejects previously sent no reply at all (bug-1480), and a vote/PreVote endpoint failure arriving after connect routed through a legacy tally path instead of the production one (extending bug-1423's connect-time fix to that remaining timing window). Also records, as a single explicit decision, which AppendEntries mechanism is production-authoritative (repl_broadcast + ha_stream_wal_to_standby; the per-peer index-driven raft_send_append_entries stays dormant by documented design, not oversight). Requirements/design/tasks generated and approved, all 15 leaf tasks (1–7.3) checked; src/controller/CLAUDE.md's Protocol Verification Checklist marks the corresponding invariants "post raft-adapter-conformance, landed 2026-07-29" — spec.json phase (tasks-generated) has not been flipped to reflect this, same administrative-staleness pattern seen elsewhere in this doc (cf. the former libxce-failover entry above). Dependencies: raft-log-replication, raft-commit-safety, ha-term-safety, raft-sim-conformance (adjacent/informational — this spec fixes the adapter layer those validate underneath).

Raft Apply-Ordering & Hook Decoupling — NEW 2026-06-24 (`/kiro-discovery`); IMPLEMENTED — see `raft-apply-decoupling` below

Goal (historical — achieved): make the controller follow canonical Raft ordering — pre-hooks → raft commit → apply → notify — with the raft protocol step lifted OUT of the notification hook fan-out. At the time this family was opened, persist_job_hook_fun (an AFTER_EVENT hook) performed the WAL append + synchronous majority-commit barrier inline, and create_job() mutated in-memory state BEFORE that commit — the structural root of the bug-795/828/851/1060 divergence family. A failed/timed-out commit did not cleanly revert the optimistic mutation. Decision (user, 2026-06-24): apply-after-commit (canonical) — a single shared apply_committed_entry() seam mutates the state machine only after commit, reused by restart replay so online-apply and replay-apply are identical. Hooks demote to notification-only. Decomposed via Path E (2 new specs + touchpoints), delivered as raft-apply-decoupling (IMPLEMENTED 2026-07-07, see below). Current code reflects the fix: persist_job_hook_fun (persist.c ~6509-6553) now explicitly returns early for WAL_MODE_WAIT entries, with a comment noting job-submit durability is invoked directly by handle_jobsubmit — persist_submit_job_payloads_and_commit(), called strictly before any job object is constructed — never from this hook; create_job() is now defined at job.c:48.

Async-drive prerequisites (surfaced 2026-06-25 by blocking audit; thread-role contract now in `ha-async-io.md`)

The protocol drive is NOT fully event-driven. Root cause: peer_ctrl_msg_handler runs inline on iomgr instead of dispatching to the vtask pool, and delaymgr (the heartbeat driver) was the documented blocking sink. Corrected thread-role contract is now steering (ha-async-io.md, 2026-06-25; SUPERSEDES "delaymgr for blocking ops = Permitted"). Two separable prerequisites (full evidence: .kiro/specs/raft-apply-decoupling/facts-in-raft.md § Keystone + "Two distinct prerequisites" + P2 offload mechanism):

P1 — commit-barrier async (the commit drive). Convert repl_wait_majority_ack() from blocking pthread_cond_timedwait (commit.c:359) to event-driven resolve(lsn,outcome) on the existing commit engine. Dissolves apply-self-deadlock (B), enables reply continuations (C), removes vtask-pool exhaustion. Owned within raft-apply-decoupling as its keystone (sequenced before its Phase A).
P2 — drive hardening (the transport/timer drive). Now its own spec raft-drive-async (brief: .kiro/specs/raft-drive-async/brief.md) — promoted because it threatens liveness independently. See the Specs list below.

Specs (dependency order — Raft Drive & Apply Async family)

raft-drive-async — IMPLEMENTED, validate-impl GO 2026-07-18 (21/21 tasks; the live sequential soak this validation ran surfaced a real crash, bug-1447 — a same-batch endpoint use-after-free in io_worker's epoll dispatch loop, exposed by this spec's own faster reconnect cadence, see ha-async-io.md; root-caused, fixed, re-soaked clean, re-validated GO same day). Original description retained below for provenance. (was: START FIRST — threatens failover liveness TODAY) — bring the code into compliance with the corrected ha-async-io.md thread-role contract: dispatch peer_ctrl_msg_handler to the vtask pool (root-cause fix); snapshot send + config-push off iomgr → admission-capped slow lane (per-peer dedup + WAL coordination); delaymgr peer connect → bounded non-blocking + offloaded (reuse xce_probe_config_tag pattern); add a general non-IO vtask-submit path + admission reservation (extend COMMIT_WAIT_RESERVED). Verify with a partition (unreachable-not-killed) soak: heartbeats to survivors continue, no spurious election. Likely root of the bug-851 chaotic-failover family. Dependencies: none hard (builds on existing commit engine + vtask + the steering contract); election-iomgr-refactor is completed upstream (provides delay_task_ms/on_connected/non-blocking vote fan-out) — STANDALONE per 2026-06-25 scope-check (ADJACENT-SAME-AREA: it fixed startup blocking, this fixes runtime drive blocking). Brief + requirements + research + design + tasks done; phase=tasks-generated (tasks approved: false). /kiro-validate-design GO 2026-06-25 — three review findings folded into design+tasks: (1) deadlock-freedom + LivenessAdmission — the documented invariant reversal (ack/commit_on_ack moves off iomgr to the vtask pool, commit.c:16) co-locates commit-barrier wakers+waiters; review found COMMIT_WAIT_RESERVED is insufficient because it counts only commit-waits while peer_ctrl also puts vote-grant fsync + config-ack waits on the pool (invisible to it). Fix: generalize the reservation to a pool-level task-class admission in vtaskmgr (VTASK_LIVENESS/LIVE_BLOCK/REQ_BLOCK; liveness never gated; COMMIT_WAIT_RESERVED folded in) — new Phase-1 task 1.6 (dormant infra), hard prerequisite of 3.1; task 3.4 is the admission gate. Transient until raft-apply-decoupling P1 makes the barrier async; (2) WAL-trim cursor-bound guard added to task 3.2 (bug-1147, see Hardening below — moving checkpoint/trim onto a concurrent lane worsens the existing latent trim-vs-cursor race); (3) in-process partition primitive in xce_test as the portable primary for R7 (firewall DROP secondary) so the soak actually runs unprivileged. Next: approve tasks (/kiro-spec-tasks raft-drive-async) → /kiro-impl; ship Phase 1 (bounded connect) first.
raft-apply-decoupling — decouple the raft protocol step from the hook fan-out (item #1); canonical pre-hooks → commit → apply → notify ordering (item #2); single apply_committed_entry() seam shared by online commit + restart replay; post-restart validation that committed log was correctly applied to materialized state (item #4); requirement + regression tests (item #5). Contains P1 (commit-barrier async) as its keystone. Commands (WAL_MODE_WAIT) apply-after-commit; facts (WAL_MODE_RETAIN) routed through commit-then-apply too — D1 RESOLVED (block-but-durable under quorum loss); formalized as ha-consistency-invariants criterion 2.9. Dependencies: raft-drive-async, raft-log-replication + raft-commit-safety (DONE/implemented), worker-rejoin-resync; coordinate with ha-term-safety. Briefs: .kiro/specs/raft-apply-decoupling/brief.md + facts-in-raft.md. IMPLEMENTED 2026-07-07: 32/32 tasks, validate-impl GO — 79/79 controller_unit_test groups (926 tests) green, live 3-node smoke boot verified, zero blocked tasks. Task 7.2 fixed bug-1416 (boot-time TOCTOU false APPLY_DIVERGENT fence). Task 7.4's own throughput benchmark found and fixed a real O(n²) apply-pump regression (56% throughput drop, superlinear scaling) via a segment-read cache on persist_t, 2 adversarial review rounds. Residual, non-blocking: design.md has 3 sections not yet reconciled with implementation deviations already self-documented in tasks.md's Implementation Notes (repl.c's planned repl_submit_majority_commit() was never built; apply pump wired via commit_resolve_pending() not ha.c's on_commit_advanced; fact hooks collapse to generic HOOK_DETAIL_JOB_UPDATE not per-type); task 5.3 touched ha.c without a formal _Boundary:_ exception tag (substantively justified, process-inconsistent only). Two out-of-scope findings flagged for follow-up: an intermittent client hang in the synchronous jobsub path (resembles but not confirmed as bug-1416's class), and trim_wal_segments's retention floor has no awareness of the local apply pump's own lag (same family as an already-tracked standby-cursor gap).
worker-rejoin-resync — PROMOTED to spec 2026-08-10 (/kiro-discovery; brief at .kiro/specs/worker-rejoin-resync/brief.md). Elevated to HARD prereq for the facts half of raft-apply-decoupling 2026-06-25 (was backlog "designated backstop"). Worker re-reports running/finished job state on rejoin so facts lost in a failover window are recovered; re-reports flow through commit-then-apply idempotently (criterion 2.9). Today workers send only hostattrs on join (bug-396). Scoped narrowly to worker job-execution-fact resync only — apiserver/scheduler resync-on-reconnect is a different data shape (cluster/queue/job list state, not job execution facts) and stays out of scope (see the deferred "lightweight rejoin re-attach" backlog item below, still YAGNI). Dependencies: ha-consistency-invariants (commit barrier + idempotency guards) — both implemented, unblocked. Next: /kiro-spec-requirements worker-rejoin-resync.
job-array-single-wal-record — PROMOTED from backlog 2026-06-24. Persist/replicate/push a job-array submission as ONE Raft record (base spec + index range/count + per-index overrides) committed atomically through a single barrier; expand on apply (online + replay) and on the scheduler side; preserve per-element identity/addressing; idempotent re-apply (item #3). Dependencies: raft-apply-decoupling (HARD — builds on the shared apply seam), ha-commit-engine, ha-commit-barrier-rollout, ha-consistency-invariants. Brief: .kiro/specs/job-array-single-wal-record/brief.md.

Existing Spec Updates (Raft Apply-Ordering phase)

raft-sim-conformance — add apply-after-commit regression assertions (item #5): no apply on a failed/uncommitted entry; online-apply ≡ replay-apply for the same committed log; ordering invariant pre-hooks → commit → apply → notify. Revisit with /kiro-spec-requirements raft-sim-conformance once raft-apply-decoupling requirements land.
ha-term-safety — coordinate: raft-apply-decoupling restructures the same controller submit→commit path this umbrella wires. Sequence the apply-decoupling edits against ha-term-safety's adapter work (disjoint-but-adjacent regions) to avoid conflicting refactors.

Direct Implementation Candidates (Raft Apply-Ordering phase)

xce_test waldump/walcompare/snapdump inspectors (already a Hardening item below) double as the offline evidence for the item #4 restart-validation requirement — wire their nonzero-exit-on-divergence into the decoupling spec's E2E rather than as a separate spec.

Specs (dependency order)

[x] = spec generated (requirements/design/tasks approved, ready for implementation). Implementation tracked via each spec's tasks.md; implement in roadmap order (cross-spec review 2026-06-11: four specs touch ha_dispatch_gate / raft_cb_become_follower in disjoint-but-adjacent regions).

ha-commit-engine — commit.c per-LSN majority engine + quorum accounting fixes, wired behind job submit. Dependencies: none. (specs done 2026-06-11; IMPLEMENTED 2026-06-12: 12/12 tasks, validate-impl GO, 467 tests green; deferred follow-ups in its tasks.md §Implementation Notes)
ha-commit-barrier-rollout — classification table, commit-first-then-reply sweep, RETAIN facts, config barrier, idempotency guards. Dependencies: ha-commit-engine. (specs done 2026-06-11; IMPLEMENTED 2026-06-13: 16/16 tasks, validate gate GO, 508 controller tests green under ASAN. Fixed bug-447 (key_find found-rc leak — queue/host control was silently unpersisted+unreplicated in production) and bug-396 (worker fact loss on quorum-loss gate rejection); task 5.2 needed one remediation round (wired the missing production config-ACK ingestion — would have bricked multi-member config upgrades — + kept the barrier off the iomgr thread). Logged not-fixed: bug-448 (job_ctrl ref leak), bug-451 (RUSAGE va_list OOB read). Barrier-before-state-removal precondition for ha-single-leader-role satisfied)
ha-single-leader-role — promotion no-op + own-term gate, transitional-state (the former fourth role) removal. Dependencies: ha-commit-engine, ha-commit-barrier-rollout (HARD: barrier before state removal). (specs done 2026-06-11: 2 reqs / 7 components / 11 sub-tasks)
ha-convergence-recovery — demotion truncate+self-restart, rejoin reconciliation, egress watermark filter, ACK-driven catch-up, quorum logging. Dependencies: ha-commit-engine (E2E additionally needs ha-single-leader-role). (specs done 2026-06-11: 9 reqs / 7 components / 13 sub-tasks)
ha-commit-durability — closes bug-661 (acknowledgment did not imply durability; committed jobs lost across failover). Makes acknowledged ⇒ durable on a majority: the primary counts its own copy toward the commit majority only at its durable LSN (majority_candidate_locked caps self at persist_durable_lsn, not the appended tip); fsync_worker repurposed timer→demand-driven group-commit flusher (one fdatasync over pending appends, advance durable_lsn, wake commit-eval) — absorbs the former "WAL group-commit batching" backlog item (see below); recovery integrity verdict (clean vs corruption-truncated WAL) + durable-backed freshness floor (candidate advertises bare persist_durable_floor, voter floor = max(durable_floor, learned_committed)) + pre-promotion eligibility fence. Establishes the controllers-never-self-terminate invariant: a recovery-truncated/behind member that cannot catch up is a LOUD PERMANENT FENCE (non-promotable, stays alive, keeps retrying — never exit/abort), self-heals on peer return (see ha-async-io.md process-lifecycle invariant + src/controller/CLAUDE.md). Dependencies: ha-commit-engine (consumes commit-position semantics), ha-wal-replication (standby ACK/catch-up), ha-commit-barrier-rollout (reply gate, unchanged). (specs done 2026-06-18; IMPLEMENTED 2026-06-18: 17/17 tasks (1.1–5.3), validate-impl GO, controller suite 0 failures under ASAN; deterministic ack-before-fdatasync E2E RED→GREEN + rolling-SIGKILL soak zero acked-job loss. Verified by tests + Protocol Verification Checklist, NOT by raft.tla (which assumes a durable log). Fixed bug-760 (adjacent test_fact_hook_append raced the async flusher — test made durable-deterministic via persist_flush). raft.tla durability-model extension deferred unless explicitly opted in.)

Future Specs / Backlog (dependency order)

config-upgrade-snapshot — descoped from ha-term-safety R8 on 2026-06-21 (requirements generated; .kiro/specs/config-upgrade-snapshot/). Closes bug-857: a configuration upgrade can leave members with DIVERGENT materialized state because today it does config-push + restart + per-node WAL replay under the new config (config_repl.c do_upgrade_config_with_bytes ~1001-1163), with no snapshot pinning an identical post-upgrade state (replay is config-dependent — lost_and_found re-homes orphaned entities). Approach: an upgrade materializes a full entity-bearing state snapshot at the config-change marker's (term,lsn) boundary, embeds the new config, and installs it on every member via the snapshot_transfer_s transport — no per-node re-derivation; supersedes ha-term-safety R7's interim config-push byte delivery for upgrades. Former HARD dependency entity-bearing-snapshots is CLEARED — it shipped as spec ha-entity-snapshots (IMPLEMENTATION-COMPLETE, validate-impl GO 2026-07-18; see Future Specs/Backlog below), and ha-term-safety (the config-change marker it anchors on) is also implemented. config-upgrade-snapshot's own spec.json now has blocked_on: []. Verify by reproducing bug-857 (upgrade during chaotic multi-kill, diff materialized state). Ready to promote to design.
component-failover-adoption — recorded 2026-06-16; depends on libxce-failover (HARD). Make the C daemons and the Go apiserver actually exercise libxce's transparent failover. The mechanism reaches them for free (worker/scheduler/controller link libxce_dep; apiserver uses it via cgo, src/frontend/apiserver/backend/clib.go), but two per-component integration items remain: (1) each component must hand libxce the full master list instead of a single endpoint (today even the CLI uses only conf->masters[0], client.c:122); (2) bounded retry makes the already-blocking xce_request block longer (seconds of backoff) — for the async C daemons (worker controller_connector.c, scheduler handler.c) this must be verified liveness-safe, i.e. it must not stall a thread that also drives heartbeats (the "never block the iomgr thread" rule). Scope: per-component master-list config + retry policy + liveness-safe blocking + per-component E2E failover for scheduler, worker, apiserver. (xce_test is NOT here — it is the in-spec verification client of libxce-failover.) Promote via /kiro-spec-init component-failover-adoption after libxce-failover lands.
~~worker-rejoin-resync~~ — PROMOTED to spec 2026-08-10, see In Flight (Raft Apply-Ordering family) above for current status and brief. No longer tracked here.
admin-config-tool — PROMOTED from backlog to spec 2026-07-06 (/kiro-discovery; brief at .kiro/specs/admin-config-tool/brief.md). NEW: xceadmin Python CLI for cluster operators — builds and deploys cluster configuration via direct TCP+TLS capnp wire to controller (NOT through apiserver). Two subcommands: (a) config build — directory scan: parse configs/{hosts,queues,resources,users}.yaml, validate against JSON schemas (configs/schema/*.schema.json), serialize ClusterConfig capnp struct to cluster.bin with SHA-256; (b) config push — zero-config dev (default localhost:9991, same as xce_test), single --controller flag for production, auto-discovers peers + primary from controller reply, sends OP_upgradeConfig, displays {versionTag,ackedCount,error}. Single dependency: controller.capnp. Pure Python (pycapnp + yaml + jsonschema + click), open-source ready (pip install). The tool is the client-side trigger for config-upgrade-snapshot's controller-side path. MUST validate against reserved name "lost_and_found" for queue/host (defence-in-depth). xcecli comparison: xcecli = user tool (HTTP/HTTPS → apiserver → CGO → controller, JWT auth); xceadmin = operator tool (direct TCP+TLS → controller, zero-config, same trust model as controller peers). Shared future: xcecommon Python package with capnp schema + wire protocol reusable by both tools. Dependencies: controller.capnp schema only. Blocks: nothing — ships independently. Controller-side upgrade snapshot path (config-upgrade-snapshot) was blocked on entity-bearing-snapshots; that blocker is now cleared (see Future Specs/Backlog above — ha-entity-snapshots implemented, config-upgrade-snapshot blocked_on: []). Next: /kiro-spec-requirements admin-config-tool.
component-rejoin-resync — PROMOTED to spec 2026-08-10 (/kiro-discovery; brief at .kiro/specs/component-rejoin-resync/brief.md) from the "lightweight rejoin re-attach" item below. apiserver + scheduler reconnect/rejoin resync efficiency — today each does a full re-join and receives full state again (whole job list, queue/resource/config state), not a delta; for a large cluster this is sizable but not yet a measured problem. Sibling of worker-rejoin-resync — disjoint data (list/config state vs. job-execution facts), NOT merged into that spec. Not started: first step is investigation (is reconnect cost actually a measured problem at realistic cluster sizes?) before committing to any delta/session-token mechanism — original YAGNI discipline still applies, this spec is the promoted vehicle for that investigation, not a rubber-stamped design. Original decision (libxce-failover design review) retained below for provenance. Dependencies: libxce-failover (rejoin hook — implemented), component-failover-adoption (daemon rejoin content — in progress, sequence alongside/after). Next: /kiro-spec-requirements component-rejoin-resync when ready to begin (not immediate).
~~lightweight rejoin re-attach (failover optimization)~~ — recorded 2026-06-16; deferred (YAGNI); apiserver/scheduler half PROMOTED to component-rejoin-resync 2026-08-10 (see above); worker half already promoted to worker-rejoin-resync. No longer tracked as a bare backlog item — provenance only. Original text: on controller failover a reconnecting daemon does a full re-join and apiserver/scheduler receive full state (e.g. the whole job list), not a delta; for a large cluster this is sizable but rare. Decision (libxce-failover design review): accept full-state rejoin — it is simple, correct, and still cheaper than a full-history replace with no snapshot (cf. IBM LSF). Future optimization: a session-token "re-attach" hello that lets the new primary skip re-pushing already-replicated state, or a delta-only state transfer keyed on the consumer's last-known committed LSN. Delta reconciliation is non-trivial (the primary must know what each consumer already holds). Promote only if failover-rejoin latency/bandwidth is measured as a problem at scale.
job-array-single-wal-record — IMPLEMENTED, validate-impl GO 2026-07-17 (reopened once 2026-07-16 for three user-directed residual gaps — bug-1443 bulk-insert partial-visibility, task 4.2's boot-replay fence gap, bug-1444 replay_wal array-blind tally — all closed, re-validated fresh GO). PROMOTED to a spec 2026-06-24 (/kiro-discovery; brief at .kiro/specs/job-array-single-wal-record/brief.md; now tracked under the Raft Apply-Ordering & Hook Decoupling phase above, sequenced after raft-apply-decoupling). Original note retained for provenance: Enhance controller & scheduler so a job-array submission persists, replicates, and pushes to the scheduler as a single WAL record rather than one record per array element. Today a job array fans out to N per-job state changes, so a 10k-element array means 10k WAL appends + 10k replicated commits + 10k scheduler pushes — amplifying commit-engine load, replication bandwidth, and failover replay time linearly in array size. Scope sketch: (a) controller writes one WAL entry describing the array (base spec + index range/count + per-index overrides) committed atomically through the commit barrier; (b) replication carries the single record so standbys reconstruct the whole array from one committed LSN; (c) scheduler receives the array as one push and expands locally. Must preserve per-element identity/state addressing after expansion (status, cancel, completion still operate per task id). Dependencies: ha-commit-engine + ha-commit-barrier-rollout (single atomic commit), ha-consistency-invariants (replicated scheduling state). Spec advised before implementation; promote via /kiro-spec-init "job-array-single-wal-record" when ready.
bulk job-array query/cancel — recorded 2026-07-13, deferred out of job-array-single-wal-record scope by design. A single client operation to query or cancel an entire array (not one element at a time) is explicitly out of scope for job-array-single-wal-record — see that spec's Boundary Context. Reason: array elements can be in heterogeneous states at request time (done/exited elements need no action; running elements need a worker kill signal; pending elements need a direct state transition to cancelled without ever running), which is a materially different bulk-operation problem from atomic array submission and needs its own requirements (partial-failure semantics, atomicity vs best-effort per element, etc.) rather than being folded into the submission spec. Dependencies: job-array-single-wal-record (per-element identity/addressing this would operate over). Promote via /kiro-discovery once job-array-single-wal-record ships.
~~WAL group-commit batching~~ — recorded 2026-06-17 as deferred; ABSORBED INTO ha-commit-durability 2026-06-18 (design-review reversal). The /kiro-validate-design review surfaced that the highest-volume WAL traffic is WAL_MODE_RETAIN facts (job start/finish/fail/rusage) — not array-collapsible and not covered by job-array-single-wal-record — so a pure per-entry synchronous design would penalize the dominant write path. Decision (with user): do group-commit now — repurpose fsync_worker from a timer into a demand-driven flusher (batch one fdatasync across pending appends, advance durable_lsn, wake commit-eval) as part of ha-commit-durability rather than deferring. No longer a separate backlog item.
primary step-down-on-quorum-loss — optional CP follow-on noted in ha-consistency-invariants brief (writes already blocked on quorum loss; step-down is an availability/clarity optimization).
read consistency / leader leases — Raft §8 read-only optimization; status queries may currently observe stale or provisional state (documented deviation in ha-consistency-invariants).
entity-bearing snapshots ("Task 11" format) — PROMOTED to spec ha-entity-snapshots (created 2026-07-01); IMPLEMENTED, validate-impl GO 2026-07-18 (21/21 tasks 1.1-8.1; 962 controller tests green including all 9 test_persist_snapshot_entities.c cases; live --force-snapshot soak: committed_divergence=0, committed_job_loss=0, confirmed real "HA: snapshot" install log line). write_snapshot() (persist.c ~2304-2372) now collects and serializes job/queue/host entity frames (collect_job_frames/collect_queue_frames/collect_host_frames) ahead of the header — no longer a bare 16-byte LSN sentinel. Original backlog note retained for provenance: real entity serialization unblocks snapshot-based state transfer and faster recovery. Cheaper than it looks (verified 2026-06-11): the dual-zarray COW machinery it needs was already built and unit-tested but dormant — xzarray_freeze/xzarray_thaw in the vtable (zarray.c:695-757, O(1) freeze, thaw with should_retain GC filter, test_zarray_cow.c) with zero production callers (kvmgr.c:297 vfreeze = NULL /* wired in P0-2 */). Note: single-generation COW — the snapshot consumer owns freeze/thaw; do not add second concurrent consumers. Slot-level only (snapshots array structure, not object contents — serialize item state under per-object locks during the freeze window). Also strengthens ha-consistency-invariants §C eviction option (iii) and faster rebuild-on-demotion. Downstream: config-upgrade-snapshot's former HARD dependency on this is now cleared (blocked_on: []) — see that item above. Also strategy B for catch-up of a far-behind / empty-rejoin standby (cursor=0) is now live: the standby-side snapshot-install/apply path is wired (ha.c ~3112-3147 wire-receive path task 6.1; persist.c ~4277-4285 local boot recovery task 8.1), with a decision path that requests a snapshot transfer when a standby's cursor falls below the leader's oldest retained WAL LSN (ha.c:4413). The bug-1147 WAL-trim cursor-bound guard (see Hardening below) is implemented in production, no longer just a stopgap.

Hardening / Direct-Implementation Items

Smaller items that likely don't warrant a full spec; pick up opportunistically or bundle into adjacent spec work.

eliminate src/frontend/Makefile, consolidate Go builds onto meson — recorded 2026-08-21, from dirty-content-fingerprint spec's task 4.3. The Makefile's build-cli/build-server targets exist as a "no meson involved" fast dev-loop path for apiserver/xcecli, but they achieve this by INDEPENDENTLY REIMPLEMENTING the same commit_id/build_time/dirty_fingerprint computation src/tools/xce-build.sh already owns (two separate computation sites for the identical values, a pattern build-time-versioning's own research.md already flagged as an accepted-but-real drift risk). That risk materialized concretely: task 4.3's Makefile-side fingerprint computation needed three adversarial review rounds to fix a shell multi-stage-pipe exit-status bug ({ cmd1; cmd2; } >file's exit status only reflects the last command, silently swallowing an earlier failure and fabricating output from incomplete data) — a bug class that could not have existed if there were only one computation site. Since XCE's nix devShell already bundles the full C/Go/Python toolchain uniformly (no dependency-avoidance benefit to a separate meson-free path the way there might be in a project where Go contributors don't want the C toolchain installed), the Makefile's marginal value may just be "skip the initial meson setup step" — meson compile -C <builddir> apiserver/xcecli should already give an equivalent single-target incremental rebuild once a build directory exists. Not started — investigation first: measure actual dev-loop friction of meson compile vs. make build-cli/build-server (first-configure cost vs. per-rebuild cost) before committing to removing the Makefile; if removed, build-deployment.md and any CI/doc references to make build-cli/build-server need updating too. Promote via /kiro-discovery or direct investigation when ready.
job_ctrl direct-mutate path — migrate to commit-then-apply — recorded 2026-07-13, investigated 2026-08-19 (finding: real risk, wider scope than originally framed). raft-apply-decoupling deliberately scoped job_ctrl's mark_job_update/mark_job_fail (admin SIGCONT/SIGSTOP) OUT-OF-BOUNDARY (task 4.1/6.1 Implementation Notes). The 2026-08-19 investigation found the original framing wrong in two ways: (1) NOT the only remaining call site — on_job_stop/on_job_resume/on_job_ckpt (job.c:2067-2222) share the identical unconverted mutate-then-notify-then-WAL-append pattern, fed by ordinary worker/scheduler traffic, not "rare, admin-only" as originally described; (2) blast radius (a) is confirmed real, not theoretical — the pre-commit mutation is synchronously pushed live to BOTH the scheduler (sched_job_update_fun) AND every connected apiserver/client (apiserver_job_update_fun) before the commit barrier resolves; on barrier timeout the admin gets a failure reply but job->job.state is never rolled back and no retraction is ever sent — matching the bug-795/828/851/1060 shape. Live-apply vs replay-apply (b) is narrower than feared for job_ctrl specifically (the allocs != NULL gate mutually excludes it from the converted deferred-apply fact types), but the investigation surfaced a SEPARATE bug underpinning that safety argument: job_ctrl (job.c:999) holds only a read lock while mutating job state — every other mutator in this file takes a write lock — so two concurrent admin commands on the same job aren't mutually excluded, undermining the "same lock, no staleness window" claim the replay-safety argument depends on (already flagged by a raft-apply-decoupling task 2.1 reviewer, never fixed). Recommendation: promote to a small follow-up spec covering all four call sites together (bounded, well-precedented fix — same shape as the fact-receipt conversion raft-apply-decoupling already built for job submit); the lock-type bug is independently fixable and low-risk regardless of when/whether the larger migration happens. Not yet promoted — user decision 2026-08-19 to correct scope/framing here first, promote later.
WAL-trim cursor-bound guard (bug-1147; surfaced 2026-06-25 by raft-drive-async design review) — IMPLEMENTED as part of raft-drive-async task 3.2 (raft-drive-async is IMPLEMENTED, validate-impl GO 2026-07-18; see above). trim_wal_segments (persist.c ~2597-2644) now computes floor = repl_trim_lsn(&p->repl, active_seg_lsn, p->config.snap_interval_s) — folding in active-standby acked_lsn and disconnected-slot last_disconnected_lsn — and only unlinks segments below that floor; the code comment at persist.c:2634 explicitly labels this the "(bug-1147 guard)". Far-behind/empty join (cursor=0) is additionally covered now that entity-bearing snapshots (strategy B) has landed as ha-entity-snapshots (see above), with a live decision path (ha.c:4413, "< oldest_wal_lsn → requesting snapshot transfer"). Related: bug-795, bug-828, bug-1060.
OP_heartbeat peers cross-validation — include a hash of the local --peers string in heartbeat requests; log ERROR on mismatch between candidates. Detects misconfigured nodes at election time instead of silently risking dual-primary split-brain. (FUTURE_WORK.md §3.)
peer-authentication-hardening — NEW 2026-07-06 from admin-config-tool discovery (/kiro-discovery). XCE currently operates under a trusted-LAN model — any CA-signed SSL peer can connect to the controller and perform ANY operation. Three gaps identified: (a) Admin→Controller: validate_host() is a stub returning success for all peers (roadmap § below). OP_upgradeConfig has no real authorization gate today. Same gap confirmed 2026-08-02 for OPCode_schedAdmin (scheduler-core design, cluster.c's case OPCode_upgradeConfig was checked as the closest precedent and has only a primary/standby routing gate, no RBAC/auth check) — scheduler-core ships OPCode_schedAdmin at parity with that existing gap (project owner decision: not a scheduler-core blocker), so this item now covers both opcodes; when validate_host() is implemented, wire it to both OP_upgradeConfig and OPCode_schedAdmin in the same pass. (b) Worker/Scheduler→Controller: workers and schedulers connect via controller_connector.c using the same TLS context as controller peers, with no role differentiation — a compromised worker could send OP_upgradeConfig or read full config via OP_replConfig. (c) Cross-component trust: no distinction between controller peers, admin tools, workers, and schedulers — they share the same X.509 identity pool. (d) xceadmin login (future): xceadmin currently uses zero-config TLS (same as xce_test). Production deployments likely need an admin login requirement — token-based session (xceadmin login → token → config push) or password-per-command. X.509 CN alone proves identity but not intent; a login step provides audit trail, session expiry, and operator accountability. Needs design: role-based X.509 CN validation (controller/admin/worker/scheduler roles), per-role OP code allowlists (now including OPCode_schedAdmin), admin login/session model, audit logging for admin operations. Separate from the existing validate_host() hardening (below) which is the minimal gate for OP_upgradeConfig/OPCode_schedAdmin specifically. Discuss + design before implementing; likely promotes to a spec via /kiro-discovery.
validate_host() real authorization — currently a stub returning success for all peers (join.c:13-16, return 1; unconditionally). CONF_ENABLE_UPGRADE_CONFIG is already 1 in the standard build (config_repl.h:27-28, no build-system override found) — the precondition this item was originally framed to guard against has already been crossed; OP_upgradeConfig (and OPCode_schedAdmin, added 2026-08-02, scheduler-core) are live and unauthenticated TODAY, not a future risk to prevent. Minimum fix: verify peer X.509 CN against a configured allowlist (--admin-cn flag or admin_cn_list config), reject non-allowlisted OP_upgradeConfig and OPCode_schedAdmin with clear error, WARN-log rejected attempts with CN+IP. Future: IP allowlist, role-based access, admin-specific CA. (FUTURE_WORK.md §6; security-critical.) Part of the broader peer-authentication-hardening item above.
OP_replConfig access control hardening — currently serves the full config blob to any CA-signed SSL peer (trusted-LAN model). Consider: verify requester is a known --peers controller, rate-limit per peer, minimum_role check (controllers only). (FUTURE_WORK.md §7.) Part of the broader peer-authentication-hardening item above.
lost-and-found drain command — xcecli lost-and-found drain: reassign or cancel sidelined jobs, reset lost_and_found_job_count. Automates operator cleanup after config upgrade. (FUTURE_WORK.md §4.)
~~OP_upgradeConfig standby redirect verification~~ — RESOLVED 2026-08-19 (finding: item's own premise doesn't match production behavior, but coverage is already complete). A standby does NOT return the primary's address for OPCode_upgradeConfig — ha_dispatch_gate rejects it before the handler's own dead defensive check ever runs, replying flat status=-XCE_ERR_STANDBY, data_which=nodata with no redirect fields; fill_redirect_reply() (join.c:27) is never called for this opcode. This is a documented, deliberate feature gap (see admin-config-tool design.md's "Revalidation Triggers"), not a missing test. Coverage is already complete from both directions: the client-side redirect-following mechanism is unit-tested against mocks (src/xceadmin/tests/test_discovery.py, test_config_push_mock.py::test_push_config_follows_one_redirect_to_primary), and the real server-side current behavior (no redirect, flat rejection) is live-verified against a real 3-node cluster and regression-locked (src/xceadmin/tests/test_e2e_live_ha_push.py::test_live_config_push_against_standby_returns_standby_rejection_not_redirect). The remaining real work — wiring fill_redirect_reply() for OPCode_upgradeConfig — is a feature task already tracked under admin-config-tool's Revalidation Triggers, not this item.
steering sync — ha-async-io.md synchronized to the three-state model + own-term gate (transitional-state row/role removed; heartbeat re-arm PRIMARY-only; PRIMARY no-quorum fact allowlist noted). Done as part of ha-single-leader-role task 6.1.
xce_test WAL/snapshot inspection tooling — extend the offline inspector family in xce_test (sibling of the shipped waldump, src/controller/testsuite/waldump.{c,h} + cmd_waldump). Two remaining commands: (a) snapdump <snap_dir|snap_file> — decode the XCESNAP\0 header (8-byte LSN; persist.c:1266 load_snapshot) and report whether the snapshot carries job/queue entities or is an LSN-only stub. This directly settles Open Fork #3 of the failover job-loss investigation (.kiro/specs/ha-commit-durability/INVESTIGATION-failover-job-loss.md): if snapshots are entity-less, any empty-rejoin/WAL-prune below committed is unrecoverable BY DESIGN. (b) walcompare <dirA> <dirB> — diff two nodes' WALs by lsn → crc/jobid to flag divergence/prefix-gaps between members instantly. Reuse the waldump.c reader/decoder; same zero-alloc capnp jobid decode + c_capnproto_dep/zlib_dep wiring already added to xce_test_deps. Nonzero exit on entity-less snapshot / divergence so both double as E2E assertions. (recorded 2026-06-20; waldump already done.)
~~xce_test shell command support~~ — IMPLEMENTED (task 2.1, spec xce-test-e2e-primitives, closed 2026-06-19; stale entry corrected 2026-08-19). preprocess_and_run() (src/controller/testsuite/test.c:950) detects a leading ! and runs the remainder via system(); the shell command's exit status becomes g_last_status (the $? auto-variable) and a non-zero status sets g_session_failed, aborting the script the same way any other failing command does — system()'s child inherits the parent's stdout/stderr fds, so output lands in whatever the invoking xce_test session's own output is already redirected to. Works in every REPL context (no leave_ctx()/disconnect), including mid-session inside worker/client contexts — used this session to fix bug-1579 (resource-value-verification's nested controller-check pattern). No parser/harness-layer change remains to do.
scheduler framework enhancement — PROMOTED to spec 2026-07-06, then DECOMPOSED 2026-07-27 into scheduler-core (IMPLEMENTATION-COMPLETE), scheduler-allocation, and scheduler-preemption — see In Flight above for current status. Scope notes from the original brief still apply: commit barrier wiring deferred (separate hardening item); GPU/affinity allocation deferred (pluggable via new allocator interface); new job selection policies deferred (framework enables them, ships separately). No longer tracked here.
[→] perf-tuning-extraction — PROMOTED to spec 2026-07-20 (clusterconfig-schema-split, .kiro/specs/clusterconfig-schema-split/). Originally recorded 2026-07-06 from admin-config-tool discovery as a documentation/audit item; the audit (2026-07-20) found the fields are load-bearing enough to need real code changes, not just a doc pass — candidates/vcpus/hash-tuning fields (jobmgr/workermgr/qmgr/resmgr/xstrings) are already dead weight in the ClusterConfig capnp blob (superseded by --peers/--vcpus CLI flags or never read into runtime state — see src/docs/cluster-config-split-analysis.md). port is NOT dead — a second review pass (2026-07-20) found cluster_open_listener (cluster.c:1216-1258) falls back to the blob's port field, and single-node mode (no --peers flag at all, main.c:293-294) depends on that fallback entirely; the spec now requires a replacement port-sourcing mechanism before the field can be removed (clusterconfig-schema-split Requirement 1.7), not a bare deletion. heartbeatInterval/inactiveHeartbeatCount (controller→worker liveness, NOT the HA/Raft election heartbeat) were investigated and deliberately kept IN ClusterConfig for operational uniformity despite passing the independence rule — see .kiro/steering/config-design.md. fetchJobInterval/netioTimeout left unresolved, flagged out-of-scope in the new spec for a future pass.
Raft/HA heartbeat interval+timeout as CLI parameters — NEW 2026-07-20 from clusterconfig-schema-split discovery. The controller-to-controller HA/Raft election heartbeat (ha->config.heartbeat_interval_ms/heartbeat_timeout_ms — the one where a cross-node mismatch risks spurious failover) is 100% hardcoded today (HA_DEFAULT_HEARTBEAT_INTERVAL_MS/HA_DEFAULT_HEARTBEAT_TIMEOUT_MS, cluster.c:1131) — there is no ClusterConfig field for it, and per user decision (2026-07-20) there should never be one: these values must be available to a controller BEFORE it has validated/received cluster configuration (a controller needs its own election-timing parameters to even participate in the election that delivers config in the first place — a bootstrap/chicken-egg constraint, not an independence/frequency judgment call). Target: --heartbeat-interval-ms/--heartbeat-timeout-ms CLI flags following the exact --vcpus pattern (OPT_* + struct arguments + myconf_t field, 0 sentinel = use hardcoded default). Kept hardcoded for now; this is a future item, not blocking clusterconfig-schema-split.
resource-distribution-mapping — created 2026-07-23, never added here (found 2026-08-19, itself a small doc gap). Physical resources have no "configured total vs. currently-consumed" concept anywhere in this codebase, live or dead — resources.yaml's resourcemap field (which expresses exactly this: a value shared across N hosts) is deliberately dropped by xceadmin config build (build.py:41-43), and the matching controller.capnp fields are commented out (controller.capnp:208-209). mark_host_res() (worker.c) actively rejects any resource flagged XCE_RESF_SHARED today, so "shared pool" and "per-host value" are mutually exclusive by construction in the live system right now. A working total/current pooling mechanism DOES exist in the scheduler (resource.c's sched_check_shared_satisfy_()), but it exclusively serves job-limit virtual resources (queue MAX_RUNNING-style), never physical shared resources — architecturally reusable, never connected. This spec (research.md done, 3 scoped options, requirements.md never generated — sitting at the decision point) is the existing owner of closing this gap; surfaced again 2026-08-19 when scheduler-core's REPL info resource command couldn't show a total for a GLOBAL/SHARED resource because there genuinely isn't one to show. Next: pick a scope option in research.md, then /kiro-spec-requirements resource-distribution-mapping.
scheduler-sim-diagnostics — NEW 2026-08-10 (/kiro-discovery). Open-ended, continuously-extended spec for scheduler simulation-mode debug/troubleshoot capability — added to over time as new investigation needs surface from discussion or from other in-flight scheduler specs (scheduler-allocation, scheduler-preemption). Extends scheduler-core's REPL (info job/queue/worker/resource/summary, breakpoints, snapshots). Two seed capabilities identified so far: (1) limit inspection — xce_limit_t/xce_vres_t config + live usage currently live only in the limit plugin's private kvmgr tables, invisible to the REPL; needs a new info limit <name|all> command + accessor plumbing. (2) decision-reasoning trace — xce_pendreason_t is captured on pend decisions but never printed by the REPL, and dispatch decisions have no reasoning trail at all (no record of which limit/resource checks passed) — needs new instrumentation in the allocation path (resource.c's sched_try_alloc_on_host_, plugin.c's limit_prepare_allocation) plus a REPL command to surface it. Explicitly simulation-mode only (no live/production controller-connected diagnostics) and read-only (no state mutation) per user decision 2026-08-10. Brief: .kiro/specs/scheduler-sim-diagnostics/brief.md. Not started — requirements to be added per-capability, not fully speced upfront. Next: /kiro-spec-init scheduler-sim-diagnostics when ready to begin.
xcecd_virtual concurrent-job support (MXJ > 1) — NEW 2026-08-19, recorded during bug-1589 investigation (.wolf/buglog.json). Live testing found that when a job array (multiple elements) is dispatched to a single xcecd_virtual worker in quick succession, only the first element's start ever gets confirmed back to the controller (on_job_start) — the rest are dispatched but never confirmed. Root-caused (or being root-caused — see bug-1589 for the confirmed mechanism once the in-flight investigation lands) to xcecd_virtual's job-simulation path: unlike the real xcecd binary, which launches an actual dbus-managed worker process per job (tracked by real pid, real IPC status reports), xcecd_virtual (src/worker/src/xcecd.c's xcecd_process_command(), VIRTUAL_CLUSTER branch) never launches a real process at all — it assigns a bare incrementing fake pid (static int virtual_pid, xcecd.c:27) and simulates job execution via a sleep-timer mechanism (xcecd_worker_utils.h's VIRTUAL_CLUSTER-only xcecd_worker_t fields start_time/sleep_time/killed_signal, check_worker_sleep_time_elapsed()). The immediate bug-1589 fix (once landed) addresses the specific reported symptom; this item is the broader follow-up: properly support N concurrent simulated jobs per xcecd_virtual worker, since MXJ (max jobs per host) > 1 is a legitimate, common configuration to want to exercise in virtual-cluster-mode testing/CI (this project's own dev workflow relies on xcecd_virtual for HA soak tests, resource-value verification, and now scheduler-core's live validation — several .wolf/buglog.json entries this session alone used it), not just a single-job-at-a-time edge case. Scope sketch (to refine once bug-1589's root cause is fully confirmed): audit every VIRTUAL_CLUSTER-only code path in src/worker/ (xcecd.c, main.c, xcecd_worker_utils.h/.c, process_worker_manager.c) for single-job assumptions (a lone poll/early-return that only processes one pending simulated job per tick, a shared mutable scratch struct reused across concurrent jobs, etc.) and generalize each to iterate the full set of in-flight simulated jobs, matching the pattern the real-process path (process_manager.c's process_info_t manager->processes[], a proper per-pid array) already establishes correctly. Dependencies: bug-1589 (must land first — its root-cause trace is the concrete map of what needs generalizing). Promote via /kiro-discovery or /kiro-spec-init once bug-1589 is closed and the exact mechanism is confirmed.

Superseded / Absorbed Items

~~Quorum-ACK gate before config restart~~ (FUTURE_WORK.md §2) — largely superseded by ha-consistency-invariants Requirement 5: configuration changes now require majority acknowledgment before being confirmed/committed, and fail to the requester on timeout (no silent rollback when standbys are offline). Residual to verify in that spec's design: restart sequencing after a committed config change (don't restart the primary before the commit barrier passes).
~~Lost-and-found keyword validation~~ (FUTURE_WORK.md §8) — folded into the admin-config-tool spec (brief: .kiro/specs/admin-config-tool/brief.md).
~~admin-config-tool backlog item~~ (FUTURE_WORK.md §1 + §8, 2026-05-12) — absorbed into admin-config-tool spec 2026-07-06 (/kiro-discovery). The one-paragraph backlog sketch is superseded by the full brief with architecture decisions, decomposition, and boundary contracts at .kiro/specs/admin-config-tool/brief.md.

Notes

Backlog items here are seeds for /kiro-discovery / /kiro-spec-init; promote an item by creating its spec under .kiro/specs/.
Provenance: items marked (FUTURE_WORK.md §N) originate from the 2026-05-12 config-replication-design.md review, merged here 2026-06-10.

2026-07-06 admin-config-tool discovery (/kiro-discovery, Path E mixed decomposition):

#	Workstream	Type	Language	Dependencies
1	admin-config-tool	NEW spec	Python	`controller.capnp` only
2	config-upgrade-snapshot	EXISTING spec extension	C	entity-bearing-snapshots (blocked upstream)
3	perf-tuning-extraction	DIRECT work	doc	None

Key architecture decisions: Python (future open-source, shared capnp infra), pure Python wire (no CGO/libxce), direct TCP+TLS to controller (not apiserver), single controller.capnp dependency, xceadmin separate from xcecli (user=REST, admin=TLS), auto primary detection via peer list + redirect. Full details: .kiro/specs/admin-config-tool/brief.md.

2026-07-06 scheduler-framework discovery (/kiro-discovery, Path C single-scope):

Key architecture decisions: Iterator-driven allocation (lazy plan iterator + host iterator composition), configurable job picker (plugin name from config, not hardcoded), combinatorial parallel job allocation (1host×N → 2hosts×⌈N/2⌉ → … → Nhhosts×1 sorted by preference), session-level unsatisfiable resreq fingerprint cache. Scope narrowed from original backlog seed: commit barrier wiring deferred, new job selection policies deferred (framework enables them), GPU/affinity allocation deferred (pluggable via allocator interface). Alternatives documented in brief.md (Approach A: incremental/rejected, Approach C: full framework/deferred). Full details: .kiro/specs/scheduler-framework/brief.md.

61 KiB Raw Blame History Unescape Escape