myxce/.kiro/steering/roadmap.md

Roadmap

Overview

XCE is in the persistence + HA transformation (branch persistent_ha). Governing model: Raft consensus + CP (see product.md key decisions). Active spec work is tracked per-spec in .kiro/specs/; this file is the single consolidated future-work document — the forward-looking queue, long-deferred items, and hardening backlog. It absorbs the former src/docs/FUTURE_WORK.md (2026-05-12, config-replication-design review), which now redirects here.

In Flight

• scheduler-framework — NEW 2026-07-06 (/kiro-discovery). Iterator-driven allocation architecture: lazy plan iterator + host iterator, combinatorial parallel job allocation, configurable job picker, session-level unsatisfiable resreq cache. Brief: .kiro/specs/scheduler-framework/brief.md. Dependencies: none hard (builds on existing plugin framework + select expression evaluator). Next: /kiro-spec-requirements scheduler-framework.
• ha-consistency-invariants — UMBRELLA CONTRACT (requirements + design + research approved/complete). Implementation decomposed 2026-06-11 into the four sub-specs below; the umbrella generates NO tasks itself. Each sub-spec brief references the umbrella docs as authoritative. COMPLETE 2026-07-13 — all four sub-specs (ha-commit-engine, ha-commit-barrier-rollout, ha-single-leader-role, ha-convergence-recovery) implemented; Criterion 2.9 (Facts-Apply invariant) satisfied via raft-apply-decoupling. No longer in flight.
• ha-wal-replication — implemented except task 5.4 (live 3-node integration test; needs /xce-deploy-ha cluster); sub-spec 4's E2E extends it task 5.4 CLOSED 2026-07-18 via /kiro-validate-impl's live 10-round sequential soak (972/972 tests, zero loss). Spec is GO, no longer in flight.
• libxce-failover — CLIENT-SIDE transparent controller failover (requirements generated 2026-06-16). Wires all libxce production request paths through the (already-built-but-orphaned) xce_request_with_retry() primitive + adds multi-master rotation; classifies on the real controller status taxonomy E_XCE_REDIRECT / E_XCE_NOMASTER / E_XCE_NOT_READY / E_XCE_STANDBY (errnox.h:22-25); bounded retry → distinct cluster-not-ready (no more opaque EINVAL); includes one small controller change so redirect replies carry the primary host/port (schema fields already exist, controller.capnp:568-569, just unpopulated in join.c). Verified E2E via xce_test (the original reproduction). Closes the integration gap left by ha-config-bootstrap Req 7 (tasks 10/10.1/10.2 shipped only the standalone primitive). See bug-608. Dependencies: ha-config-bootstrap (primitive + error codes), ha-commit-barrier-rollout (write idempotency makes write-retry safe). IMPLEMENTED, validate-impl GO 2026-07-18 (all 19 tasks were done since 2026-06-18; the ready_for_implementation flag was just never flipped — administrative staleness, not a real block). No longer in flight. component-failover-adoption (backlog, below) is the next layer on top of this.

Raft TLA+ Conformance — raft-library enhancement series (NEW 2026-06-21)

Goal: evolve the election-only src/lib/raft/raft.{c,h} into a STRICT implementation of raft/raft.tla (Ongaro canonical) covering the log + commit half, keeping the pure callback-driven design so it stays independently verifiable via the deterministic simulator (raft_sim.c) against the TLA+ invariants (Election Safety, Log Matching, Leader Completeness, State Machine Safety). Motivation: bug-851 — the log protocol is currently hand-rolled across persist.c/repl.c/ha.c/ha_convergence.c and violates the TLA+ log invariants. Gap analysis (six gaps G1–G6): .kiro/specs/ha-term-safety/TLA-gap-analysis.md. Implement in dependency order; each spec lands with simulator coverage. The library core stays pure/synchronous/deterministic — all I/O (async vtask, fan-out replication, the sync commit barrier, snapshot transport) lives in controller-side callback adapters. Preserve as policy (do NOT canonicalize away): PreVote, priority leadership, synchronous commit-barrier + persist-before-ACK durability, "controller never self-terminates".

• raft-log-replication — (specs generated + auto-approved 2026-06-21: 8 reqs, 24 sub-tasks, cross-spec review PASSED. Ready for /kiro-impl.) Extend the library to OWN the replicated log. TLA+ actions: ClientRequest, AppendEntries, HandleAppendEntriesRequest, UpdateTerm-on-AppendEntries. Per-entry term in the library log model (closes G1); AppendEntries logOk = prevLogTerm match (G2); conflict branch deletes the divergent (term,lsn) suffix (G3); ANY higher-term RPC incl. AppendEntries/heartbeat advances currentTerm and steps down (G5). New pure callbacks: on_persist_entry, on_get_entry_term(lsn), on_send_append_entries(peer, prevLogIdx, prevLogTerm, entries, commitIdx), on_truncate_from(lsn); controller adapters implement them over the existing WAL/repl transport. Simulator: drive AppendEntries across partitions/restarts, assert Log Matching. Dependencies: raft-library (election core). Closes bug-851 G1/G2/G3/G5.
• raft-commit-safety — (specs generated + auto-approved 2026-06-21: 6 reqs, 18 sub-tasks, cross-spec review PASSED.) AdvanceCommitIndex with the Figure-8 current-term rule (commit Max(agreeIndexes) only if log[Max].term == currentTerm) (G4); RequestVote/PreVote freshness becomes (lastLogTerm, lastLogIndex) instead of LSN-only (G6). New callback on_commit_advanced(lsn) feeds the existing synchronous commit barrier / durability (persist-before-ACK preserved as policy). Simulator: assert Leader Completeness. Dependencies: raft-log-replication. Closes bug-851 G4/G6.
• raft-sim-conformance — (specs generated + auto-approved 2026-06-21: 8 reqs, 23 sub-tasks, cross-spec review PASSED.) Extend raft_sim.c into a conformance/verification harness driving the library through raft.tla-action-equivalent transitions (randomized + targeted) and asserting the four Raft safety invariants; optionally trace-check against TLC output of raft.tla. The "use the verified spec to help" payoff: a regression oracle for the consensus core. Dependencies: raft-log-replication, raft-commit-safety. IMPLEMENTED, validate-impl GO 2026-07-18 (23/23 tasks; 293/293 lib tests, zero ASAN findings; this is the spec this worktree/branch — raft_tla — is named for).

ha-term-safety (.kiro/specs/ha-term-safety/) is the XCE-side INTEGRATION + REGRESSION umbrella for this family: it wires the controller WAL / commit-barrier into the new raft callbacks (thin adapter — protocol delegated to the library), owns the config-change marker (R7 — config changes as (term,lsn) log markers; bytes via the existing out-of-band config-push transport, hash-verified against the marker), and holds the live bug-851 chaotic-failover regression. Its protocol invariants R1–R6 are realized by the library specs above (not re-implemented in the controller). Scope: R1–R7 only. The former R8 (config-upgrade full-snapshot consistency, bug-857) was descoped 2026-06-21 to the independent config-upgrade-snapshot spec (below), because it requires entity-bearing snapshots that do not exist yet. IMPLEMENTED, validate-impl GO 2026-07-17 (20/20 tasks; closed bug-1445, a first-boot promotion-check race with job-array-single-wal-record's Live Apply Resource Fence, as an out-of-boundary follow-up fix).

Raft Apply-Ordering & Hook Decoupling — NEW 2026-06-24 (/kiro-discovery)

Goal: make the controller follow canonical Raft ordering — pre-hooks → raft commit → apply → notify — with the raft protocol step lifted OUT of the notification hook fan-out. Today persist_job_hook_fun (an AFTER_EVENT hook) performs the WAL append + synchronous majority-commit barrier inline, and create_job() mutates in-memory state BEFORE that commit (job.c:558 before job.c:593; persist.c:2532/persist.c:2198/ha.c:3080). A failed/timed-out commit does not cleanly revert the optimistic mutation — the structural root of the bug-795/828/851/1060 divergence family. Decision (user, 2026-06-24): apply-after-commit (canonical) — a single shared apply_committed_entry() seam mutates the state machine only after commit, reused by restart replay so online-apply and replay-apply are identical. Hooks demote to notification-only. Decompose via Path E (2 new specs + touchpoints).

Async-drive prerequisites (surfaced 2026-06-25 by blocking audit; thread-role contract now in ha-async-io.md)

The protocol drive is NOT fully event-driven. Root cause: peer_ctrl_msg_handler runs inline on iomgr instead of dispatching to the vtask pool, and delaymgr (the heartbeat driver) was the documented blocking sink. Corrected thread-role contract is now steering (ha-async-io.md, 2026-06-25; SUPERSEDES "delaymgr for blocking ops = Permitted"). Two separable prerequisites (full evidence: .kiro/specs/raft-apply-decoupling/facts-in-raft.md § Keystone + "Two distinct prerequisites" + P2 offload mechanism):

• P1 — commit-barrier async (the commit drive). Convert repl_wait_majority_ack() from blocking pthread_cond_timedwait (commit.c:359) to event-driven resolve(lsn,outcome) on the existing commit engine. Dissolves apply-self-deadlock (B), enables reply continuations (C), removes vtask-pool exhaustion. Owned within raft-apply-decoupling as its keystone (sequenced before its Phase A).
• P2 — drive hardening (the transport/timer drive). Now its own spec raft-drive-async (brief: .kiro/specs/raft-drive-async/brief.md) — promoted because it threatens liveness independently. See the Specs list below.

Specs (dependency order — Raft Drive & Apply Async family)

• raft-drive-async — IMPLEMENTED, validate-impl GO 2026-07-18 (21/21 tasks; the live sequential soak this validation ran surfaced a real crash, bug-1447 — a same-batch endpoint use-after-free in io_worker's epoll dispatch loop, exposed by this spec's own faster reconnect cadence, see ha-async-io.md; root-caused, fixed, re-soaked clean, re-validated GO same day). Original description retained below for provenance. (was: START FIRST — threatens failover liveness TODAY) — bring the code into compliance with the corrected ha-async-io.md thread-role contract: dispatch peer_ctrl_msg_handler to the vtask pool (root-cause fix); snapshot send + config-push off iomgr → admission-capped slow lane (per-peer dedup + WAL coordination); delaymgr peer connect → bounded non-blocking + offloaded (reuse xce_probe_config_tag pattern); add a general non-IO vtask-submit path + admission reservation (extend COMMIT_WAIT_RESERVED). Verify with a partition (unreachable-not-killed) soak: heartbeats to survivors continue, no spurious election. Likely root of the bug-851 chaotic-failover family. Dependencies: none hard (builds on existing commit engine + vtask + the steering contract); election-iomgr-refactor is completed upstream (provides delay_task_ms/on_connected/non-blocking vote fan-out) — STANDALONE per 2026-06-25 scope-check (ADJACENT-SAME-AREA: it fixed startup blocking, this fixes runtime drive blocking). Brief + requirements + research + design + tasks done; phase=tasks-generated (tasks approved: false). /kiro-validate-design GO 2026-06-25 — three review findings folded into design+tasks: (1) deadlock-freedom + LivenessAdmission — the documented invariant reversal (ack/commit_on_ack moves off iomgr to the vtask pool, commit.c:16) co-locates commit-barrier wakers+waiters; review found COMMIT_WAIT_RESERVED is insufficient because it counts only commit-waits while peer_ctrl also puts vote-grant fsync + config-ack waits on the pool (invisible to it). Fix: generalize the reservation to a pool-level task-class admission in vtaskmgr (VTASK_LIVENESS/LIVE_BLOCK/REQ_BLOCK; liveness never gated; COMMIT_WAIT_RESERVED folded in) — new Phase-1 task 1.6 (dormant infra), hard prerequisite of 3.1; task 3.4 is the admission gate. Transient until raft-apply-decoupling P1 makes the barrier async; (2) WAL-trim cursor-bound guard added to task 3.2 (bug-1147, see Hardening below — moving checkpoint/trim onto a concurrent lane worsens the existing latent trim-vs-cursor race); (3) in-process partition primitive in xce_test as the portable primary for R7 (firewall DROP secondary) so the soak actually runs unprivileged. Next: approve tasks (/kiro-spec-tasks raft-drive-async) → /kiro-impl; ship Phase 1 (bounded connect) first.
• raft-apply-decoupling — decouple the raft protocol step from the hook fan-out (item #1); canonical pre-hooks → commit → apply → notify ordering (item #2); single apply_committed_entry() seam shared by online commit + restart replay; post-restart validation that committed log was correctly applied to materialized state (item #4); requirement + regression tests (item #5). Contains P1 (commit-barrier async) as its keystone. Commands (WAL_MODE_WAIT) apply-after-commit; facts (WAL_MODE_RETAIN) routed through commit-then-apply too — D1 RESOLVED (block-but-durable under quorum loss); formalized as ha-consistency-invariants criterion 2.9. Dependencies: raft-drive-async, raft-log-replication + raft-commit-safety (DONE/implemented), worker-rejoin-resync; coordinate with ha-term-safety. Briefs: .kiro/specs/raft-apply-decoupling/brief.md + facts-in-raft.md. IMPLEMENTED 2026-07-07: 32/32 tasks, validate-impl GO — 79/79 controller_unit_test groups (926 tests) green, live 3-node smoke boot verified, zero blocked tasks. Task 7.2 fixed bug-1416 (boot-time TOCTOU false APPLY_DIVERGENT fence). Task 7.4's own throughput benchmark found and fixed a real O(n²) apply-pump regression (56% throughput drop, superlinear scaling) via a segment-read cache on persist_t, 2 adversarial review rounds. Residual, non-blocking: design.md has 3 sections not yet reconciled with implementation deviations already self-documented in tasks.md's Implementation Notes (repl.c's planned repl_submit_majority_commit() was never built; apply pump wired via commit_resolve_pending() not ha.c's on_commit_advanced; fact hooks collapse to generic HOOK_DETAIL_JOB_UPDATE not per-type); task 5.3 touched ha.c without a formal _Boundary:_ exception tag (substantively justified, process-inconsistent only). Two out-of-scope findings flagged for follow-up: an intermittent client hang in the synchronous jobsub path (resembles but not confirmed as bug-1416's class), and trim_wal_segments's retention floor has no awareness of the local apply pump's own lag (same family as an already-tracked standby-cursor gap).
• worker-rejoin-resync — elevated to HARD prereq for the facts half of raft-apply-decoupling 2026-06-25 (was backlog "designated backstop"). Worker re-reports running/finished job state on rejoin so facts lost in a failover window are recovered; re-reports flow through commit-then-apply idempotently (criterion 2.9). Today workers send only hostattrs on join (bug-396). Already on the backlog below; pulled forward into this family. Dependencies: ha-consistency-invariants (commit barrier + idempotency guards).
• job-array-single-wal-record — PROMOTED from backlog 2026-06-24. Persist/replicate/push a job-array submission as ONE Raft record (base spec + index range/count + per-index overrides) committed atomically through a single barrier; expand on apply (online + replay) and on the scheduler side; preserve per-element identity/addressing; idempotent re-apply (item #3). Dependencies: raft-apply-decoupling (HARD — builds on the shared apply seam), ha-commit-engine, ha-commit-barrier-rollout, ha-consistency-invariants. Brief: .kiro/specs/job-array-single-wal-record/brief.md.

Existing Spec Updates (Raft Apply-Ordering phase)

• raft-sim-conformance — add apply-after-commit regression assertions (item #5): no apply on a failed/uncommitted entry; online-apply ≡ replay-apply for the same committed log; ordering invariant pre-hooks → commit → apply → notify. Revisit with /kiro-spec-requirements raft-sim-conformance once raft-apply-decoupling requirements land.
• ha-term-safety — coordinate: raft-apply-decoupling restructures the same controller submit→commit path this umbrella wires. Sequence the apply-decoupling edits against ha-term-safety's adapter work (disjoint-but-adjacent regions) to avoid conflicting refactors.

Direct Implementation Candidates (Raft Apply-Ordering phase)

• xce_test waldump/walcompare/snapdump inspectors (already a Hardening item below) double as the offline evidence for the item #4 restart-validation requirement — wire their nonzero-exit-on-divergence into the decoupling spec's E2E rather than as a separate spec.

Specs (dependency order)

[x] = spec generated (requirements/design/tasks approved, ready for implementation). Implementation tracked via each spec's tasks.md; implement in roadmap order (cross-spec review 2026-06-11: four specs touch ha_dispatch_gate / raft_cb_become_follower in disjoint-but-adjacent regions).

• ha-commit-engine — commit.c per-LSN majority engine + quorum accounting fixes, wired behind job submit. Dependencies: none. (specs done 2026-06-11; IMPLEMENTED 2026-06-12: 12/12 tasks, validate-impl GO, 467 tests green; deferred follow-ups in its tasks.md §Implementation Notes)
• ha-commit-barrier-rollout — classification table, commit-first-then-reply sweep, RETAIN facts, config barrier, idempotency guards. Dependencies: ha-commit-engine. (specs done 2026-06-11; IMPLEMENTED 2026-06-13: 16/16 tasks, validate gate GO, 508 controller tests green under ASAN. Fixed bug-447 (key_find found-rc leak — queue/host control was silently unpersisted+unreplicated in production) and bug-396 (worker fact loss on quorum-loss gate rejection); task 5.2 needed one remediation round (wired the missing production config-ACK ingestion — would have bricked multi-member config upgrades — + kept the barrier off the iomgr thread). Logged not-fixed: bug-448 (job_ctrl ref leak), bug-451 (RUSAGE va_list OOB read). Barrier-before-state-removal precondition for ha-single-leader-role satisfied)
• ha-single-leader-role — promotion no-op + own-term gate, transitional-state (the former fourth role) removal. Dependencies: ha-commit-engine, ha-commit-barrier-rollout (HARD: barrier before state removal). (specs done 2026-06-11: 2 reqs / 7 components / 11 sub-tasks)
• ha-convergence-recovery — demotion truncate+self-restart, rejoin reconciliation, egress watermark filter, ACK-driven catch-up, quorum logging. Dependencies: ha-commit-engine (E2E additionally needs ha-single-leader-role). (specs done 2026-06-11: 9 reqs / 7 components / 13 sub-tasks)
• ha-commit-durability — closes bug-661 (acknowledgment did not imply durability; committed jobs lost across failover). Makes acknowledged ⇒ durable on a majority: the primary counts its own copy toward the commit majority only at its durable LSN (majority_candidate_locked caps self at persist_durable_lsn, not the appended tip); fsync_worker repurposed timer→demand-driven group-commit flusher (one fdatasync over pending appends, advance durable_lsn, wake commit-eval) — absorbs the former "WAL group-commit batching" backlog item (see below); recovery integrity verdict (clean vs corruption-truncated WAL) + durable-backed freshness floor (candidate advertises bare persist_durable_floor, voter floor = max(durable_floor, learned_committed)) + pre-promotion eligibility fence. Establishes the controllers-never-self-terminate invariant: a recovery-truncated/behind member that cannot catch up is a LOUD PERMANENT FENCE (non-promotable, stays alive, keeps retrying — never exit/abort), self-heals on peer return (see ha-async-io.md process-lifecycle invariant + src/controller/CLAUDE.md). Dependencies: ha-commit-engine (consumes commit-position semantics), ha-wal-replication (standby ACK/catch-up), ha-commit-barrier-rollout (reply gate, unchanged). (specs done 2026-06-18; IMPLEMENTED 2026-06-18: 17/17 tasks (1.1–5.3), validate-impl GO, controller suite 0 failures under ASAN; deterministic ack-before-fdatasync E2E RED→GREEN + rolling-SIGKILL soak zero acked-job loss. Verified by tests + Protocol Verification Checklist, NOT by raft.tla (which assumes a durable log). Fixed bug-760 (adjacent test_fact_hook_append raced the async flusher — test made durable-deterministic via persist_flush). raft.tla durability-model extension deferred unless explicitly opted in.)

Future Specs / Backlog (dependency order)

• config-upgrade-snapshot — descoped from ha-term-safety R8 on 2026-06-21 (requirements generated; .kiro/specs/config-upgrade-snapshot/). Closes bug-857: a configuration upgrade can leave members with DIVERGENT materialized state because today it does config-push + restart + per-node WAL replay under the new config (config_repl.c do_upgrade_config_with_bytes ~1001-1163), with no snapshot pinning an identical post-upgrade state (replay is config-dependent — lost_and_found re-homes orphaned entities). Approach: an upgrade materializes a full entity-bearing state snapshot at the config-change marker's (term,lsn) boundary, embeds the new config, and installs it on every member via the snapshot_transfer_s transport — no per-node re-derivation; supersedes ha-term-safety R7's interim config-push byte delivery for upgrades. HARD dependency: entity-bearing-snapshots (today snapshots are a 16-byte LSN sentinel; the entity payload does not exist) AND ha-term-safety (the config-change marker it anchors on). Verify by reproducing bug-857 (upgrade during chaotic multi-kill, diff materialized state). Promote to design once entity-bearing-snapshots lands.
• component-failover-adoption — recorded 2026-06-16; depends on libxce-failover (HARD). Make the C daemons and the Go apiserver actually exercise libxce's transparent failover. The mechanism reaches them for free (worker/scheduler/controller link libxce_dep; apiserver uses it via cgo, src/frontend/apiserver/backend/clib.go), but two per-component integration items remain: (1) each component must hand libxce the full master list instead of a single endpoint (today even the CLI uses only conf->masters[0], client.c:122); (2) bounded retry makes the already-blocking xce_request block longer (seconds of backoff) — for the async C daemons (worker controller_connector.c, scheduler handler.c) this must be verified liveness-safe, i.e. it must not stall a thread that also drives heartbeats (the "never block the iomgr thread" rule). Scope: per-component master-list config + retry policy + liveness-safe blocking + per-component E2E failover for scheduler, worker, apiserver. (xce_test is NOT here — it is the in-spec verification client of libxce-failover.) Promote via /kiro-spec-init component-failover-adoption after libxce-failover lands.
• worker-rejoin-resync — planned long ago, no effort yet (recorded 2026-06-10). When a worker rejoins after disconnect or controller failover, it shall re-report its running/finished job states so the controller recovers facts whose notifications were lost. Evidence of the gap (ha-consistency-invariants research deep-dive §A): xce_join_req carries only hostattrs_t (src/lib/request.c:1154-1181) — no job state; worker notify is fire-and-forget with zero redelivery (bug-396), so any job start/finish/fail sent during a disconnect or failover window is permanently lost today. Scope sketch: extend the join/hello exchange (or a post-join message) with a running+recent job-state report; controller reconciles against its job table through the commit barrier (entries go through majority commit like any state change); requires the mark_job_* idempotency guards (shared need with ha-consistency-invariants' pending-facts queue). Complements — does not replace — the primary-side pending-facts queue: the queue covers quorum-loss windows while connected; resync covers disconnect/failover windows. Elevated 2026-06-11: designated as the official backstop in ha-consistency-invariants Req 2.8 / boundary context — facts the controller received are retained-and-retried until committed; facts that never arrived (or died with the controller) are recovered only by this resync. Dependencies: ha-consistency-invariants (commit barrier + idempotency guards). Extends: worker-join-refactor (implemented — covers attribute/endpoint update on rejoin only).
• admin-config-tool — PROMOTED from backlog to spec 2026-07-06 (/kiro-discovery; brief at .kiro/specs/admin-config-tool/brief.md). NEW: xceadmin Python CLI for cluster operators — builds and deploys cluster configuration via direct TCP+TLS capnp wire to controller (NOT through apiserver). Two subcommands: (a) config build — directory scan: parse configs/{hosts,queues,resources,users}.yaml, validate against JSON schemas (configs/schema/*.schema.json), serialize ClusterConfig capnp struct to cluster.bin with SHA-256; (b) config push — zero-config dev (default localhost:9991, same as xce_test), single --controller flag for production, auto-discovers peers + primary from controller reply, sends OP_upgradeConfig, displays {versionTag,ackedCount,error}. Single dependency: controller.capnp. Pure Python (pycapnp + yaml + jsonschema + click), open-source ready (pip install). The tool is the client-side trigger for config-upgrade-snapshot's controller-side path. MUST validate against reserved name "lost_and_found" for queue/host (defence-in-depth). xcecli comparison: xcecli = user tool (HTTP/HTTPS → apiserver → CGO → controller, JWT auth); xceadmin = operator tool (direct TCP+TLS → controller, zero-config, same trust model as controller peers). Shared future: xcecommon Python package with capnp schema + wire protocol reusable by both tools. Dependencies: controller.capnp schema only. Blocks: nothing — ships independently. Controller-side upgrade snapshot path (config-upgrade-snapshot) remains blocked on entity-bearing-snapshots (pre-existing). Next: /kiro-spec-requirements admin-config-tool.
• lightweight rejoin re-attach (failover optimization) — recorded 2026-06-16; deferred (YAGNI). Today, on controller failover a reconnecting daemon does a full re-join and apiserver/scheduler receive full state (e.g. the whole job list), not a delta; for a large cluster this is sizable but rare. Decision (libxce-failover design review): accept full-state rejoin — it is simple, correct, and still cheaper than a full-history replace with no snapshot (cf. IBM LSF). Future optimization: a session-token "re-attach" hello that lets the new primary skip re-pushing already-replicated state, or a delta-only state transfer keyed on the consumer's last-known committed LSN. Delta reconciliation is non-trivial (the primary must know what each consumer already holds). Promote only if failover-rejoin latency/bandwidth is measured as a problem at scale. Dependencies: libxce-failover (rejoin hook), component-failover-adoption (daemon rejoin content), worker-rejoin-resync (worker job state).
• job-array-single-wal-record — IMPLEMENTED, validate-impl GO 2026-07-17 (reopened once 2026-07-16 for three user-directed residual gaps — bug-1443 bulk-insert partial-visibility, task 4.2's boot-replay fence gap, bug-1444 replay_wal array-blind tally — all closed, re-validated fresh GO). PROMOTED to a spec 2026-06-24 (/kiro-discovery; brief at .kiro/specs/job-array-single-wal-record/brief.md; now tracked under the Raft Apply-Ordering & Hook Decoupling phase above, sequenced after raft-apply-decoupling). Original note retained for provenance: Enhance controller & scheduler so a job-array submission persists, replicates, and pushes to the scheduler as a single WAL record rather than one record per array element. Today a job array fans out to N per-job state changes, so a 10k-element array means 10k WAL appends + 10k replicated commits + 10k scheduler pushes — amplifying commit-engine load, replication bandwidth, and failover replay time linearly in array size. Scope sketch: (a) controller writes one WAL entry describing the array (base spec + index range/count + per-index overrides) committed atomically through the commit barrier; (b) replication carries the single record so standbys reconstruct the whole array from one committed LSN; (c) scheduler receives the array as one push and expands locally. Must preserve per-element identity/state addressing after expansion (status, cancel, completion still operate per task id). Dependencies: ha-commit-engine + ha-commit-barrier-rollout (single atomic commit), ha-consistency-invariants (replicated scheduling state). Spec advised before implementation; promote via /kiro-spec-init "job-array-single-wal-record" when ready.
• bulk job-array query/cancel — recorded 2026-07-13, deferred out of job-array-single-wal-record scope by design. A single client operation to query or cancel an entire array (not one element at a time) is explicitly out of scope for job-array-single-wal-record — see that spec's Boundary Context. Reason: array elements can be in heterogeneous states at request time (done/exited elements need no action; running elements need a worker kill signal; pending elements need a direct state transition to cancelled without ever running), which is a materially different bulk-operation problem from atomic array submission and needs its own requirements (partial-failure semantics, atomicity vs best-effort per element, etc.) rather than being folded into the submission spec. Dependencies: job-array-single-wal-record (per-element identity/addressing this would operate over). Promote via /kiro-discovery once job-array-single-wal-record ships.
• WAL group-commit batching — recorded 2026-06-17 as deferred; ABSORBED INTO ha-commit-durability 2026-06-18 (design-review reversal). The /kiro-validate-design review surfaced that the highest-volume WAL traffic is WAL_MODE_RETAIN facts (job start/finish/fail/rusage) — not array-collapsible and not covered by job-array-single-wal-record — so a pure per-entry synchronous design would penalize the dominant write path. Decision (with user): do group-commit now — repurpose fsync_worker from a timer into a demand-driven flusher (batch one fdatasync across pending appends, advance durable_lsn, wake commit-eval) as part of ha-commit-durability rather than deferring. No longer a separate backlog item.
• primary step-down-on-quorum-loss — optional CP follow-on noted in ha-consistency-invariants brief (writes already blocked on quorum loss; step-down is an availability/clarity optimization).
• read consistency / leader leases — Raft §8 read-only optimization; status queries may currently observe stale or provisional state (documented deviation in ha-consistency-invariants).
• entity-bearing snapshots ("Task 11" format) — snapshot files are currently a 16-byte LSN sentinel (persist.c:435-437); real entity serialization unblocks snapshot-based state transfer and faster recovery. Cheaper than it looks (verified 2026-06-11): the dual-zarray COW machinery it needs is already built and unit-tested but dormant — xzarray_freeze/xzarray_thaw in the vtable (zarray.c:695-757, O(1) freeze, thaw with should_retain GC filter, test_zarray_cow.c) with zero production callers (kvmgr.c:297 vfreeze = NULL /* wired in P0-2 */). Note: single-generation COW — the snapshot consumer owns freeze/thaw; do not add second concurrent consumers. Slot-level only (snapshots array structure, not object contents — serialize item state under per-object locks during the freeze window). Once landed, also strengthens ha-consistency-invariants §C eviction option (iii) and faster rebuild-on-demotion. Downstream (recorded 2026-06-21): config-upgrade-snapshot HARD-depends on this — it needs an entity-bearing snapshot to deliver an identical materialized post-upgrade state to every member (bug-857). This is also strategy B for catch-up of a far-behind / empty-rejoin standby (cursor=0) — recorded 2026-06-25: WAL streaming alone cannot serve a standby whose needed range was trimmed (bug-795/828/1147), and a new node always starts empty, so safe aggressive WAL trimming + new-node join requires InstallSnapshot (snapshot-at-boundary + tail WAL). Until this lands, trim must stay cursor-bounded (strategy-A stopgap, see Hardening) and the system cannot safely admit a standby that has fallen below the trim floor. Scope to add when promoted: the standby-side snapshot-install / apply path (request snapshot when its cursor is below the leader's earliest retained LSN, install, resume from snapshot tail).

Hardening / Direct-Implementation Items

Smaller items that likely don't warrant a full spec; pick up opportunistically or bundle into adjacent spec work.

• investigate: job_ctrl direct-mutate path (admin suspend/resume) — migrate to commit-then-apply? — recorded 2026-07-13, surfaced during job-array-single-wal-record's persist_job_hook_fun investigation. raft-apply-decoupling deliberately scoped job_ctrl's mark_job_update/mark_job_fail (admin SIGCONT/SIGSTOP) OUT-OF-BOUNDARY (task 4.1/6.1 Implementation Notes) — confirmed the ONLY remaining live call site that still mutates state directly, THEN fires the AFTER_EVENT hook, with persist_job_hook_fun doing the real WAL append + async commit from inside the hook (not through apply_committed_entry()). This is the same mutate-before-commit shape as the bug-795/828/851/1060 divergence family raft-apply-decoupling was built to eliminate for job submit + worker facts, narrowly scoped to admin suspend/resume. Not yet known to have caused a live incident — rare, synchronous, admin-only traffic. Investigate before committing to a fix: (a) actual blast radius if a commit fails/times out post-mutation (does it ever get observed/notified before commit — violates criterion 2.4-style provisional-state-export concerns?); (b) live-apply vs replay-apply asymmetry for this one WAL entry type — does replay actually diverge from live in any reachable scenario, or is it benign in practice; (c) cost of migrating vs. accepted-residual status quo. Promote to a spec (likely a small raft-apply-decoupling follow-up, not standalone) only if the investigation finds real risk, not proactively.
• WAL-trim cursor-bound guard (bug-1147; surfaced 2026-06-25 by raft-drive-async design review) — trim_wal_segments (persist.c:1374) bounds deletion ONLY by the open segment (if (seg_start_lsn < active_seg_lsn) unlink()) — it consults no standby acked_lsn, no committed_lsn, no in-flight snapshot boundary. With no working snapshot backfill (write_snapshot emits a 16-byte sentinel only), a standby lagging by more than ~one open segment hits a log range that exists nowhere → permanent committed-prefix gap (the bug-795/828 family). This is strategy-A retention discipline that should exist regardless. Stopgap fix: compute floor = min(all standby acked_lsn, in-flight snapshot boundaries) under p->lock and refuse to delete at/above floor. Folded into raft-drive-async task 3.2 (which moves checkpoint/trim onto a concurrent lane, worsening the race) — but stands alone if that slips. Does NOT fix empty/far-behind join (cursor=0); that needs entity-bearing snapshots (strategy B, backlog). Related: bug-795, bug-828, bug-1060.
• OP_heartbeat peers cross-validation — include a hash of the local --peers string in heartbeat requests; log ERROR on mismatch between candidates. Detects misconfigured nodes at election time instead of silently risking dual-primary split-brain. (FUTURE_WORK.md §3.)
• peer-authentication-hardening — NEW 2026-07-06 from admin-config-tool discovery (/kiro-discovery). XCE currently operates under a trusted-LAN model — any CA-signed SSL peer can connect to the controller and perform ANY operation. Three gaps identified: (a) Admin→Controller: validate_host() is a stub returning success for all peers (roadmap § below). OP_upgradeConfig has no real authorization gate today. (b) Worker/Scheduler→Controller: workers and schedulers connect via controller_connector.c using the same TLS context as controller peers, with no role differentiation — a compromised worker could send OP_upgradeConfig or read full config via OP_replConfig. (c) Cross-component trust: no distinction between controller peers, admin tools, workers, and schedulers — they share the same X.509 identity pool. (d) xceadmin login (future): xceadmin currently uses zero-config TLS (same as xce_test). Production deployments likely need an admin login requirement — token-based session (xceadmin login → token → config push) or password-per-command. X.509 CN alone proves identity but not intent; a login step provides audit trail, session expiry, and operator accountability. Needs design: role-based X.509 CN validation (controller/admin/worker/scheduler roles), per-role OP code allowlists, admin login/session model, audit logging for admin operations. Separate from the existing validate_host() hardening (below) which is the minimal gate for OP_upgradeConfig specifically. Discuss + design before implementing; likely promotes to a spec via /kiro-discovery.
• validate_host() real authorization — currently a stub returning success for all peers; MUST be implemented before CONF_ENABLE_UPGRADE_CONFIG=1. Minimum: verify peer X.509 CN against a configured allowlist (--admin-cn flag or admin_cn_list config), reject non-allowlisted OP_upgradeConfig with clear error, WARN-log rejected attempts with CN+IP. Future: IP allowlist, role-based access, admin-specific CA. (FUTURE_WORK.md §6; security-critical.) Part of the broader peer-authentication-hardening item above.
• OP_replConfig access control hardening — currently serves the full config blob to any CA-signed SSL peer (trusted-LAN model). Consider: verify requester is a known --peers controller, rate-limit per peer, minimum_role check (controllers only). (FUTURE_WORK.md §7.) Part of the broader peer-authentication-hardening item above.
• lost-and-found drain command — xcecli lost-and-found drain: reassign or cancel sidelined jobs, reset lost_and_found_job_count. Automates operator cleanup after config upgrade. (FUTURE_WORK.md §4.)
• OP_upgradeConfig standby redirect verification — test that a standby returns the primary's address to an admin tool and the tool retries against the primary. Note: redirect semantics change under ha-consistency-invariants Req 1 (no transitional-state redirect); verify against the new behavior. (FUTURE_WORK.md §5.)
• steering sync — ha-async-io.md synchronized to the three-state model + own-term gate (transitional-state row/role removed; heartbeat re-arm PRIMARY-only; PRIMARY no-quorum fact allowlist noted). Done as part of ha-single-leader-role task 6.1.
• xce_test WAL/snapshot inspection tooling — extend the offline inspector family in xce_test (sibling of the shipped waldump, src/controller/testsuite/waldump.{c,h} + cmd_waldump). Two remaining commands: (a) snapdump <snap_dir|snap_file> — decode the XCESNAP\0 header (8-byte LSN; persist.c:1266 load_snapshot) and report whether the snapshot carries job/queue entities or is an LSN-only stub. This directly settles Open Fork #3 of the failover job-loss investigation (.kiro/specs/ha-commit-durability/INVESTIGATION-failover-job-loss.md): if snapshots are entity-less, any empty-rejoin/WAL-prune below committed is unrecoverable BY DESIGN. (b) walcompare <dirA> <dirB> — diff two nodes' WALs by lsn → crc/jobid to flag divergence/prefix-gaps between members instantly. Reuse the waldump.c reader/decoder; same zero-alloc capnp jobid decode + c_capnproto_dep/zlib_dep wiring already added to xce_test_deps. Nonzero exit on entity-less snapshot / divergence so both double as E2E assertions. (recorded 2026-06-20; waldump already done.)
• xce_test shell command support — add !<command> prefix to xce_test test scripts to execute arbitrary shell commands inline during test runs. Enables test scripts to drive environment setup/teardown (e.g., kill a node, write a file, run a helper) without calling out to a separate harness layer. Scope: parser change in the xce_test runner to detect !-prefixed lines, fork/exec the remainder as a shell command, fail the test on non-zero exit, capture stdout/stderr to the test log.
• [→] scheduler framework enhancement — PROMOTED to spec 2026-07-06 (/kiro-discovery; brief at .kiro/specs/scheduler-framework/brief.md). Original backlog seed superseded by the full brief with iterator-driven allocation architecture, configurable job picker, combinatorial parallel job allocation, and session-level unsatisfiable resreq cache. Scope narrowed from the original seed: commit barrier wiring deferred (separate hardening item); GPU/affinity allocation deferred (pluggable via new allocator interface); new job selection policies deferred (framework enables them, ships separately). Next: /kiro-spec-requirements scheduler-framework.
• perf-tuning-extraction — NEW 2026-07-06 from admin-config-tool discovery (/kiro-discovery). Audit ClusterConfig fields in controller.capnp and identify parameters used only for controller performance tuning (e.g., heartbeat intervals, timeout values, pool sizes, WAL segment sizing). Extract them from the cluster config and create a separate integration plan document for future performance tuning (how to expose, configure, validate, and apply perf tunables without requiring a full cluster config upgrade). This is direct-implementation work (documentation + field audit), not a spec.

Superseded / Absorbed Items

• Quorum-ACK gate before config restart (FUTURE_WORK.md §2) — largely superseded by ha-consistency-invariants Requirement 5: configuration changes now require majority acknowledgment before being confirmed/committed, and fail to the requester on timeout (no silent rollback when standbys are offline). Residual to verify in that spec's design: restart sequencing after a committed config change (don't restart the primary before the commit barrier passes).
• Lost-and-found keyword validation (FUTURE_WORK.md §8) — folded into the admin-config-tool spec (brief: .kiro/specs/admin-config-tool/brief.md).
• admin-config-tool backlog item (FUTURE_WORK.md §1 + §8, 2026-05-12) — absorbed into admin-config-tool spec 2026-07-06 (/kiro-discovery). The one-paragraph backlog sketch is superseded by the full brief with architecture decisions, decomposition, and boundary contracts at .kiro/specs/admin-config-tool/brief.md.

Notes

• Backlog items here are seeds for /kiro-discovery / /kiro-spec-init; promote an item by creating its spec under .kiro/specs/.

• Provenance: items marked (FUTURE_WORK.md §N) originate from the 2026-05-12 config-replication-design.md review, merged here 2026-06-10.

• 2026-07-06 admin-config-tool discovery (/kiro-discovery, Path E mixed decomposition):

  #	Workstream	Type	Language	Dependencies
  1	admin-config-tool	NEW spec	Python	controller.capnp only
  2	config-upgrade-snapshot	EXISTING spec extension	C	entity-bearing-snapshots (blocked upstream)
  3	perf-tuning-extraction	DIRECT work	doc	None

  Key architecture decisions: Python (future open-source, shared capnp infra), pure Python wire (no CGO/libxce), direct TCP+TLS to controller (not apiserver), single controller.capnp dependency, xceadmin separate from xcecli (user=REST, admin=TLS), auto primary detection via peer list + redirect. Full details: .kiro/specs/admin-config-tool/brief.md.

  • 2026-07-06 scheduler-framework discovery (/kiro-discovery, Path C single-scope):

    Key architecture decisions: Iterator-driven allocation (lazy plan iterator + host iterator composition), configurable job picker (plugin name from config, not hardcoded), combinatorial parallel job allocation (1host×N → 2hosts×⌈N/2⌉ → … → Nhhosts×1 sorted by preference), session-level unsatisfiable resreq fingerprint cache. Scope narrowed from original backlog seed: commit barrier wiring deferred, new job selection policies deferred (framework enables them), GPU/affinity allocation deferred (pluggable via allocator interface). Alternatives documented in brief.md (Approach A: incremental/rejected, Approach C: full framework/deferred). Full details: .kiro/specs/scheduler-framework/brief.md.