115 lines
46 KiB
Markdown
115 lines
46 KiB
Markdown
# Roadmap
|
||
|
||
## Overview
|
||
|
||
XCE is in the persistence + HA transformation (branch `persistent_ha`). Governing model: Raft consensus + CP (see `product.md` key decisions). Active spec work is tracked per-spec in `.kiro/specs/`; this file is the **single consolidated future-work document** — the forward-looking queue, long-deferred items, and hardening backlog. It absorbs the former `src/docs/FUTURE_WORK.md` (2026-05-12, config-replication-design review), which now redirects here.
|
||
|
||
## In Flight
|
||
|
||
- **scheduler-framework** — NEW 2026-07-06 (`/kiro-discovery`). Iterator-driven allocation architecture: lazy plan iterator + host iterator, combinatorial parallel job allocation, configurable job picker, session-level unsatisfiable resreq cache. Brief: `.kiro/specs/scheduler-framework/brief.md`. Dependencies: none hard (builds on existing plugin framework + select expression evaluator). Next: `/kiro-spec-requirements scheduler-framework`.
|
||
- ~~**ha-consistency-invariants** — UMBRELLA CONTRACT (requirements + design + research approved/complete). Implementation decomposed 2026-06-11 into the four sub-specs below; the umbrella generates NO tasks itself. Each sub-spec brief references the umbrella docs as authoritative.~~ **COMPLETE 2026-07-13** — all four sub-specs (ha-commit-engine, ha-commit-barrier-rollout, ha-single-leader-role, ha-convergence-recovery) implemented; Criterion 2.9 (Facts-Apply invariant) satisfied via raft-apply-decoupling. No longer in flight.
|
||
- ~~ha-wal-replication — implemented except task 5.4 (live 3-node integration test; needs `/xce-deploy-ha` cluster); sub-spec 4's E2E extends it~~ **task 5.4 CLOSED 2026-07-18** via `/kiro-validate-impl`'s live 10-round sequential soak (972/972 tests, zero loss). Spec is GO, no longer in flight.
|
||
- ~~**libxce-failover** — CLIENT-SIDE transparent controller failover (requirements generated 2026-06-16). Wires all libxce production request paths through the (already-built-but-orphaned) `xce_request_with_retry()` primitive + adds multi-master rotation; classifies on the real controller status taxonomy `E_XCE_REDIRECT` / `E_XCE_NOMASTER` / `E_XCE_NOT_READY` / `E_XCE_STANDBY` (`errnox.h:22-25`); bounded retry → distinct cluster-not-ready (no more opaque EINVAL); includes one small controller change so redirect replies carry the primary host/port (schema fields already exist, `controller.capnp:568-569`, just unpopulated in `join.c`). Verified E2E via `xce_test` (the original reproduction). Closes the integration gap left by `ha-config-bootstrap` Req 7 (tasks 10/10.1/10.2 shipped only the standalone primitive). See bug-608. Dependencies: ha-config-bootstrap (primitive + error codes), ha-commit-barrier-rollout (write idempotency makes write-retry safe).~~ **IMPLEMENTED, validate-impl GO 2026-07-18** (all 19 tasks were done since 2026-06-18; the `ready_for_implementation` flag was just never flipped — administrative staleness, not a real block). No longer in flight. `component-failover-adoption` (backlog, below) is the next layer on top of this.
|
||
|
||
## Raft TLA+ Conformance — raft-library enhancement series (NEW 2026-06-21)
|
||
|
||
**Goal:** evolve the election-only `src/lib/raft/raft.{c,h}` into a STRICT implementation of `raft/raft.tla` (Ongaro canonical) covering the log + commit half, keeping the pure callback-driven design so it stays independently verifiable via the deterministic simulator (`raft_sim.c`) against the TLA+ invariants (Election Safety, Log Matching, Leader Completeness, State Machine Safety). Motivation: **bug-851** — the log protocol is currently hand-rolled across `persist.c`/`repl.c`/`ha.c`/`ha_convergence.c` and violates the TLA+ log invariants. Gap analysis (six gaps G1–G6): `.kiro/specs/ha-term-safety/TLA-gap-analysis.md`. Implement in dependency order; each spec lands with simulator coverage. The library core stays pure/synchronous/deterministic — all I/O (async vtask, fan-out replication, the sync commit barrier, snapshot transport) lives in controller-side callback adapters. Preserve as policy (do NOT canonicalize away): PreVote, priority leadership, synchronous commit-barrier + persist-before-ACK durability, "controller never self-terminates".
|
||
|
||
- [x] raft-log-replication — *(specs generated + auto-approved 2026-06-21: 8 reqs, 24 sub-tasks, cross-spec review PASSED. Ready for /kiro-impl.)* Extend the library to OWN the replicated log. TLA+ actions: `ClientRequest`, `AppendEntries`, `HandleAppendEntriesRequest`, `UpdateTerm`-on-AppendEntries. Per-entry term in the library log model (closes **G1**); `AppendEntries` `logOk` = `prevLogTerm` match (**G2**); conflict branch deletes the divergent `(term,lsn)` suffix (**G3**); ANY higher-term RPC incl. AppendEntries/heartbeat advances `currentTerm` and steps down (**G5**). New pure callbacks: `on_persist_entry`, `on_get_entry_term(lsn)`, `on_send_append_entries(peer, prevLogIdx, prevLogTerm, entries, commitIdx)`, `on_truncate_from(lsn)`; controller adapters implement them over the existing WAL/repl transport. Simulator: drive AppendEntries across partitions/restarts, assert Log Matching. Dependencies: raft-library (election core). Closes bug-851 G1/G2/G3/G5.
|
||
- [x] raft-commit-safety — *(specs generated + auto-approved 2026-06-21: 6 reqs, 18 sub-tasks, cross-spec review PASSED.)* `AdvanceCommitIndex` with the Figure-8 current-term rule (commit `Max(agreeIndexes)` only if `log[Max].term == currentTerm`) (**G4**); `RequestVote`/PreVote freshness becomes `(lastLogTerm, lastLogIndex)` instead of LSN-only (**G6**). New callback `on_commit_advanced(lsn)` feeds the existing synchronous commit barrier / durability (persist-before-ACK preserved as policy). Simulator: assert Leader Completeness. Dependencies: raft-log-replication. Closes bug-851 G4/G6.
|
||
- [x] raft-sim-conformance — *(specs generated + auto-approved 2026-06-21: 8 reqs, 23 sub-tasks, cross-spec review PASSED.)* Extend `raft_sim.c` into a conformance/verification harness driving the library through `raft.tla`-action-equivalent transitions (randomized + targeted) and asserting the four Raft safety invariants; optionally trace-check against TLC output of `raft.tla`. The "use the verified spec to help" payoff: a regression oracle for the consensus core. Dependencies: raft-log-replication, raft-commit-safety. **IMPLEMENTED, validate-impl GO 2026-07-18** (23/23 tasks; 293/293 lib tests, zero ASAN findings; this is the spec this worktree/branch — `raft_tla` — is named for).
|
||
|
||
> **ha-term-safety** (`.kiro/specs/ha-term-safety/`) is the XCE-side INTEGRATION + REGRESSION umbrella for this family: it wires the controller WAL / commit-barrier into the new raft callbacks (thin adapter — protocol delegated to the library), owns the **config-change marker** (R7 — config changes as `(term,lsn)` log markers; bytes via the existing out-of-band config-push transport, hash-verified against the marker), and holds the live **bug-851 chaotic-failover regression**. Its protocol invariants R1–R6 are realized by the library specs above (not re-implemented in the controller). **Scope: R1–R7 only.** The former R8 (config-upgrade full-snapshot consistency, bug-857) was **descoped 2026-06-21 to the independent `config-upgrade-snapshot` spec** (below), because it requires entity-bearing snapshots that do not exist yet. **IMPLEMENTED, validate-impl GO 2026-07-17** (20/20 tasks; closed bug-1445, a first-boot promotion-check race with job-array-single-wal-record's Live Apply Resource Fence, as an out-of-boundary follow-up fix).
|
||
|
||
## Raft Apply-Ordering & Hook Decoupling — NEW 2026-06-24 (`/kiro-discovery`)
|
||
|
||
**Goal:** make the controller follow canonical Raft ordering — *pre-hooks → raft commit → apply → notify* — with the raft protocol step lifted OUT of the notification hook fan-out. Today `persist_job_hook_fun` (an `AFTER_EVENT` hook) performs the WAL append + synchronous majority-commit barrier inline, and `create_job()` mutates in-memory state BEFORE that commit (`job.c:558` before `job.c:593`; `persist.c:2532`/`persist.c:2198`/`ha.c:3080`). A failed/timed-out commit does not cleanly revert the optimistic mutation — the structural root of the bug-795/828/851/1060 divergence family. Decision (user, 2026-06-24): **apply-after-commit (canonical)** — a single shared `apply_committed_entry()` seam mutates the state machine only after commit, reused by restart replay so online-apply and replay-apply are identical. Hooks demote to notification-only. Decompose via Path E (2 new specs + touchpoints).
|
||
|
||
### Async-drive prerequisites (surfaced 2026-06-25 by blocking audit; thread-role contract now in `ha-async-io.md`)
|
||
|
||
The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_handler` runs inline on iomgr instead of dispatching to the vtask pool, and delaymgr (the heartbeat driver) was the documented blocking sink. Corrected thread-role contract is now steering (`ha-async-io.md`, 2026-06-25; SUPERSEDES "delaymgr for blocking ops = Permitted"). Two separable prerequisites (full evidence: `.kiro/specs/raft-apply-decoupling/facts-in-raft.md` § Keystone + "Two distinct prerequisites" + P2 offload mechanism):
|
||
|
||
- **P1 — commit-barrier async** (the *commit* drive). Convert `repl_wait_majority_ack()` from blocking `pthread_cond_timedwait` (commit.c:359) to event-driven `resolve(lsn,outcome)` on the existing commit engine. Dissolves apply-self-deadlock (B), enables reply continuations (C), removes vtask-pool exhaustion. **Owned within `raft-apply-decoupling` as its keystone** (sequenced before its Phase A).
|
||
- **P2 — drive hardening** (the *transport/timer* drive). Now its own spec **`raft-drive-async`** (brief: `.kiro/specs/raft-drive-async/brief.md`) — promoted because it threatens liveness independently. See the Specs list below.
|
||
|
||
### Specs (dependency order — Raft Drive & Apply Async family)
|
||
|
||
- [x] **raft-drive-async** — **IMPLEMENTED, validate-impl GO 2026-07-18** (21/21 tasks; the live sequential soak this validation ran surfaced a real crash, bug-1447 — a same-batch endpoint use-after-free in `io_worker`'s epoll dispatch loop, exposed by this spec's own faster reconnect cadence, see `ha-async-io.md`; root-caused, fixed, re-soaked clean, re-validated GO same day). Original description retained below for provenance. *(was: START FIRST — threatens failover liveness TODAY)* — bring the code into compliance with the corrected `ha-async-io.md` thread-role contract: dispatch `peer_ctrl_msg_handler` to the vtask pool (root-cause fix); snapshot send + config-push off iomgr → admission-capped slow lane (per-peer dedup + WAL coordination); delaymgr peer connect → bounded non-blocking + offloaded (reuse `xce_probe_config_tag` pattern); add a general non-IO vtask-submit path + admission reservation (extend `COMMIT_WAIT_RESERVED`). Verify with a **partition** (unreachable-not-killed) soak: heartbeats to survivors continue, no spurious election. Likely root of the bug-851 chaotic-failover family. Dependencies: none hard (builds on existing commit engine + vtask + the steering contract); `election-iomgr-refactor` is completed upstream (provides `delay_task_ms`/`on_connected`/non-blocking vote fan-out) — STANDALONE per 2026-06-25 scope-check (ADJACENT-SAME-AREA: it fixed startup blocking, this fixes runtime drive blocking). Brief + requirements + research + **design + tasks done**; phase=tasks-generated (tasks `approved: false`). **`/kiro-validate-design` GO 2026-06-25** — three review findings folded into design+tasks: (1) **deadlock-freedom + LivenessAdmission** — the documented invariant reversal (ack/`commit_on_ack` moves off iomgr to the vtask pool, `commit.c:16`) co-locates commit-barrier wakers+waiters; review found `COMMIT_WAIT_RESERVED` is **insufficient** because it counts only commit-waits while peer_ctrl also puts vote-grant fsync + config-ack waits on the pool (invisible to it). Fix: generalize the reservation to a **pool-level task-class admission in vtaskmgr** (`VTASK_LIVENESS`/`LIVE_BLOCK`/`REQ_BLOCK`; liveness never gated; `COMMIT_WAIT_RESERVED` folded in) — new **Phase-1 task 1.6** (dormant infra), hard prerequisite of 3.1; task 3.4 is the admission gate. Transient until `raft-apply-decoupling` P1 makes the barrier async; (2) **WAL-trim cursor-bound guard** added to task 3.2 (bug-1147, see Hardening below — moving checkpoint/trim onto a concurrent lane worsens the existing latent trim-vs-cursor race); (3) **in-process partition primitive** in `xce_test` as the portable primary for R7 (firewall DROP secondary) so the soak actually runs unprivileged. **Next: approve tasks (`/kiro-spec-tasks raft-drive-async`) → `/kiro-impl`; ship Phase 1 (bounded connect) first.**
|
||
- [x] raft-apply-decoupling — decouple the raft protocol step from the hook fan-out (item #1); canonical pre-hooks → commit → apply → notify ordering (item #2); single `apply_committed_entry()` seam shared by online commit + restart replay; post-restart validation that committed log was correctly applied to materialized state (item #4); requirement + regression tests (item #5). **Contains P1 (commit-barrier async) as its keystone.** Commands (WAL_MODE_WAIT) apply-after-commit; **facts (WAL_MODE_RETAIN) routed through commit-then-apply too** — D1 RESOLVED (block-but-durable under quorum loss); formalized as `ha-consistency-invariants` criterion 2.9. Dependencies: raft-drive-async, raft-log-replication + raft-commit-safety (DONE/implemented), worker-rejoin-resync; coordinate with ha-term-safety. Briefs: `.kiro/specs/raft-apply-decoupling/brief.md` + `facts-in-raft.md`. **IMPLEMENTED 2026-07-07: 32/32 tasks, validate-impl GO** — 79/79 controller_unit_test groups (926 tests) green, live 3-node smoke boot verified, zero blocked tasks. Task 7.2 fixed bug-1416 (boot-time TOCTOU false APPLY_DIVERGENT fence). Task 7.4's own throughput benchmark found and fixed a real O(n²) apply-pump regression (56% throughput drop, superlinear scaling) via a segment-read cache on `persist_t`, 2 adversarial review rounds. Residual, non-blocking: design.md has 3 sections not yet reconciled with implementation deviations already self-documented in tasks.md's Implementation Notes (repl.c's planned `repl_submit_majority_commit()` was never built; apply pump wired via `commit_resolve_pending()` not `ha.c`'s `on_commit_advanced`; fact hooks collapse to generic `HOOK_DETAIL_JOB_UPDATE` not per-type); task 5.3 touched `ha.c` without a formal `_Boundary:_` exception tag (substantively justified, process-inconsistent only). Two out-of-scope findings flagged for follow-up: an intermittent client hang in the synchronous `jobsub` path (resembles but not confirmed as bug-1416's class), and `trim_wal_segments`'s retention floor has no awareness of the local apply pump's own lag (same family as an already-tracked standby-cursor gap).
|
||
- [ ] worker-rejoin-resync — **elevated to HARD prereq for the facts half of raft-apply-decoupling 2026-06-25** (was backlog "designated backstop"). Worker re-reports running/finished job state on rejoin so facts lost in a failover window are recovered; re-reports flow through commit-then-apply idempotently (criterion 2.9). Today workers send only `hostattrs` on join (bug-396). Already on the backlog below; pulled forward into this family. Dependencies: ha-consistency-invariants (commit barrier + idempotency guards).
|
||
- [ ] job-array-single-wal-record — **PROMOTED from backlog 2026-06-24.** Persist/replicate/push a job-array submission as ONE Raft record (base spec + index range/count + per-index overrides) committed atomically through a single barrier; expand on apply (online + replay) and on the scheduler side; preserve per-element identity/addressing; idempotent re-apply (item #3). Dependencies: raft-apply-decoupling (HARD — builds on the shared apply seam), ha-commit-engine, ha-commit-barrier-rollout, ha-consistency-invariants. Brief: `.kiro/specs/job-array-single-wal-record/brief.md`.
|
||
|
||
### Existing Spec Updates (Raft Apply-Ordering phase)
|
||
|
||
- [ ] raft-sim-conformance — add apply-after-commit regression assertions (item #5): no apply on a failed/uncommitted entry; online-apply ≡ replay-apply for the same committed log; ordering invariant pre-hooks → commit → apply → notify. Revisit with `/kiro-spec-requirements raft-sim-conformance` once raft-apply-decoupling requirements land.
|
||
- [ ] ha-term-safety — coordinate: raft-apply-decoupling restructures the same controller submit→commit path this umbrella wires. Sequence the apply-decoupling edits against ha-term-safety's adapter work (disjoint-but-adjacent regions) to avoid conflicting refactors.
|
||
|
||
### Direct Implementation Candidates (Raft Apply-Ordering phase)
|
||
|
||
- [ ] xce_test waldump/walcompare/snapdump inspectors (already a Hardening item below) double as the offline evidence for the item #4 restart-validation requirement — wire their nonzero-exit-on-divergence into the decoupling spec's E2E rather than as a separate spec.
|
||
|
||
## Specs (dependency order)
|
||
|
||
> `[x]` = spec generated (requirements/design/tasks approved, ready for implementation). Implementation tracked via each spec's tasks.md; implement in roadmap order (cross-spec review 2026-06-11: four specs touch ha_dispatch_gate / raft_cb_become_follower in disjoint-but-adjacent regions).
|
||
|
||
- [x] ha-commit-engine — commit.c per-LSN majority engine + quorum accounting fixes, wired behind job submit. Dependencies: none. (specs done 2026-06-11; **IMPLEMENTED 2026-06-12**: 12/12 tasks, validate-impl GO, 467 tests green; deferred follow-ups in its tasks.md §Implementation Notes)
|
||
- [x] ha-commit-barrier-rollout — classification table, commit-first-then-reply sweep, RETAIN facts, config barrier, idempotency guards. Dependencies: ha-commit-engine. (specs done 2026-06-11; **IMPLEMENTED 2026-06-13**: 16/16 tasks, validate gate GO, 508 controller tests green under ASAN. Fixed bug-447 (key_find found-rc leak — queue/host control was silently unpersisted+unreplicated in production) and bug-396 (worker fact loss on quorum-loss gate rejection); task 5.2 needed one remediation round (wired the missing production config-ACK ingestion — would have bricked multi-member config upgrades — + kept the barrier off the iomgr thread). Logged not-fixed: bug-448 (job_ctrl ref leak), bug-451 (RUSAGE va_list OOB read). Barrier-before-state-removal precondition for ha-single-leader-role satisfied)
|
||
- [x] ha-single-leader-role — promotion no-op + own-term gate, transitional-state (the former fourth role) removal. Dependencies: ha-commit-engine, ha-commit-barrier-rollout (HARD: barrier before state removal). (specs done 2026-06-11: 2 reqs / 7 components / 11 sub-tasks)
|
||
- [x] ha-convergence-recovery — demotion truncate+self-restart, rejoin reconciliation, egress watermark filter, ACK-driven catch-up, quorum logging. Dependencies: ha-commit-engine (E2E additionally needs ha-single-leader-role). (specs done 2026-06-11: 9 reqs / 7 components / 13 sub-tasks)
|
||
- [x] ha-commit-durability — closes bug-661 (acknowledgment did not imply durability; committed jobs lost across failover). Makes acknowledged ⇒ durable on a majority: the primary counts its own copy toward the commit majority only at its durable LSN (`majority_candidate_locked` caps self at `persist_durable_lsn`, not the appended tip); `fsync_worker` repurposed timer→demand-driven group-commit flusher (one `fdatasync` over pending appends, advance `durable_lsn`, wake commit-eval) — absorbs the former "WAL group-commit batching" backlog item (see below); recovery integrity verdict (clean vs corruption-truncated WAL) + durable-backed freshness floor (candidate advertises bare `persist_durable_floor`, voter floor = `max(durable_floor, learned_committed)`) + pre-promotion eligibility fence. Establishes the **controllers-never-self-terminate** invariant: a recovery-truncated/behind member that cannot catch up is a LOUD PERMANENT FENCE (non-promotable, stays alive, keeps retrying — never `exit`/`abort`), self-heals on peer return (see `ha-async-io.md` process-lifecycle invariant + `src/controller/CLAUDE.md`). Dependencies: ha-commit-engine (consumes commit-position semantics), ha-wal-replication (standby ACK/catch-up), ha-commit-barrier-rollout (reply gate, unchanged). (specs done 2026-06-18; **IMPLEMENTED 2026-06-18**: 17/17 tasks (1.1–5.3), validate-impl GO, controller suite 0 failures under ASAN; deterministic ack-before-fdatasync E2E RED→GREEN + rolling-SIGKILL soak zero acked-job loss. Verified by tests + Protocol Verification Checklist, NOT by raft.tla (which assumes a durable log). Fixed bug-760 (adjacent `test_fact_hook_append` raced the async flusher — test made durable-deterministic via `persist_flush`). raft.tla durability-model extension deferred unless explicitly opted in.)
|
||
|
||
## Future Specs / Backlog (dependency order)
|
||
|
||
- [ ] config-upgrade-snapshot — **descoped from `ha-term-safety` R8 on 2026-06-21 (requirements generated; `.kiro/specs/config-upgrade-snapshot/`).** Closes **bug-857**: a configuration upgrade can leave members with DIVERGENT materialized state because today it does config-push + restart + per-node WAL replay under the new config (`config_repl.c` `do_upgrade_config_with_bytes` ~1001-1163), with no snapshot pinning an identical post-upgrade state (replay is config-dependent — `lost_and_found` re-homes orphaned entities). Approach: an upgrade materializes a **full entity-bearing state snapshot at the config-change marker's `(term,lsn)` boundary**, embeds the new config, and installs it on every member via the `snapshot_transfer_s` transport — no per-node re-derivation; supersedes `ha-term-safety` R7's interim config-push byte delivery for upgrades. **HARD dependency: `entity-bearing-snapshots`** (today snapshots are a 16-byte LSN sentinel; the entity payload does not exist) AND `ha-term-safety` (the config-change marker it anchors on). Verify by reproducing bug-857 (upgrade during chaotic multi-kill, diff materialized state). Promote to design once `entity-bearing-snapshots` lands.
|
||
- [ ] component-failover-adoption — **recorded 2026-06-16; depends on libxce-failover (HARD).** Make the C daemons and the Go apiserver actually exercise libxce's transparent failover. The mechanism reaches them for free (worker/scheduler/controller link `libxce_dep`; apiserver uses it via cgo, `src/frontend/apiserver/backend/clib.go`), but two per-component integration items remain: (1) each component must hand libxce the **full master list** instead of a single endpoint (today even the CLI uses only `conf->masters[0]`, `client.c:122`); (2) bounded retry makes the already-blocking `xce_request` block **longer** (seconds of backoff) — for the async C daemons (worker `controller_connector.c`, scheduler `handler.c`) this must be verified **liveness-safe**, i.e. it must not stall a thread that also drives heartbeats (the "never block the iomgr thread" rule). Scope: per-component master-list config + retry policy + liveness-safe blocking + per-component E2E failover for **scheduler, worker, apiserver**. (`xce_test` is NOT here — it is the in-spec verification client of libxce-failover.) Promote via `/kiro-spec-init component-failover-adoption` after libxce-failover lands.
|
||
- [ ] worker-rejoin-resync — **planned long ago, no effort yet (recorded 2026-06-10).** When a worker rejoins after disconnect or controller failover, it shall re-report its running/finished job states so the controller recovers facts whose notifications were lost. Evidence of the gap (ha-consistency-invariants research deep-dive §A): `xce_join_req` carries only `hostattrs_t` (src/lib/request.c:1154-1181) — no job state; worker notify is fire-and-forget with zero redelivery (bug-396), so any job start/finish/fail sent during a disconnect or failover window is permanently lost today. Scope sketch: extend the join/hello exchange (or a post-join message) with a running+recent job-state report; controller reconciles against its job table through the commit barrier (entries go through majority commit like any state change); requires the `mark_job_*` idempotency guards (shared need with ha-consistency-invariants' pending-facts queue). Complements — does not replace — the primary-side pending-facts queue: the queue covers quorum-loss windows while connected; resync covers disconnect/failover windows. **Elevated 2026-06-11**: designated as the official backstop in ha-consistency-invariants Req 2.8 / boundary context — facts the controller received are retained-and-retried until committed; facts that never arrived (or died with the controller) are recovered only by this resync. Dependencies: ha-consistency-invariants (commit barrier + idempotency guards). Extends: worker-join-refactor (implemented — covers attribute/endpoint update on rejoin only).
|
||
- [ ] **admin-config-tool** — **PROMOTED from backlog to spec 2026-07-06** (`/kiro-discovery`; brief at `.kiro/specs/admin-config-tool/brief.md`). **NEW: `xceadmin` Python CLI** for cluster operators — builds and deploys cluster configuration via direct TCP+TLS capnp wire to controller (NOT through apiserver). Two subcommands: **(a) `config build`** — directory scan: parse `configs/{hosts,queues,resources,users}.yaml`, validate against JSON schemas (`configs/schema/*.schema.json`), serialize `ClusterConfig` capnp struct to `cluster.bin` with SHA-256; **(b) `config push`** — zero-config dev (default `localhost:9991`, same as xce_test), single `--controller` flag for production, auto-discovers peers + primary from controller reply, sends `OP_upgradeConfig`, displays `{versionTag,ackedCount,error}`. **Single dependency: `controller.capnp`.** Pure Python (pycapnp + yaml + jsonschema + click), open-source ready (`pip install`). The tool is the client-side trigger for `config-upgrade-snapshot`'s controller-side path. MUST validate against reserved name `"lost_and_found"` for queue/host (defence-in-depth). xcecli comparison: `xcecli` = user tool (HTTP/HTTPS → apiserver → CGO → controller, JWT auth); `xceadmin` = operator tool (direct TCP+TLS → controller, zero-config, same trust model as controller peers). Shared future: `xcecommon` Python package with capnp schema + wire protocol reusable by both tools. Dependencies: `controller.capnp` schema only. **Blocks: nothing — ships independently.** Controller-side upgrade snapshot path (`config-upgrade-snapshot`) remains blocked on `entity-bearing-snapshots` (pre-existing). Next: `/kiro-spec-requirements admin-config-tool`.
|
||
- [ ] lightweight rejoin re-attach (failover optimization) — **recorded 2026-06-16; deferred (YAGNI).** Today, on controller failover a reconnecting daemon does a full re-join and apiserver/scheduler receive **full** state (e.g. the whole job list), not a delta; for a large cluster this is sizable but rare. Decision (libxce-failover design review): accept full-state rejoin — it is simple, correct, and still cheaper than a full-history replace with no snapshot (cf. IBM LSF). Future optimization: a session-token "re-attach" hello that lets the new primary skip re-pushing already-replicated state, or a delta-only state transfer keyed on the consumer's last-known committed LSN. Delta reconciliation is non-trivial (the primary must know what each consumer already holds). Promote only if failover-rejoin latency/bandwidth is **measured** as a problem at scale. Dependencies: libxce-failover (rejoin hook), component-failover-adoption (daemon rejoin content), worker-rejoin-resync (worker job state).
|
||
- [x] job-array-single-wal-record — **IMPLEMENTED, validate-impl GO 2026-07-17** (reopened once 2026-07-16 for three user-directed residual gaps — bug-1443 bulk-insert partial-visibility, task 4.2's boot-replay fence gap, bug-1444 replay_wal array-blind tally — all closed, re-validated fresh GO). **PROMOTED to a spec 2026-06-24** (`/kiro-discovery`; brief at `.kiro/specs/job-array-single-wal-record/brief.md`; now tracked under the *Raft Apply-Ordering & Hook Decoupling* phase above, sequenced after raft-apply-decoupling). Original note retained for provenance: Enhance controller & scheduler so a job-array submission persists, replicates, and pushes to the scheduler as a **single WAL record** rather than one record per array element. Today a job array fans out to N per-job state changes, so a 10k-element array means 10k WAL appends + 10k replicated commits + 10k scheduler pushes — amplifying commit-engine load, replication bandwidth, and failover replay time linearly in array size. Scope sketch: (a) controller writes one WAL entry describing the array (base spec + index range/count + per-index overrides) committed atomically through the commit barrier; (b) replication carries the single record so standbys reconstruct the whole array from one committed LSN; (c) scheduler receives the array as one push and expands locally. Must preserve per-element identity/state addressing after expansion (status, cancel, completion still operate per task id). Dependencies: ha-commit-engine + ha-commit-barrier-rollout (single atomic commit), ha-consistency-invariants (replicated scheduling state). Spec advised before implementation; promote via `/kiro-spec-init "job-array-single-wal-record"` when ready.
|
||
- [ ] bulk job-array query/cancel — **recorded 2026-07-13, deferred out of `job-array-single-wal-record` scope by design.** A single client operation to query or cancel an entire array (not one element at a time) is explicitly out of scope for `job-array-single-wal-record` — see that spec's Boundary Context. Reason: array elements can be in heterogeneous states at request time (done/exited elements need no action; running elements need a worker kill signal; pending elements need a direct state transition to cancelled without ever running), which is a materially different bulk-operation problem from atomic array *submission* and needs its own requirements (partial-failure semantics, atomicity vs best-effort per element, etc.) rather than being folded into the submission spec. Dependencies: `job-array-single-wal-record` (per-element identity/addressing this would operate over). Promote via `/kiro-discovery` once job-array-single-wal-record ships.
|
||
- [x] ~~WAL group-commit batching~~ — **recorded 2026-06-17 as deferred; ABSORBED INTO ha-commit-durability 2026-06-18 (design-review reversal).** The `/kiro-validate-design` review surfaced that the highest-volume WAL traffic is `WAL_MODE_RETAIN` facts (job start/finish/fail/rusage) — not array-collapsible and not covered by `job-array-single-wal-record` — so a pure per-entry synchronous design would penalize the dominant write path. Decision (with user): **do group-commit now** — repurpose `fsync_worker` from a timer into a demand-driven flusher (batch one `fdatasync` across pending appends, advance `durable_lsn`, wake commit-eval) as part of ha-commit-durability rather than deferring. No longer a separate backlog item.
|
||
- [ ] primary step-down-on-quorum-loss — optional CP follow-on noted in ha-consistency-invariants brief (writes already blocked on quorum loss; step-down is an availability/clarity optimization).
|
||
- [ ] read consistency / leader leases — Raft §8 read-only optimization; status queries may currently observe stale or provisional state (documented deviation in ha-consistency-invariants).
|
||
- [ ] entity-bearing snapshots ("Task 11" format) — snapshot files are currently a 16-byte LSN sentinel (persist.c:435-437); real entity serialization unblocks snapshot-based state transfer and faster recovery. **Cheaper than it looks (verified 2026-06-11)**: the dual-zarray COW machinery it needs is already built and unit-tested but dormant — `xzarray_freeze`/`xzarray_thaw` in the vtable (zarray.c:695-757, O(1) freeze, thaw with `should_retain` GC filter, test_zarray_cow.c) with zero production callers (kvmgr.c:297 `vfreeze = NULL /* wired in P0-2 */`). Note: single-generation COW — the snapshot consumer owns freeze/thaw; do not add second concurrent consumers. Slot-level only (snapshots array structure, not object contents — serialize item state under per-object locks during the freeze window). Once landed, also strengthens ha-consistency-invariants §C eviction option (iii) and faster rebuild-on-demotion. **Downstream (recorded 2026-06-21): `config-upgrade-snapshot` HARD-depends on this** — it needs an entity-bearing snapshot to deliver an identical materialized post-upgrade state to every member (bug-857). **This is also strategy B for catch-up of a far-behind / empty-rejoin standby (cursor=0) — recorded 2026-06-25:** WAL streaming alone cannot serve a standby whose needed range was trimmed (bug-795/828/1147), and a new node always starts empty, so safe aggressive WAL trimming + new-node join *requires* InstallSnapshot (snapshot-at-boundary + tail WAL). Until this lands, trim must stay cursor-bounded (strategy-A stopgap, see Hardening) and the system cannot safely admit a standby that has fallen below the trim floor. Scope to add when promoted: the standby-side snapshot-install / apply path (request snapshot when its cursor is below the leader's earliest retained LSN, install, resume from snapshot tail).
|
||
|
||
## Hardening / Direct-Implementation Items
|
||
|
||
Smaller items that likely don't warrant a full spec; pick up opportunistically or bundle into adjacent spec work.
|
||
|
||
- [ ] **investigate: job_ctrl direct-mutate path (admin suspend/resume) — migrate to commit-then-apply?** — **recorded 2026-07-13, surfaced during job-array-single-wal-record's `persist_job_hook_fun` investigation.** `raft-apply-decoupling` deliberately scoped `job_ctrl`'s `mark_job_update`/`mark_job_fail` (admin SIGCONT/SIGSTOP) OUT-OF-BOUNDARY (task 4.1/6.1 Implementation Notes) — confirmed the ONLY remaining live call site that still mutates state directly, THEN fires the `AFTER_EVENT` hook, with `persist_job_hook_fun` doing the real WAL append + async commit from inside the hook (not through `apply_committed_entry()`). This is the same mutate-before-commit shape as the bug-795/828/851/1060 divergence family raft-apply-decoupling was built to eliminate for job submit + worker facts, narrowly scoped to admin suspend/resume. Not yet known to have caused a live incident — rare, synchronous, admin-only traffic. **Investigate before committing to a fix**: (a) actual blast radius if a commit fails/times out post-mutation (does it ever get observed/notified before commit — violates criterion 2.4-style provisional-state-export concerns?); (b) live-apply vs replay-apply asymmetry for this one WAL entry type — does replay actually diverge from live in any reachable scenario, or is it benign in practice; (c) cost of migrating vs. accepted-residual status quo. Promote to a spec (likely a small `raft-apply-decoupling` follow-up, not standalone) only if the investigation finds real risk, not proactively.
|
||
- [ ] **WAL-trim cursor-bound guard (bug-1147; surfaced 2026-06-25 by raft-drive-async design review)** — `trim_wal_segments` (`persist.c:1374`) bounds deletion ONLY by the open segment (`if (seg_start_lsn < active_seg_lsn) unlink()`) — it consults **no** standby `acked_lsn`, **no** `committed_lsn`, **no** in-flight snapshot boundary. With no working snapshot backfill (`write_snapshot` emits a 16-byte sentinel only), a standby lagging by more than ~one open segment hits a log range that exists nowhere → permanent committed-prefix gap (the bug-795/828 family). This is **strategy-A retention discipline that should exist regardless**. Stopgap fix: compute `floor = min(all standby acked_lsn, in-flight snapshot boundaries)` under `p->lock` and refuse to delete at/above `floor`. **Folded into `raft-drive-async` task 3.2** (which moves checkpoint/trim onto a concurrent lane, worsening the race) — but stands alone if that slips. Does NOT fix empty/far-behind join (cursor=0); that needs `entity-bearing snapshots` (strategy B, backlog). Related: bug-795, bug-828, bug-1060.
|
||
- [ ] OP_heartbeat peers cross-validation — include a hash of the local `--peers` string in heartbeat requests; log ERROR on mismatch between candidates. Detects misconfigured nodes at election time instead of silently risking dual-primary split-brain. _(FUTURE_WORK.md §3.)_
|
||
- [ ] **peer-authentication-hardening** — **NEW 2026-07-06 from admin-config-tool discovery** (`/kiro-discovery`). XCE currently operates under a **trusted-LAN model** — any CA-signed SSL peer can connect to the controller and perform ANY operation. Three gaps identified: **(a) Admin→Controller**: `validate_host()` is a stub returning success for all peers (roadmap § below). `OP_upgradeConfig` has no real authorization gate today. **(b) Worker/Scheduler→Controller**: workers and schedulers connect via `controller_connector.c` using the same TLS context as controller peers, with no role differentiation — a compromised worker could send `OP_upgradeConfig` or read full config via `OP_replConfig`. **(c) Cross-component trust**: no distinction between controller peers, admin tools, workers, and schedulers — they share the same X.509 identity pool. **(d) xceadmin login (future)**: xceadmin currently uses zero-config TLS (same as xce_test). Production deployments likely need an admin login requirement — token-based session (`xceadmin login → token → config push`) or password-per-command. X.509 CN alone proves identity but not intent; a login step provides audit trail, session expiry, and operator accountability. **Needs design**: role-based X.509 CN validation (controller/admin/worker/scheduler roles), per-role OP code allowlists, admin login/session model, audit logging for admin operations. Separate from the existing `validate_host()` hardening (below) which is the minimal gate for `OP_upgradeConfig` specifically. Discuss + design before implementing; likely promotes to a spec via `/kiro-discovery`.
|
||
- [ ] validate_host() real authorization — currently a stub returning success for all peers; MUST be implemented before `CONF_ENABLE_UPGRADE_CONFIG=1`. Minimum: verify peer X.509 CN against a configured allowlist (`--admin-cn` flag or `admin_cn_list` config), reject non-allowlisted `OP_upgradeConfig` with clear error, WARN-log rejected attempts with CN+IP. Future: IP allowlist, role-based access, admin-specific CA. _(FUTURE_WORK.md §6; security-critical.)_ **Part of the broader `peer-authentication-hardening` item above.**
|
||
- [ ] OP_replConfig access control hardening — currently serves the full config blob to any CA-signed SSL peer (trusted-LAN model). Consider: verify requester is a known `--peers` controller, rate-limit per peer, `minimum_role` check (controllers only). _(FUTURE_WORK.md §7.)_ **Part of the broader `peer-authentication-hardening` item above.**
|
||
- [ ] lost-and-found drain command — `xcecli lost-and-found drain`: reassign or cancel sidelined jobs, reset `lost_and_found_job_count`. Automates operator cleanup after config upgrade. _(FUTURE_WORK.md §4.)_
|
||
- [ ] OP_upgradeConfig standby redirect verification — test that a standby returns the primary's address to an admin tool and the tool retries against the primary. Note: redirect semantics change under ha-consistency-invariants Req 1 (no transitional-state redirect); verify against the new behavior. _(FUTURE_WORK.md §5.)_
|
||
- [x] steering sync — `ha-async-io.md` synchronized to the three-state model + own-term gate (transitional-state row/role removed; heartbeat re-arm PRIMARY-only; PRIMARY no-quorum fact allowlist noted). Done as part of ha-single-leader-role task 6.1.
|
||
- [ ] xce_test WAL/snapshot inspection tooling — extend the offline inspector family in `xce_test` (sibling of the shipped `waldump`, `src/controller/testsuite/waldump.{c,h}` + `cmd_waldump`). Two remaining commands: **(a) `snapdump <snap_dir|snap_file>`** — decode the `XCESNAP\0` header (8-byte LSN; `persist.c:1266` `load_snapshot`) and report whether the snapshot carries job/queue entities or is an LSN-only stub. This directly settles Open Fork #3 of the failover job-loss investigation (`.kiro/specs/ha-commit-durability/INVESTIGATION-failover-job-loss.md`): if snapshots are entity-less, any empty-rejoin/WAL-prune below committed is unrecoverable BY DESIGN. **(b) `walcompare <dirA> <dirB>`** — diff two nodes' WALs by `lsn → crc/jobid` to flag divergence/prefix-gaps between members instantly. Reuse the `waldump.c` reader/decoder; same zero-alloc capnp jobid decode + `c_capnproto_dep`/`zlib_dep` wiring already added to `xce_test_deps`. Nonzero exit on entity-less snapshot / divergence so both double as E2E assertions. _(recorded 2026-06-20; `waldump` already done.)_
|
||
- [ ] xce_test shell command support — add `!<command>` prefix to `xce_test` test scripts to execute arbitrary shell commands inline during test runs. Enables test scripts to drive environment setup/teardown (e.g., kill a node, write a file, run a helper) without calling out to a separate harness layer. Scope: parser change in the xce_test runner to detect `!`-prefixed lines, fork/exec the remainder as a shell command, fail the test on non-zero exit, capture stdout/stderr to the test log.
|
||
- [→] scheduler framework enhancement — **PROMOTED to spec 2026-07-06** (`/kiro-discovery`; brief at `.kiro/specs/scheduler-framework/brief.md`). Original backlog seed superseded by the full brief with iterator-driven allocation architecture, configurable job picker, combinatorial parallel job allocation, and session-level unsatisfiable resreq cache. Scope narrowed from the original seed: commit barrier wiring deferred (separate hardening item); GPU/affinity allocation deferred (pluggable via new allocator interface); new job selection policies deferred (framework enables them, ships separately). Next: `/kiro-spec-requirements scheduler-framework`.
|
||
- [→] **perf-tuning-extraction** — **PROMOTED to spec 2026-07-20** (`clusterconfig-schema-split`, `.kiro/specs/clusterconfig-schema-split/`). Originally recorded 2026-07-06 from admin-config-tool discovery as a documentation/audit item; the audit (2026-07-20) found the fields are load-bearing enough to need real code changes, not just a doc pass — `candidates`/`vcpus`/hash-tuning fields (`jobmgr`/`workermgr`/`qmgr`/`resmgr`/`xstrings`) are already dead weight in the `ClusterConfig` capnp blob (superseded by `--peers`/`--vcpus` CLI flags or never read into runtime state — see `src/docs/cluster-config-split-analysis.md`). **`port` is NOT dead** — a second review pass (2026-07-20) found `cluster_open_listener` (`cluster.c:1216-1258`) falls back to the blob's `port` field, and single-node mode (no `--peers` flag at all, `main.c:293-294`) depends on that fallback entirely; the spec now requires a replacement port-sourcing mechanism before the field can be removed (`clusterconfig-schema-split` Requirement 1.7), not a bare deletion. `heartbeatInterval`/`inactiveHeartbeatCount` (controller→worker liveness, NOT the HA/Raft election heartbeat) were investigated and deliberately kept IN `ClusterConfig` for operational uniformity despite passing the independence rule — see `.kiro/steering/config-design.md`. `fetchJobInterval`/`netioTimeout` left unresolved, flagged out-of-scope in the new spec for a future pass.
|
||
- [ ] **Raft/HA heartbeat interval+timeout as CLI parameters** — **NEW 2026-07-20 from clusterconfig-schema-split discovery.** The controller-to-controller HA/Raft election heartbeat (`ha->config.heartbeat_interval_ms`/`heartbeat_timeout_ms` — the one where a cross-node mismatch risks spurious failover) is 100% hardcoded today (`HA_DEFAULT_HEARTBEAT_INTERVAL_MS`/`HA_DEFAULT_HEARTBEAT_TIMEOUT_MS`, `cluster.c:1131`) — there is no `ClusterConfig` field for it, and per user decision (2026-07-20) there should never be one: these values must be available to a controller BEFORE it has validated/received cluster configuration (a controller needs its own election-timing parameters to even participate in the election that delivers config in the first place — a bootstrap/chicken-egg constraint, not an independence/frequency judgment call). Target: `--heartbeat-interval-ms`/`--heartbeat-timeout-ms` CLI flags following the exact `--vcpus` pattern (`OPT_*` + `struct arguments` + `myconf_t` field, `0` sentinel = use hardcoded default). Kept hardcoded for now; this is a future item, not blocking `clusterconfig-schema-split`.
|
||
|
||
## Superseded / Absorbed Items
|
||
|
||
- ~~Quorum-ACK gate before config restart~~ _(FUTURE_WORK.md §2)_ — **largely superseded by ha-consistency-invariants Requirement 5**: configuration changes now require majority acknowledgment before being confirmed/committed, and fail to the requester on timeout (no silent rollback when standbys are offline). Residual to verify in that spec's design: restart sequencing after a committed config change (don't restart the primary before the commit barrier passes).
|
||
- ~~Lost-and-found keyword validation~~ _(FUTURE_WORK.md §8)_ — folded into the admin-config-tool spec (brief: `.kiro/specs/admin-config-tool/brief.md`).
|
||
- ~~admin-config-tool backlog item~~ _(FUTURE_WORK.md §1 + §8, 2026-05-12)_ — **absorbed into `admin-config-tool` spec 2026-07-06** (`/kiro-discovery`). The one-paragraph backlog sketch is superseded by the full brief with architecture decisions, decomposition, and boundary contracts at `.kiro/specs/admin-config-tool/brief.md`.
|
||
|
||
## Notes
|
||
|
||
- Backlog items here are seeds for `/kiro-discovery` / `/kiro-spec-init`; promote an item by creating its spec under `.kiro/specs/`.
|
||
- Provenance: items marked _(FUTURE_WORK.md §N)_ originate from the 2026-05-12 config-replication-design.md review, merged here 2026-06-10.
|
||
- **2026-07-06 admin-config-tool discovery** (`/kiro-discovery`, Path E mixed decomposition):
|
||
|
||
| # | Workstream | Type | Language | Dependencies |
|
||
|---|-----------|------|----------|--------------|
|
||
| 1 | **admin-config-tool** | NEW spec | Python | `controller.capnp` only |
|
||
| 2 | **config-upgrade-snapshot** | EXISTING spec extension | C | entity-bearing-snapshots (blocked upstream) |
|
||
| 3 | **perf-tuning-extraction** | DIRECT work | doc | None |
|
||
|
||
**Key architecture decisions:** Python (future open-source, shared capnp infra), pure Python wire (no CGO/libxce), direct TCP+TLS to controller (not apiserver), single `controller.capnp` dependency, `xceadmin` separate from `xcecli` (user=REST, admin=TLS), auto primary detection via peer list + redirect. Full details: `.kiro/specs/admin-config-tool/brief.md`.
|
||
- **2026-07-06 scheduler-framework discovery** (`/kiro-discovery`, Path C single-scope):
|
||
|
||
**Key architecture decisions:** Iterator-driven allocation (lazy plan iterator + host iterator composition), configurable job picker (plugin name from config, not hardcoded), combinatorial parallel job allocation (1host×N → 2hosts×⌈N/2⌉ → … → Nhhosts×1 sorted by preference), session-level unsatisfiable resreq fingerprint cache. Scope narrowed from original backlog seed: commit barrier wiring deferred, new job selection policies deferred (framework enables them), GPU/affinity allocation deferred (pluggable via allocator interface). Alternatives documented in brief.md (Approach A: incremental/rejected, Approach C: full framework/deferred). Full details: `.kiro/specs/scheduler-framework/brief.md`.
|