myxce/.kiro/steering/roadmap.md

Roadmap

Overview

XCE is in the persistence + HA transformation (branch persistent_ha). Governing model: Raft consensus + CP (see product.md key decisions). Active spec work is tracked per-spec in .kiro/specs/; this file is the single consolidated future-work document — the forward-looking queue, long-deferred items, and hardening backlog. It absorbs the former src/docs/FUTURE_WORK.md (2026-05-12, config-replication-design review), which now redirects here.

In Flight

• ha-consistency-invariants — UMBRELLA CONTRACT (requirements + design + research approved/complete). Implementation decomposed 2026-06-11 into the four sub-specs below; the umbrella generates NO tasks itself. Each sub-spec brief references the umbrella docs as authoritative.
• ha-wal-replication — implemented except task 5.4 (live 3-node integration test; needs /xce-deploy-ha cluster); sub-spec 4's E2E extends it
• libxce-failover — CLIENT-SIDE transparent controller failover (requirements generated 2026-06-16). Wires all libxce production request paths through the (already-built-but-orphaned) xce_request_with_retry() primitive + adds multi-master rotation; classifies on the real controller status taxonomy E_XCE_REDIRECT / E_XCE_NOMASTER / E_XCE_NOT_READY / E_XCE_STANDBY (errnox.h:22-25); bounded retry → distinct cluster-not-ready (no more opaque EINVAL); includes one small controller change so redirect replies carry the primary host/port (schema fields already exist, controller.capnp:568-569, just unpopulated in join.c). Verified E2E via xce_test (the original reproduction). Closes the integration gap left by ha-config-bootstrap Req 7 (tasks 10/10.1/10.2 shipped only the standalone primitive). See bug-608. Dependencies: ha-config-bootstrap (primitive + error codes), ha-commit-barrier-rollout (write idempotency makes write-retry safe).

Raft TLA+ Conformance — raft-library enhancement series (NEW 2026-06-21)

Goal: evolve the election-only src/lib/raft/raft.{c,h} into a STRICT implementation of raft/raft.tla (Ongaro canonical) covering the log + commit half, keeping the pure callback-driven design so it stays independently verifiable via the deterministic simulator (raft_sim.c) against the TLA+ invariants (Election Safety, Log Matching, Leader Completeness, State Machine Safety). Motivation: bug-851 — the log protocol is currently hand-rolled across persist.c/repl.c/ha.c/ha_convergence.c and violates the TLA+ log invariants. Gap analysis (six gaps G1–G6): .kiro/specs/ha-term-safety/TLA-gap-analysis.md. Implement in dependency order; each spec lands with simulator coverage. The library core stays pure/synchronous/deterministic — all I/O (async vtask, fan-out replication, the sync commit barrier, snapshot transport) lives in controller-side callback adapters. Preserve as policy (do NOT canonicalize away): PreVote, priority leadership, synchronous commit-barrier + persist-before-ACK durability, "controller never self-terminates".

• raft-log-replication — (specs generated + auto-approved 2026-06-21: 8 reqs, 24 sub-tasks, cross-spec review PASSED. Ready for /kiro-impl.) Extend the library to OWN the replicated log. TLA+ actions: ClientRequest, AppendEntries, HandleAppendEntriesRequest, UpdateTerm-on-AppendEntries. Per-entry term in the library log model (closes G1); AppendEntries logOk = prevLogTerm match (G2); conflict branch deletes the divergent (term,lsn) suffix (G3); ANY higher-term RPC incl. AppendEntries/heartbeat advances currentTerm and steps down (G5). New pure callbacks: on_persist_entry, on_get_entry_term(lsn), on_send_append_entries(peer, prevLogIdx, prevLogTerm, entries, commitIdx), on_truncate_from(lsn); controller adapters implement them over the existing WAL/repl transport. Simulator: drive AppendEntries across partitions/restarts, assert Log Matching. Dependencies: raft-library (election core). Closes bug-851 G1/G2/G3/G5.
• raft-commit-safety — (specs generated + auto-approved 2026-06-21: 6 reqs, 18 sub-tasks, cross-spec review PASSED.) AdvanceCommitIndex with the Figure-8 current-term rule (commit Max(agreeIndexes) only if log[Max].term == currentTerm) (G4); RequestVote/PreVote freshness becomes (lastLogTerm, lastLogIndex) instead of LSN-only (G6). New callback on_commit_advanced(lsn) feeds the existing synchronous commit barrier / durability (persist-before-ACK preserved as policy). Simulator: assert Leader Completeness. Dependencies: raft-log-replication. Closes bug-851 G4/G6.
• raft-sim-conformance — (specs generated + auto-approved 2026-06-21: 8 reqs, 23 sub-tasks, cross-spec review PASSED.) Extend raft_sim.c into a conformance/verification harness driving the library through raft.tla-action-equivalent transitions (randomized + targeted) and asserting the four Raft safety invariants; optionally trace-check against TLC output of raft.tla. The "use the verified spec to help" payoff: a regression oracle for the consensus core. Dependencies: raft-log-replication, raft-commit-safety.

ha-term-safety (requirements + design generated 2026-06-21; .kiro/specs/ha-term-safety/) is the XCE-side INTEGRATION + REGRESSION umbrella for this family: it wires the controller WAL / commit-barrier into the new raft callbacks (thin adapter — protocol delegated to the library), owns the config-change marker (R7 — config changes as (term,lsn) log markers; bytes via the existing out-of-band config-push transport, hash-verified against the marker), and holds the live bug-851 chaotic-failover regression. Its protocol invariants R1–R6 are realized by the library specs above (not re-implemented in the controller). Scope: R1–R7 only. The former R8 (config-upgrade full-snapshot consistency, bug-857) was descoped 2026-06-21 to the independent config-upgrade-snapshot spec (below), because it requires entity-bearing snapshots that do not exist yet. Next: /kiro-spec-tasks ha-term-safety.

Raft Apply-Ordering & Hook Decoupling — NEW 2026-06-24 (/kiro-discovery)

Goal: make the controller follow canonical Raft ordering — pre-hooks → raft commit → apply → notify — with the raft protocol step lifted OUT of the notification hook fan-out. Today persist_job_hook_fun (an AFTER_EVENT hook) performs the WAL append + synchronous majority-commit barrier inline, and create_job() mutates in-memory state BEFORE that commit (job.c:558 before job.c:593; persist.c:2532/persist.c:2198/ha.c:3080). A failed/timed-out commit does not cleanly revert the optimistic mutation — the structural root of the bug-795/828/851/1060 divergence family. Decision (user, 2026-06-24): apply-after-commit (canonical) — a single shared apply_committed_entry() seam mutates the state machine only after commit, reused by restart replay so online-apply and replay-apply are identical. Hooks demote to notification-only. Decompose via Path E (2 new specs + touchpoints).

Async-drive prerequisites (surfaced 2026-06-25 by blocking audit; thread-role contract now in ha-async-io.md)

The protocol drive is NOT fully event-driven. Root cause: peer_ctrl_msg_handler runs inline on iomgr instead of dispatching to the vtask pool, and delaymgr (the heartbeat driver) was the documented blocking sink. Corrected thread-role contract is now steering (ha-async-io.md, 2026-06-25; SUPERSEDES "delaymgr for blocking ops = Permitted"). Two separable prerequisites (full evidence: .kiro/specs/raft-apply-decoupling/facts-in-raft.md § Keystone + "Two distinct prerequisites" + P2 offload mechanism):

• P1 — commit-barrier async (the commit drive). Convert repl_wait_majority_ack() from blocking pthread_cond_timedwait (commit.c:359) to event-driven resolve(lsn,outcome) on the existing commit engine. Dissolves apply-self-deadlock (B), enables reply continuations (C), removes vtask-pool exhaustion. Owned within raft-apply-decoupling as its keystone (sequenced before its Phase A).
• P2 — drive hardening (the transport/timer drive). Now its own spec raft-drive-async (brief: .kiro/specs/raft-drive-async/brief.md) — promoted because it threatens liveness independently. See the Specs list below.

Specs (dependency order — Raft Drive & Apply Async family)

• raft-drive-async (START FIRST — threatens failover liveness TODAY) — bring the code into compliance with the corrected ha-async-io.md thread-role contract: dispatch peer_ctrl_msg_handler to the vtask pool (root-cause fix); snapshot send + config-push off iomgr → admission-capped slow lane (per-peer dedup + WAL coordination); delaymgr peer connect → bounded non-blocking + offloaded (reuse xce_probe_config_tag pattern); add a general non-IO vtask-submit path + admission reservation (extend COMMIT_WAIT_RESERVED). Verify with a partition (unreachable-not-killed) soak: heartbeats to survivors continue, no spurious election. Likely root of the bug-851 chaotic-failover family. Dependencies: none hard (builds on existing commit engine + vtask + the steering contract); election-iomgr-refactor is completed upstream (provides delay_task_ms/on_connected/non-blocking vote fan-out) — STANDALONE per 2026-06-25 scope-check (ADJACENT-SAME-AREA: it fixed startup blocking, this fixes runtime drive blocking). Brief + requirements + research + design + tasks done; phase=tasks-generated (tasks approved: false). /kiro-validate-design GO 2026-06-25 — three review findings folded into design+tasks: (1) deadlock-freedom + LivenessAdmission — the documented invariant reversal (ack/commit_on_ack moves off iomgr to the vtask pool, commit.c:16) co-locates commit-barrier wakers+waiters; review found COMMIT_WAIT_RESERVED is insufficient because it counts only commit-waits while peer_ctrl also puts vote-grant fsync + config-ack waits on the pool (invisible to it). Fix: generalize the reservation to a pool-level task-class admission in vtaskmgr (VTASK_LIVENESS/LIVE_BLOCK/REQ_BLOCK; liveness never gated; COMMIT_WAIT_RESERVED folded in) — new Phase-1 task 1.6 (dormant infra), hard prerequisite of 3.1; task 3.4 is the admission gate. Transient until raft-apply-decoupling P1 makes the barrier async; (2) WAL-trim cursor-bound guard added to task 3.2 (bug-1147, see Hardening below — moving checkpoint/trim onto a concurrent lane worsens the existing latent trim-vs-cursor race); (3) in-process partition primitive in xce_test as the portable primary for R7 (firewall DROP secondary) so the soak actually runs unprivileged. Next: approve tasks (/kiro-spec-tasks raft-drive-async) → /kiro-impl; ship Phase 1 (bounded connect) first.
• raft-apply-decoupling — decouple the raft protocol step from the hook fan-out (item #1); canonical pre-hooks → commit → apply → notify ordering (item #2); single apply_committed_entry() seam shared by online commit + restart replay; post-restart validation that committed log was correctly applied to materialized state (item #4); requirement + regression tests (item #5). Contains P1 (commit-barrier async) as its keystone. Commands (WAL_MODE_WAIT) apply-after-commit; facts (WAL_MODE_RETAIN) routed through commit-then-apply too — D1 RESOLVED (block-but-durable under quorum loss); formalized as ha-consistency-invariants criterion 2.9. Dependencies: raft-drive-async (HARD — reuses its slow-lane + admission infra; drive must be async first), raft-log-replication + raft-commit-safety (DONE/implemented), worker-rejoin-resync (HARD for the facts-recovery half — see below); coordinate with ha-term-safety. Briefs: .kiro/specs/raft-apply-decoupling/brief.md + facts-in-raft.md.
• worker-rejoin-resync — elevated to HARD prereq for the facts half of raft-apply-decoupling 2026-06-25 (was backlog "designated backstop"). Worker re-reports running/finished job state on rejoin so facts lost in a failover window are recovered; re-reports flow through commit-then-apply idempotently (criterion 2.9). Today workers send only hostattrs on join (bug-396). Already on the backlog below; pulled forward into this family. Dependencies: ha-consistency-invariants (commit barrier + idempotency guards).
• job-array-single-wal-record — PROMOTED from backlog 2026-06-24. Persist/replicate/push a job-array submission as ONE Raft record (base spec + index range/count + per-index overrides) committed atomically through a single barrier; expand on apply (online + replay) and on the scheduler side; preserve per-element identity/addressing; idempotent re-apply (item #3). Dependencies: raft-apply-decoupling (HARD — builds on the shared apply seam), ha-commit-engine, ha-commit-barrier-rollout, ha-consistency-invariants. Brief: .kiro/specs/job-array-single-wal-record/brief.md.

Existing Spec Updates (Raft Apply-Ordering phase)

• raft-sim-conformance — add apply-after-commit regression assertions (item #5): no apply on a failed/uncommitted entry; online-apply ≡ replay-apply for the same committed log; ordering invariant pre-hooks → commit → apply → notify. Revisit with /kiro-spec-requirements raft-sim-conformance once raft-apply-decoupling requirements land.
• ha-term-safety — coordinate: raft-apply-decoupling restructures the same controller submit→commit path this umbrella wires. Sequence the apply-decoupling edits against ha-term-safety's adapter work (disjoint-but-adjacent regions) to avoid conflicting refactors.

Direct Implementation Candidates (Raft Apply-Ordering phase)

• xce_test waldump/walcompare/snapdump inspectors (already a Hardening item below) double as the offline evidence for the item #4 restart-validation requirement — wire their nonzero-exit-on-divergence into the decoupling spec's E2E rather than as a separate spec.

Specs (dependency order)

[x] = spec generated (requirements/design/tasks approved, ready for implementation). Implementation tracked via each spec's tasks.md; implement in roadmap order (cross-spec review 2026-06-11: four specs touch ha_dispatch_gate / raft_cb_become_follower in disjoint-but-adjacent regions).

• ha-commit-engine — commit.c per-LSN majority engine + quorum accounting fixes, wired behind job submit. Dependencies: none. (specs done 2026-06-11; IMPLEMENTED 2026-06-12: 12/12 tasks, validate-impl GO, 467 tests green; deferred follow-ups in its tasks.md §Implementation Notes)
• ha-commit-barrier-rollout — classification table, commit-first-then-reply sweep, RETAIN facts, config barrier, idempotency guards. Dependencies: ha-commit-engine. (specs done 2026-06-11; IMPLEMENTED 2026-06-13: 16/16 tasks, validate gate GO, 508 controller tests green under ASAN. Fixed bug-447 (key_find found-rc leak — queue/host control was silently unpersisted+unreplicated in production) and bug-396 (worker fact loss on quorum-loss gate rejection); task 5.2 needed one remediation round (wired the missing production config-ACK ingestion — would have bricked multi-member config upgrades — + kept the barrier off the iomgr thread). Logged not-fixed: bug-448 (job_ctrl ref leak), bug-451 (RUSAGE va_list OOB read). Barrier-before-state-removal precondition for ha-single-leader-role satisfied)
• ha-single-leader-role — promotion no-op + own-term gate, transitional-state (the former fourth role) removal. Dependencies: ha-commit-engine, ha-commit-barrier-rollout (HARD: barrier before state removal). (specs done 2026-06-11: 2 reqs / 7 components / 11 sub-tasks)
• ha-convergence-recovery — demotion truncate+self-restart, rejoin reconciliation, egress watermark filter, ACK-driven catch-up, quorum logging. Dependencies: ha-commit-engine (E2E additionally needs ha-single-leader-role). (specs done 2026-06-11: 9 reqs / 7 components / 13 sub-tasks)
• ha-commit-durability — closes bug-661 (acknowledgment did not imply durability; committed jobs lost across failover). Makes acknowledged ⇒ durable on a majority: the primary counts its own copy toward the commit majority only at its durable LSN (majority_candidate_locked caps self at persist_durable_lsn, not the appended tip); fsync_worker repurposed timer→demand-driven group-commit flusher (one fdatasync over pending appends, advance durable_lsn, wake commit-eval) — absorbs the former "WAL group-commit batching" backlog item (see below); recovery integrity verdict (clean vs corruption-truncated WAL) + durable-backed freshness floor (candidate advertises bare persist_durable_floor, voter floor = max(durable_floor, learned_committed)) + pre-promotion eligibility fence. Establishes the controllers-never-self-terminate invariant: a recovery-truncated/behind member that cannot catch up is a LOUD PERMANENT FENCE (non-promotable, stays alive, keeps retrying — never exit/abort), self-heals on peer return (see ha-async-io.md process-lifecycle invariant + src/controller/CLAUDE.md). Dependencies: ha-commit-engine (consumes commit-position semantics), ha-wal-replication (standby ACK/catch-up), ha-commit-barrier-rollout (reply gate, unchanged). (specs done 2026-06-18; IMPLEMENTED 2026-06-18: 17/17 tasks (1.1–5.3), validate-impl GO, controller suite 0 failures under ASAN; deterministic ack-before-fdatasync E2E RED→GREEN + rolling-SIGKILL soak zero acked-job loss. Verified by tests + Protocol Verification Checklist, NOT by raft.tla (which assumes a durable log). Fixed bug-760 (adjacent test_fact_hook_append raced the async flusher — test made durable-deterministic via persist_flush). raft.tla durability-model extension deferred unless explicitly opted in.)

Future Specs / Backlog (dependency order)

• config-upgrade-snapshot — descoped from ha-term-safety R8 on 2026-06-21 (requirements generated; .kiro/specs/config-upgrade-snapshot/). Closes bug-857: a configuration upgrade can leave members with DIVERGENT materialized state because today it does config-push + restart + per-node WAL replay under the new config (config_repl.c do_upgrade_config_with_bytes ~1001-1163), with no snapshot pinning an identical post-upgrade state (replay is config-dependent — lost_and_found re-homes orphaned entities). Approach: an upgrade materializes a full entity-bearing state snapshot at the config-change marker's (term,lsn) boundary, embeds the new config, and installs it on every member via the snapshot_transfer_s transport — no per-node re-derivation; supersedes ha-term-safety R7's interim config-push byte delivery for upgrades. HARD dependency: entity-bearing-snapshots (today snapshots are a 16-byte LSN sentinel; the entity payload does not exist) AND ha-term-safety (the config-change marker it anchors on). Verify by reproducing bug-857 (upgrade during chaotic multi-kill, diff materialized state). Promote to design once entity-bearing-snapshots lands.
• component-failover-adoption — recorded 2026-06-16; depends on libxce-failover (HARD). Make the C daemons and the Go apiserver actually exercise libxce's transparent failover. The mechanism reaches them for free (worker/scheduler/controller link libxce_dep; apiserver uses it via cgo, src/frontend/apiserver/backend/clib.go), but two per-component integration items remain: (1) each component must hand libxce the full master list instead of a single endpoint (today even the CLI uses only conf->masters[0], client.c:122); (2) bounded retry makes the already-blocking xce_request block longer (seconds of backoff) — for the async C daemons (worker controller_connector.c, scheduler handler.c) this must be verified liveness-safe, i.e. it must not stall a thread that also drives heartbeats (the "never block the iomgr thread" rule). Scope: per-component master-list config + retry policy + liveness-safe blocking + per-component E2E failover for scheduler, worker, apiserver. (xce_test is NOT here — it is the in-spec verification client of libxce-failover.) Promote via /kiro-spec-init component-failover-adoption after libxce-failover lands.
• worker-rejoin-resync — planned long ago, no effort yet (recorded 2026-06-10). When a worker rejoins after disconnect or controller failover, it shall re-report its running/finished job states so the controller recovers facts whose notifications were lost. Evidence of the gap (ha-consistency-invariants research deep-dive §A): xce_join_req carries only hostattrs_t (src/lib/request.c:1154-1181) — no job state; worker notify is fire-and-forget with zero redelivery (bug-396), so any job start/finish/fail sent during a disconnect or failover window is permanently lost today. Scope sketch: extend the join/hello exchange (or a post-join message) with a running+recent job-state report; controller reconciles against its job table through the commit barrier (entries go through majority commit like any state change); requires the mark_job_* idempotency guards (shared need with ha-consistency-invariants' pending-facts queue). Complements — does not replace — the primary-side pending-facts queue: the queue covers quorum-loss windows while connected; resync covers disconnect/failover windows. Elevated 2026-06-11: designated as the official backstop in ha-consistency-invariants Req 2.8 / boundary context — facts the controller received are retained-and-retried until committed; facts that never arrived (or died with the controller) are recovered only by this resync. Dependencies: ha-consistency-invariants (commit barrier + idempotency guards). Extends: worker-join-refactor (implemented — covers attribute/endpoint update on rejoin only).
• admin-config-tool — drives OP_upgradeConfig (prerequisite for config-upgrade Phase 2). Encode ClusterConfig YAML/JSON → Cap'n Proto binary, SHA-256 over the bytes, connect to primary (SSL, same CA), send OP_upgradeConfig, display reply (version tag, acked count, errors), poll OP_probe after restart. Suggested CLI: xce-config push --controller <host:port> --config cluster.yaml. MUST validate that user configs don't use the reserved name "lost_and_found" for any queue/host (defence-in-depth: controller enforces at runtime via load_primary_config() FATAL, tool prevents at authoring time; migration: pre-existing clusters must rename before upgrade — see config-replication-design.md §6.5/§7.1). (From FUTURE_WORK.md §1 + §8, 2026-05-12.)
• lightweight rejoin re-attach (failover optimization) — recorded 2026-06-16; deferred (YAGNI). Today, on controller failover a reconnecting daemon does a full re-join and apiserver/scheduler receive full state (e.g. the whole job list), not a delta; for a large cluster this is sizable but rare. Decision (libxce-failover design review): accept full-state rejoin — it is simple, correct, and still cheaper than a full-history replace with no snapshot (cf. IBM LSF). Future optimization: a session-token "re-attach" hello that lets the new primary skip re-pushing already-replicated state, or a delta-only state transfer keyed on the consumer's last-known committed LSN. Delta reconciliation is non-trivial (the primary must know what each consumer already holds). Promote only if failover-rejoin latency/bandwidth is measured as a problem at scale. Dependencies: libxce-failover (rejoin hook), component-failover-adoption (daemon rejoin content), worker-rejoin-resync (worker job state).
• [→] job-array-single-wal-record — PROMOTED to a spec 2026-06-24 (/kiro-discovery; brief at .kiro/specs/job-array-single-wal-record/brief.md; now tracked under the Raft Apply-Ordering & Hook Decoupling phase above, sequenced after raft-apply-decoupling). Original note retained for provenance: Enhance controller & scheduler so a job-array submission persists, replicates, and pushes to the scheduler as a single WAL record rather than one record per array element. Today a job array fans out to N per-job state changes, so a 10k-element array means 10k WAL appends + 10k replicated commits + 10k scheduler pushes — amplifying commit-engine load, replication bandwidth, and failover replay time linearly in array size. Scope sketch: (a) controller writes one WAL entry describing the array (base spec + index range/count + per-index overrides) committed atomically through the commit barrier; (b) replication carries the single record so standbys reconstruct the whole array from one committed LSN; (c) scheduler receives the array as one push and expands locally. Must preserve per-element identity/state addressing after expansion (status, cancel, completion still operate per task id). Dependencies: ha-commit-engine + ha-commit-barrier-rollout (single atomic commit), ha-consistency-invariants (replicated scheduling state). Spec advised before implementation; promote via /kiro-spec-init "job-array-single-wal-record" when ready.
• WAL group-commit batching — recorded 2026-06-17 as deferred; ABSORBED INTO ha-commit-durability 2026-06-18 (design-review reversal). The /kiro-validate-design review surfaced that the highest-volume WAL traffic is WAL_MODE_RETAIN facts (job start/finish/fail/rusage) — not array-collapsible and not covered by job-array-single-wal-record — so a pure per-entry synchronous design would penalize the dominant write path. Decision (with user): do group-commit now — repurpose fsync_worker from a timer into a demand-driven flusher (batch one fdatasync across pending appends, advance durable_lsn, wake commit-eval) as part of ha-commit-durability rather than deferring. No longer a separate backlog item.
• primary step-down-on-quorum-loss — optional CP follow-on noted in ha-consistency-invariants brief (writes already blocked on quorum loss; step-down is an availability/clarity optimization).
• read consistency / leader leases — Raft §8 read-only optimization; status queries may currently observe stale or provisional state (documented deviation in ha-consistency-invariants).
• entity-bearing snapshots ("Task 11" format) — snapshot files are currently a 16-byte LSN sentinel (persist.c:435-437); real entity serialization unblocks snapshot-based state transfer and faster recovery. Cheaper than it looks (verified 2026-06-11): the dual-zarray COW machinery it needs is already built and unit-tested but dormant — xzarray_freeze/xzarray_thaw in the vtable (zarray.c:695-757, O(1) freeze, thaw with should_retain GC filter, test_zarray_cow.c) with zero production callers (kvmgr.c:297 vfreeze = NULL /* wired in P0-2 */). Note: single-generation COW — the snapshot consumer owns freeze/thaw; do not add second concurrent consumers. Slot-level only (snapshots array structure, not object contents — serialize item state under per-object locks during the freeze window). Once landed, also strengthens ha-consistency-invariants §C eviction option (iii) and faster rebuild-on-demotion. Downstream (recorded 2026-06-21): config-upgrade-snapshot HARD-depends on this — it needs an entity-bearing snapshot to deliver an identical materialized post-upgrade state to every member (bug-857). This is also strategy B for catch-up of a far-behind / empty-rejoin standby (cursor=0) — recorded 2026-06-25: WAL streaming alone cannot serve a standby whose needed range was trimmed (bug-795/828/1147), and a new node always starts empty, so safe aggressive WAL trimming + new-node join requires InstallSnapshot (snapshot-at-boundary + tail WAL). Until this lands, trim must stay cursor-bounded (strategy-A stopgap, see Hardening) and the system cannot safely admit a standby that has fallen below the trim floor. Scope to add when promoted: the standby-side snapshot-install / apply path (request snapshot when its cursor is below the leader's earliest retained LSN, install, resume from snapshot tail).

Hardening / Direct-Implementation Items

Smaller items that likely don't warrant a full spec; pick up opportunistically or bundle into adjacent spec work.

• WAL-trim cursor-bound guard (bug-1147; surfaced 2026-06-25 by raft-drive-async design review) — trim_wal_segments (persist.c:1374) bounds deletion ONLY by the open segment (if (seg_start_lsn < active_seg_lsn) unlink()) — it consults no standby acked_lsn, no committed_lsn, no in-flight snapshot boundary. With no working snapshot backfill (write_snapshot emits a 16-byte sentinel only), a standby lagging by more than ~one open segment hits a log range that exists nowhere → permanent committed-prefix gap (the bug-795/828 family). This is strategy-A retention discipline that should exist regardless. Stopgap fix: compute floor = min(all standby acked_lsn, in-flight snapshot boundaries) under p->lock and refuse to delete at/above floor. Folded into raft-drive-async task 3.2 (which moves checkpoint/trim onto a concurrent lane, worsening the race) — but stands alone if that slips. Does NOT fix empty/far-behind join (cursor=0); that needs entity-bearing snapshots (strategy B, backlog). Related: bug-795, bug-828, bug-1060.
• OP_heartbeat peers cross-validation — include a hash of the local --peers string in heartbeat requests; log ERROR on mismatch between candidates. Detects misconfigured nodes at election time instead of silently risking dual-primary split-brain. (FUTURE_WORK.md §3.)
• validate_host() real authorization — currently a stub returning success for all peers; MUST be implemented before CONF_ENABLE_UPGRADE_CONFIG=1. Minimum: verify peer X.509 CN against a configured allowlist (--admin-cn flag or admin_cn_list config), reject non-allowlisted OP_upgradeConfig with clear error, WARN-log rejected attempts with CN+IP. Future: IP allowlist, role-based access, admin-specific CA. (FUTURE_WORK.md §6; security-critical.)
• OP_replConfig access control hardening — currently serves the full config blob to any CA-signed SSL peer (trusted-LAN model). Consider: verify requester is a known --peers controller, rate-limit per peer, minimum_role check (controllers only). (FUTURE_WORK.md §7.)
• lost-and-found drain command — xcecli lost-and-found drain: reassign or cancel sidelined jobs, reset lost_and_found_job_count. Automates operator cleanup after config upgrade. (FUTURE_WORK.md §4.)
• OP_upgradeConfig standby redirect verification — test that a standby returns the primary's address to an admin tool and the tool retries against the primary. Note: redirect semantics change under ha-consistency-invariants Req 1 (no transitional-state redirect); verify against the new behavior. (FUTURE_WORK.md §5.)
• steering sync — ha-async-io.md synchronized to the three-state model + own-term gate (transitional-state row/role removed; heartbeat re-arm PRIMARY-only; PRIMARY no-quorum fact allowlist noted). Done as part of ha-single-leader-role task 6.1.
• xce_test WAL/snapshot inspection tooling — extend the offline inspector family in xce_test (sibling of the shipped waldump, src/controller/testsuite/waldump.{c,h} + cmd_waldump). Two remaining commands: (a) snapdump <snap_dir|snap_file> — decode the XCESNAP\0 header (8-byte LSN; persist.c:1266 load_snapshot) and report whether the snapshot carries job/queue entities or is an LSN-only stub. This directly settles Open Fork #3 of the failover job-loss investigation (.kiro/specs/ha-commit-durability/INVESTIGATION-failover-job-loss.md): if snapshots are entity-less, any empty-rejoin/WAL-prune below committed is unrecoverable BY DESIGN. (b) walcompare <dirA> <dirB> — diff two nodes' WALs by lsn → crc/jobid to flag divergence/prefix-gaps between members instantly. Reuse the waldump.c reader/decoder; same zero-alloc capnp jobid decode + c_capnproto_dep/zlib_dep wiring already added to xce_test_deps. Nonzero exit on entity-less snapshot / divergence so both double as E2E assertions. (recorded 2026-06-20; waldump already done.)
• xce_test shell command support — add !<command> prefix to xce_test test scripts to execute arbitrary shell commands inline during test runs. Enables test scripts to drive environment setup/teardown (e.g., kill a node, write a file, run a helper) without calling out to a separate harness layer. Scope: parser change in the xce_test runner to detect !-prefixed lines, fork/exec the remainder as a shell command, fail the test on non-zero exit, capture stdout/stderr to the test log.
• scheduler framework enhancement — strengthen the scheduler subsystem: (a) improve job scheduling policy extensibility (pluggable queue disciplines beyond FIFO), (b) add priority-aware dispatch, (c) improve resource-aware scheduling (CPU/memory/GPU slot tracking per host), (d) wire scheduler decisions through the commit barrier so scheduling state is replicated and survives failover. Dependencies: ha-consistency-invariants (commit barrier). Spec advised before implementation; promote via /kiro-spec-init "scheduler-framework" when ready.

Superseded / Absorbed Items

• Quorum-ACK gate before config restart (FUTURE_WORK.md §2) — largely superseded by ha-consistency-invariants Requirement 5: configuration changes now require majority acknowledgment before being confirmed/committed, and fail to the requester on timeout (no silent rollback when standbys are offline). Residual to verify in that spec's design: restart sequencing after a committed config change (don't restart the primary before the commit barrier passes).
• Lost-and-found keyword validation (FUTURE_WORK.md §8) — folded into the admin-config-tool backlog item above.

Notes

• Backlog items here are seeds for /kiro-discovery / /kiro-spec-init; promote an item by creating its spec under .kiro/specs/.
• Provenance: items marked (FUTURE_WORK.md §N) originate from the 2026-05-12 config-replication-design.md review, merged here 2026-06-10.