myxce/.kiro/steering/roadmap.md

Roadmap

Overview

XCE is in the persistence + HA transformation (branch persistent_ha). Governing model: Raft consensus + CP (see product.md key decisions). Active spec work is tracked per-spec in .kiro/specs/; this file is the single consolidated future-work document — the forward-looking queue, long-deferred items, and hardening backlog. It absorbs the former src/docs/FUTURE_WORK.md (2026-05-12, config-replication-design review), which now redirects here.

In Flight

• ha-consistency-invariants — UMBRELLA CONTRACT (requirements + design + research approved/complete). Implementation decomposed 2026-06-11 into the four sub-specs below; the umbrella generates NO tasks itself. Each sub-spec brief references the umbrella docs as authoritative.
• ha-wal-replication — implemented except task 5.4 (live 3-node integration test; needs /xce-deploy-ha cluster); sub-spec 4's E2E extends it

Specs (dependency order)

[x] = spec generated (requirements/design/tasks approved, ready for implementation). Implementation tracked via each spec's tasks.md; implement in roadmap order (cross-spec review 2026-06-11: four specs touch ha_dispatch_gate / raft_cb_become_follower in disjoint-but-adjacent regions).

• ha-commit-engine — commit.c per-LSN majority engine + quorum accounting fixes, wired behind job submit. Dependencies: none. (specs done 2026-06-11; IMPLEMENTED 2026-06-12: 12/12 tasks, validate-impl GO, 467 tests green; deferred follow-ups in its tasks.md §Implementation Notes)
• ha-commit-barrier-rollout — classification table, commit-first-then-reply sweep, RETAIN facts, config barrier, idempotency guards. Dependencies: ha-commit-engine. (specs done 2026-06-11; IMPLEMENTED 2026-06-13: 16/16 tasks, validate gate GO, 508 controller tests green under ASAN. Fixed bug-447 (key_find found-rc leak — queue/host control was silently unpersisted+unreplicated in production) and bug-396 (worker fact loss on quorum-loss gate rejection); task 5.2 needed one remediation round (wired the missing production config-ACK ingestion — would have bricked multi-member config upgrades — + kept the barrier off the iomgr thread). Logged not-fixed: bug-448 (job_ctrl ref leak), bug-451 (RUSAGE va_list OOB read). Barrier-before-state-removal precondition for ha-single-leader-role satisfied)
• ha-single-leader-role — promotion no-op + own-term gate, transitional-state (the former fourth role) removal. Dependencies: ha-commit-engine, ha-commit-barrier-rollout (HARD: barrier before state removal). (specs done 2026-06-11: 2 reqs / 7 components / 11 sub-tasks)
• ha-convergence-recovery — demotion truncate+self-restart, rejoin reconciliation, egress watermark filter, ACK-driven catch-up, quorum logging. Dependencies: ha-commit-engine (E2E additionally needs ha-single-leader-role). (specs done 2026-06-11: 9 reqs / 7 components / 13 sub-tasks)

Future Specs / Backlog (dependency order)

• worker-rejoin-resync — planned long ago, no effort yet (recorded 2026-06-10). When a worker rejoins after disconnect or controller failover, it shall re-report its running/finished job states so the controller recovers facts whose notifications were lost. Evidence of the gap (ha-consistency-invariants research deep-dive §A): xce_join_req carries only hostattrs_t (src/lib/request.c:1154-1181) — no job state; worker notify is fire-and-forget with zero redelivery (bug-396), so any job start/finish/fail sent during a disconnect or failover window is permanently lost today. Scope sketch: extend the join/hello exchange (or a post-join message) with a running+recent job-state report; controller reconciles against its job table through the commit barrier (entries go through majority commit like any state change); requires the mark_job_* idempotency guards (shared need with ha-consistency-invariants' pending-facts queue). Complements — does not replace — the primary-side pending-facts queue: the queue covers quorum-loss windows while connected; resync covers disconnect/failover windows. Elevated 2026-06-11: designated as the official backstop in ha-consistency-invariants Req 2.8 / boundary context — facts the controller received are retained-and-retried until committed; facts that never arrived (or died with the controller) are recovered only by this resync. Dependencies: ha-consistency-invariants (commit barrier + idempotency guards). Extends: worker-join-refactor (implemented — covers attribute/endpoint update on rejoin only).
• admin-config-tool — drives OP_upgradeConfig (prerequisite for config-upgrade Phase 2). Encode ClusterConfig YAML/JSON → Cap'n Proto binary, SHA-256 over the bytes, connect to primary (SSL, same CA), send OP_upgradeConfig, display reply (version tag, acked count, errors), poll OP_probe after restart. Suggested CLI: xce-config push --controller <host:port> --config cluster.yaml. MUST validate that user configs don't use the reserved name "lost_and_found" for any queue/host (defence-in-depth: controller enforces at runtime via load_primary_config() FATAL, tool prevents at authoring time; migration: pre-existing clusters must rename before upgrade — see config-replication-design.md §6.5/§7.1). (From FUTURE_WORK.md §1 + §8, 2026-05-12.)
• primary step-down-on-quorum-loss — optional CP follow-on noted in ha-consistency-invariants brief (writes already blocked on quorum loss; step-down is an availability/clarity optimization).
• read consistency / leader leases — Raft §8 read-only optimization; status queries may currently observe stale or provisional state (documented deviation in ha-consistency-invariants).
• entity-bearing snapshots ("Task 11" format) — snapshot files are currently a 16-byte LSN sentinel (persist.c:435-437); real entity serialization unblocks snapshot-based state transfer and faster recovery. Cheaper than it looks (verified 2026-06-11): the dual-zarray COW machinery it needs is already built and unit-tested but dormant — xzarray_freeze/xzarray_thaw in the vtable (zarray.c:695-757, O(1) freeze, thaw with should_retain GC filter, test_zarray_cow.c) with zero production callers (kvmgr.c:297 vfreeze = NULL /* wired in P0-2 */). Note: single-generation COW — the snapshot consumer owns freeze/thaw; do not add second concurrent consumers. Slot-level only (snapshots array structure, not object contents — serialize item state under per-object locks during the freeze window). Once landed, also strengthens ha-consistency-invariants §C eviction option (iii) and faster rebuild-on-demotion.

Hardening / Direct-Implementation Items

Smaller items that likely don't warrant a full spec; pick up opportunistically or bundle into adjacent spec work.

• OP_heartbeat peers cross-validation — include a hash of the local --peers string in heartbeat requests; log ERROR on mismatch between candidates. Detects misconfigured nodes at election time instead of silently risking dual-primary split-brain. (FUTURE_WORK.md §3.)
• validate_host() real authorization — currently a stub returning success for all peers; MUST be implemented before CONF_ENABLE_UPGRADE_CONFIG=1. Minimum: verify peer X.509 CN against a configured allowlist (--admin-cn flag or admin_cn_list config), reject non-allowlisted OP_upgradeConfig with clear error, WARN-log rejected attempts with CN+IP. Future: IP allowlist, role-based access, admin-specific CA. (FUTURE_WORK.md §6; security-critical.)
• OP_replConfig access control hardening — currently serves the full config blob to any CA-signed SSL peer (trusted-LAN model). Consider: verify requester is a known --peers controller, rate-limit per peer, minimum_role check (controllers only). (FUTURE_WORK.md §7.)
• lost-and-found drain command — xcecli lost-and-found drain: reassign or cancel sidelined jobs, reset lost_and_found_job_count. Automates operator cleanup after config upgrade. (FUTURE_WORK.md §4.)
• OP_upgradeConfig standby redirect verification — test that a standby returns the primary's address to an admin tool and the tool retries against the primary. Note: redirect semantics change under ha-consistency-invariants Req 1 (no transitional-state redirect); verify against the new behavior. (FUTURE_WORK.md §5.)
• steering sync — ha-async-io.md synchronized to the three-state model + own-term gate (transitional-state row/role removed; heartbeat re-arm PRIMARY-only; PRIMARY no-quorum fact allowlist noted). Done as part of ha-single-leader-role task 6.1.

Superseded / Absorbed Items

• Quorum-ACK gate before config restart (FUTURE_WORK.md §2) — largely superseded by ha-consistency-invariants Requirement 5: configuration changes now require majority acknowledgment before being confirmed/committed, and fail to the requester on timeout (no silent rollback when standbys are offline). Residual to verify in that spec's design: restart sequencing after a committed config change (don't restart the primary before the commit barrier passes).
• Lost-and-found keyword validation (FUTURE_WORK.md §8) — folded into the admin-config-tool backlog item above.

Notes

• Backlog items here are seeds for /kiro-discovery / /kiro-spec-init; promote an item by creating its spec under .kiro/specs/.
• Provenance: items marked (FUTURE_WORK.md §N) originate from the 2026-05-12 config-replication-design.md review, merged here 2026-06-10.