docs(kiro): reconcile spec metadata, promote 3 new specs, fix template path

- config-upgrade-snapshot, raft-adapter-conformance: correct stale
  phase metadata (both showed "tasks-generated" though every task was
  already complete) and record their validate-impl GO decisions with
  known-limitations notes (bug-1471/bug-1472-remainder, bug-1474
  angle-b) for future spec ownership.
- component-failover-adoption: catch up design/requirements/research
  docs to the Phase 6 fix (bug-1548/bug-1549, commit d479b707) that
  landed after this spec was already marked closed.
- Promote 3 roadmap backlog items to specs (2026-08-10 discovery):
  component-rejoin-resync (apiserver/scheduler rejoin resync),
  worker-rejoin-resync (worker job-fact resync, hard prereq for
  raft-apply-decoupling's facts half), scheduler-sim-diagnostics
  (open-ended scheduler REPL debug/troubleshoot capability).
- roadmap.md updated to reflect all of the above.
- kiro-steering/kiro-steering-custom SKILL.md: fix stale template path
  (templates/steering/ -> .kiro/settings/templates/steering/) plus
  minor doc corrections in the other kiro-spec-* skills.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

2026-08-16 11:02:06 +00:00

4.7 KiB

Raw Blame History

Brief: component-rejoin-resync

Problem

When the apiserver or scheduler disconnects and reconnects to the controller (network blip, or reconnect to a new primary after controller failover), each does a full re-join and receives full state again (e.g. the whole job list, full queue/resource/config state) rather than a delta. For a large cluster this resend is sizable — bandwidth and re-processing cost scale with cluster size on every reconnect, not just on first join.

This is a different problem from worker-rejoin-resync: that spec recovers worker-reported job-execution facts (start/finish/fail) that would otherwise be silently lost during a disconnect window. This spec is about the cost/efficiency of apiserver/scheduler's full-state resend on reconnect — no data is lost today (full resend is correct), it is potentially just expensive at scale.

Current State

Full-state rejoin is already accepted, working behavior (libxce-failover design review decision): "simple, correct, and still cheaper than a full-history replace with no snapshot" (cf. IBM LSF).
Recorded as a deferred YAGNI backlog item ("lightweight rejoin re-attach", 2026-06-16, roadmap.md Future Specs/Backlog) — explicitly not promoted previously because no measured problem existed yet.
No delta-resync or session-token re-attach mechanism exists for either component today.

Desired Outcome

Not yet decided — this spec starts at the discussion/investigation stage, not a committed design. Candidate outcome: apiserver and/or scheduler reconnect without re-receiving state that is unchanged since their last known position (e.g. keyed on last-acknowledged committed LSN), reducing reconnect cost at scale — IF a real problem is measured. Equally plausible outcome: investigation confirms full-state resend remains fine and this spec closes as "no action needed."

Approach

Not yet chosen. Prior design review flagged the general shape (session-token re-attach hello, or delta-only transfer keyed on last-known committed LSN) but flagged delta reconciliation as "non-trivial (the primary must know what each consumer already holds)" — a real design cost, not a rubber stamp. First step for this spec is likely measurement/investigation (does reconnect cost actually hurt at realistic cluster sizes?) before committing to a mechanism.

Scope

In: apiserver rejoin content/cost. Scheduler rejoin content/cost. Investigation of whether full-state resend is actually a measured problem before designing a fix.
Out: Worker job-execution-fact resync (→ worker-rejoin-resync, separate spec, different data shape — facts vs. list-state). Any change to the pending-facts queue. New failover mechanics beyond what libxce-failover/component-failover-adoption already provide.

Boundary Candidates

apiserver reconnect path (Go, src/frontend/apiserver/backend/clib.go cgo boundary + libxce failover hook).
scheduler reconnect path (src/scheduler/src/handler.c, controller-side on_scheduler_join/handle_scheduler_join, join.c).
Measurement/instrumentation to quantify current reconnect cost — likely a prerequisite sub-task before any mechanism design.

Out of Boundary

Worker-side job-fact resync (separate spec, worker-rejoin-resync).
Redesigning the join protocol wholesale — this is about reconnect efficiency, not join correctness.

Upstream / Downstream

Upstream: libxce-failover (rejoin hook — implemented). component-failover-adoption (daemon rejoin content — currently tasks-generated, in progress) — this spec's scheduler/apiserver reconnect paths overlap with that spec's territory; sequence after or alongside it, not independently duplicating its liveness-safety work.
Downstream: none yet.

Existing Spec Touchpoints

Adjacent: worker-rejoin-resync — sibling, same trigger (reconnect/failover) but disjoint data (job facts vs. list state). Do not merge scope — different information, different mechanism.
Adjacent: component-failover-adoption — covers making the daemons use libxce failover safely (liveness, master-list config); this spec covers what they resync once reconnected, a distinct concern layered on top.
Supersedes discussion for: the "lightweight rejoin re-attach" backlog item (roadmap.md) — this spec is that item promoted, scoped to apiserver+scheduler only (worker's share of that item is now worker-rejoin-resync).

Constraints

Not started — investigation/discussion first, no committed mechanism yet.
Should not be pursued as a speculative optimization — prior YAGNI decision stands until this spec's own investigation finds a measured cost, per the same discipline that deferred it originally.

4.7 KiB Raw Blame History