Rongsong Shen
ce0ee78b0f
- config-upgrade-snapshot, raft-adapter-conformance: correct stale
phase metadata (both showed "tasks-generated" though every task was
already complete) and record their validate-impl GO decisions with
known-limitations notes (bug-1471/bug-1472-remainder, bug-1474
angle-b) for future spec ownership.
- component-failover-adoption: catch up design/requirements/research
docs to the Phase 6 fix (bug-1548/bug-1549, commit d479b707) that
landed after this spec was already marked closed.
- Promote 3 roadmap backlog items to specs (2026-08-10 discovery):
component-rejoin-resync (apiserver/scheduler rejoin resync),
worker-rejoin-resync (worker job-fact resync, hard prereq for
raft-apply-decoupling's facts half), scheduler-sim-diagnostics
(open-ended scheduler REPL debug/troubleshoot capability).
- roadmap.md updated to reflect all of the above.
- kiro-steering/kiro-steering-custom SKILL.md: fix stale template path
(templates/steering/ -> .kiro/settings/templates/steering/) plus
minor doc corrections in the other kiro-spec-* skills.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
53 lines
4.7 KiB
Markdown
53 lines
4.7 KiB
Markdown
# Brief: component-rejoin-resync
|
|
|
|
## Problem
|
|
|
|
When the apiserver or scheduler disconnects and reconnects to the controller (network blip, or reconnect to a new primary after controller failover), each does a full re-join and receives **full** state again (e.g. the whole job list, full queue/resource/config state) rather than a delta. For a large cluster this resend is sizable — bandwidth and re-processing cost scale with cluster size on every reconnect, not just on first join.
|
|
|
|
This is a **different problem from `worker-rejoin-resync`**: that spec recovers worker-reported job-*execution facts* (start/finish/fail) that would otherwise be silently lost during a disconnect window. This spec is about the *cost/efficiency* of apiserver/scheduler's full-state resend on reconnect — no data is lost today (full resend is correct), it is potentially just expensive at scale.
|
|
|
|
## Current State
|
|
|
|
- Full-state rejoin is already accepted, working behavior (`libxce-failover` design review decision): "simple, correct, and still cheaper than a full-history replace with no snapshot" (cf. IBM LSF).
|
|
- Recorded as a **deferred YAGNI backlog item** ("lightweight rejoin re-attach", 2026-06-16, roadmap.md Future Specs/Backlog) — explicitly *not* promoted previously because no measured problem existed yet.
|
|
- No delta-resync or session-token re-attach mechanism exists for either component today.
|
|
|
|
## Desired Outcome
|
|
|
|
Not yet decided — this spec starts at the discussion/investigation stage, not a committed design. Candidate outcome: apiserver and/or scheduler reconnect without re-receiving state that is unchanged since their last known position (e.g. keyed on last-acknowledged committed LSN), reducing reconnect cost at scale — IF a real problem is measured. Equally plausible outcome: investigation confirms full-state resend remains fine and this spec closes as "no action needed."
|
|
|
|
## Approach
|
|
|
|
Not yet chosen. Prior design review flagged the general shape (session-token re-attach hello, or delta-only transfer keyed on last-known committed LSN) but flagged delta reconciliation as "non-trivial (the primary must know what each consumer already holds)" — a real design cost, not a rubber stamp. First step for this spec is likely measurement/investigation (does reconnect cost actually hurt at realistic cluster sizes?) before committing to a mechanism.
|
|
|
|
## Scope
|
|
|
|
- **In**: apiserver rejoin content/cost. Scheduler rejoin content/cost. Investigation of whether full-state resend is actually a measured problem before designing a fix.
|
|
- **Out**: Worker job-execution-fact resync (→ `worker-rejoin-resync`, separate spec, different data shape — facts vs. list-state). Any change to the pending-facts queue. New failover mechanics beyond what `libxce-failover`/`component-failover-adoption` already provide.
|
|
|
|
## Boundary Candidates
|
|
|
|
- apiserver reconnect path (Go, `src/frontend/apiserver/backend/clib.go` cgo boundary + libxce failover hook).
|
|
- scheduler reconnect path (`src/scheduler/src/handler.c`, controller-side `on_scheduler_join`/`handle_scheduler_join`, `join.c`).
|
|
- Measurement/instrumentation to quantify current reconnect cost — likely a prerequisite sub-task before any mechanism design.
|
|
|
|
## Out of Boundary
|
|
|
|
- Worker-side job-fact resync (separate spec, `worker-rejoin-resync`).
|
|
- Redesigning the join protocol wholesale — this is about reconnect efficiency, not join correctness.
|
|
|
|
## Upstream / Downstream
|
|
|
|
- **Upstream**: `libxce-failover` (rejoin hook — implemented). `component-failover-adoption` (daemon rejoin content — currently tasks-generated, in progress) — this spec's scheduler/apiserver reconnect paths overlap with that spec's territory; sequence after or alongside it, not independently duplicating its liveness-safety work.
|
|
- **Downstream**: none yet.
|
|
|
|
## Existing Spec Touchpoints
|
|
|
|
- **Adjacent**: `worker-rejoin-resync` — sibling, same trigger (reconnect/failover) but disjoint data (job facts vs. list state). Do not merge scope — different information, different mechanism.
|
|
- **Adjacent**: `component-failover-adoption` — covers making the daemons *use* libxce failover safely (liveness, master-list config); this spec covers *what* they resync once reconnected, a distinct concern layered on top.
|
|
- **Supersedes discussion for**: the "lightweight rejoin re-attach" backlog item (roadmap.md) — this spec is that item promoted, scoped to apiserver+scheduler only (worker's share of that item is now `worker-rejoin-resync`).
|
|
|
|
## Constraints
|
|
|
|
- Not started — investigation/discussion first, no committed mechanism yet.
|
|
- Should not be pursued as a speculative optimization — prior YAGNI decision stands until this spec's own investigation finds a measured cost, per the same discipline that deferred it originally.
|