# Brief: component-rejoin-resync

## Problem

When the apiserver or scheduler disconnects and reconnects to the controller (network blip, or reconnect to a new primary after controller failover), each does a full re-join and receives **full** state again (e.g. the whole job list, full queue/resource/config state) rather than a delta. For a large cluster this resend is sizable — bandwidth and re-processing cost scale with cluster size on every reconnect, not just on first join.

This is a **different problem from `worker-rejoin-resync`**: that spec recovers worker-reported job-*execution facts* (start/finish/fail) that would otherwise be silently lost during a disconnect window. This spec is about the *cost/efficiency* of apiserver/scheduler's full-state resend on reconnect — no data is lost today (full resend is correct), it is potentially just expensive at scale.

## Current State

- Full-state rejoin is already accepted, working behavior (`libxce-failover` design review decision): "simple, correct, and still cheaper than a full-history replace with no snapshot" (cf. IBM LSF).
- Recorded as a **deferred YAGNI backlog item** ("lightweight rejoin re-attach", 2026-06-16, roadmap.md Future Specs/Backlog) — explicitly *not* promoted previously because no measured problem existed yet.
- No delta-resync or session-token re-attach mechanism exists for either component today.

## Desired Outcome

Not yet decided — this spec starts at the discussion/investigation stage, not a committed design. Candidate outcome: apiserver and/or scheduler reconnect without re-receiving state that is unchanged since their last known position (e.g. keyed on last-acknowledged committed LSN), reducing reconnect cost at scale — IF a real problem is measured. Equally plausible outcome: investigation confirms full-state resend remains fine and this spec closes as "no action needed."

## Approach

Not yet chosen. Prior design review flagged the general shape (session-token re-attach hello, or delta-only transfer keyed on last-known committed LSN) but flagged delta reconciliation as "non-trivial (the primary must know what each consumer already holds)" — a real design cost, not a rubber stamp. First step for this spec is likely measurement/investigation (does reconnect cost actually hurt at realistic cluster sizes?) before committing to a mechanism.

## Scope

- **In**: apiserver rejoin content/cost. Scheduler rejoin content/cost. Investigation of whether full-state resend is actually a measured problem before designing a fix.
- **Out**: Worker job-execution-fact resync (→ `worker-rejoin-resync`, separate spec, different data shape — facts vs. list-state). Any change to the pending-facts queue. New failover mechanics beyond what `libxce-failover`/`component-failover-adoption` already provide.

## Boundary Candidates

- apiserver reconnect path (Go, `src/frontend/apiserver/backend/clib.go` cgo boundary + libxce failover hook).
- scheduler reconnect path (`src/scheduler/src/handler.c`, controller-side `on_scheduler_join`/`handle_scheduler_join`, `join.c`).
- Measurement/instrumentation to quantify current reconnect cost — likely a prerequisite sub-task before any mechanism design.

## Out of Boundary

- Worker-side job-fact resync (separate spec, `worker-rejoin-resync`).
- Redesigning the join protocol wholesale — this is about reconnect efficiency, not join correctness.

## Upstream / Downstream

- **Upstream**: `libxce-failover` (rejoin hook — implemented). `component-failover-adoption` (daemon rejoin content — currently tasks-generated, in progress) — this spec's scheduler/apiserver reconnect paths overlap with that spec's territory; sequence after or alongside it, not independently duplicating its liveness-safety work.
- **Downstream**: none yet.

## Existing Spec Touchpoints

- **Adjacent**: `worker-rejoin-resync` — sibling, same trigger (reconnect/failover) but disjoint data (job facts vs. list state). Do not merge scope — different information, different mechanism.
- **Adjacent**: `component-failover-adoption` — covers making the daemons *use* libxce failover safely (liveness, master-list config); this spec covers *what* they resync once reconnected, a distinct concern layered on top.
- **Supersedes discussion for**: the "lightweight rejoin re-attach" backlog item (roadmap.md) — this spec is that item promoted, scoped to apiserver+scheduler only (worker's share of that item is now `worker-rejoin-resync`).

## Constraints

- Not started — investigation/discussion first, no committed mechanism yet.
- Should not be pursued as a speculative optimization — prior YAGNI decision stands until this spec's own investigation finds a measured cost, per the same discipline that deferred it originally.