myxce/.kiro/specs/worker-rejoin-resync/brief.md

Brief: worker-rejoin-resync

Problem

When a worker disconnects and reconnects (rejoin after network blip, or reconnect to a new primary after controller failover), any job start/finish/fail fact it sent during the disconnected window — and never got acknowledged — is permanently lost. Worker notify today is fire-and-forget with zero redelivery (bug-396). The controller has no way to ask "what happened to my jobs while you were gone," and the worker has no way to answer.

Evidence: xce_join_req (src/lib/request.c:1398-1429) only attaches hostattrs_t when role == HostRole_worker (joinmsg.data.attrs) — no job-state payload exists on the join wire message at all. handle_worker_join/handle_join_ (src/controller/join.c) process the join purely as a host-attribute registration; there is no code path that reconciles worker-reported job state against the controller's job table on (re)join.

Current State

• worker-join-refactor (implemented) covers only attribute/endpoint update on rejoin — no job-state content.
• The controller-side pending-facts queue (ha-consistency-invariants Req 2.8) covers quorum-loss windows while the worker stays connected — it retries facts the controller received but couldn't commit. It does NOT cover facts that never arrived because the worker was disconnected or the controller itself failed over mid-flight. This spec is the designated backstop for that second case (ha-consistency-invariants Req 2.8 boundary context, elevated 2026-06-11).
• raft-apply-decoupling (implemented) elevated this spec to a hard prerequisite for the facts half of its own work (2026-06-25) — commit-then-apply idempotency guards this spec needs are now in place.

Desired Outcome

On worker rejoin (disconnect/reconnect or failover reconnect), the worker reports its current running + recently-finished job state; the controller reconciles that report against its job table through the existing commit barrier, idempotently (no duplicate facts, no clobbering newer state the controller already has). Facts that would otherwise be silently lost in a disconnect/failover window are recovered.

Approach

Extend the join/hello exchange (or add a post-join follow-up message) so a worker attaches a running+recent-job-state report, analogous to how hostattrs_t already rides the join message today. Controller-side, add a reconciliation path that runs this report through mark_job_*-style idempotent transitions (reusing the commit-then-apply guards raft-apply-decoupling/ha-consistency-invariants already built) rather than a new ad hoc mutation path.

Scope

• In: Worker-side job-state rejoin reporting (running jobs + recently-finished/failed jobs not yet acked); controller-side idempotent reconciliation against the job table via the existing commit barrier; wire schema addition to the join exchange (or a new post-join message type); regression coverage for disconnect-during-job and failover-during-job scenarios.
• Out: apiserver/scheduler resync-on-reconnect (different data — cluster/queue/job list state, not job execution facts — tracked separately, see roadmap's deferred "lightweight rejoin re-attach" item, YAGNI until measured). Delta/incremental state transfer optimizations (session-token re-attach) — separate, deferred backlog item, not this spec's concern. Any change to the primary-side pending-facts queue itself (already implemented, this spec only complements it).

Boundary Candidates

• Wire/schema: extending xce_join_req/join message (or new message type) to carry job-state report — worker + controller.capnp touchpoint.
• Worker-side: collecting "running + recently-finished-but-unacked" job state at rejoin time (worker.c / controller_connector.c).
• Controller-side: reconciliation logic reusing mark_job_update/mark_job_fail-style idempotent commit-then-apply transitions (job.c).

Out of Boundary

• apiserver rejoin content (full state resend — separate, deferred item).
• scheduler rejoin content (separate, deferred item).
• Delta-based / session-token re-attach optimization (separate backlog item, explicit YAGNI until measured).

Upstream / Downstream

• Upstream (hard deps): ha-consistency-invariants (commit barrier + idempotency guards — implemented). raft-apply-decoupling (commit-then-apply seam this reuses — implemented, and the thing that elevated this spec to hard-prereq status for its own facts half).
• Downstream: job-array-single-wal-record and the broader Raft Apply-Ordering family reference this as a complements-not-replaces sibling to the pending-facts queue. The deferred "lightweight rejoin re-attach" backlog item lists this as one of its own dependencies (worker job-state content).

Existing Spec Touchpoints

• Extends: worker-join-refactor (implemented — attribute/endpoint update only; this adds the job-state content that was explicitly out of that spec's scope).
• Adjacent: ha-consistency-invariants (Req 2.8 designates this as the official backstop — do not duplicate the pending-facts queue's mechanism, complement it). component-failover-adoption (in-progress sibling — makes daemons exercise libxce failover; does not itself cover job-state resync content).

Constraints

• Must not introduce a second, competing idempotency mechanism — reconciliation goes through the same commit-then-apply guards the pending-facts queue and raft-apply-decoupling already use.
• Must not regress worker-join-refactor's existing attribute/endpoint update behavior on rejoin.