myxce/.kiro/specs/worker-rejoin-resync/design.md

Design Document: worker-rejoin-resync

Overview

On every worker (re)join — a fresh network reconnect after a disconnect, or a reconnect to a new primary after controller failover — this feature has the controller ask the worker directly what really happened to every job the controller still believes is running there, and reconciles the worker's answer into the job table through the existing idempotent commit-then-apply path. This closes the gap left by fire-and-forget job start/finish/fail notifications (XCECLI_FF_CALL): a report lost to a disconnect or failover window today leaves a job stuck in RUN forever, with nothing in the cluster ever correcting it (bug-1741, bug-396).

Users: Internal controller/worker/scheduler subsystems — there is no end-user-facing surface. Cluster operators benefit indirectly: jobs self-heal out of a stuck-RUN state without manual intervention.

Impact: Extends rejoin_worker/apply_join_state (worker-join-refactor, implemented) with a job-state reconciliation round; adds one new opcode pair to the wire protocol; adds a short-lived finished-job retention to the worker daemon (xcecd), which retains none today; adds two narrow new fact-recording functions to the controller's job table, reusing its existing idempotency guards for everything else; adds two new shared-config fields.

Goals

• Trigger reconciliation on every worker join where the controller holds non-terminal jobs for that host, without needing to distinguish first-join from reconnect (Requirement 1).
• Query only the jobs in question; let the worker answer definitively for each, plus report anything running/recently-finished the controller didn't ask about (Requirements 2, 3).
• Apply every reconciled outcome through the existing idempotent commit-then-apply path — no second mutation mechanism (Requirement 4).
• Withhold new dispatch to a worker while its reconciliation is outstanding, and fail fast (not queue) a dispatch to a disconnected worker (Requirement 5).
• Stay safe under repeated/interrupted rejoin (Requirement 6) and reach the scheduler through the same notification path as any ordinary job-state change (Requirement 7).
• Work regardless of how many jobs a host is running, via a bounded/paginated exchange (Requirement 8).

Non-Goals

• apiserver/scheduler cluster/queue/job-list resync on reconnect (component-rejoin-resync, a disjoint sibling spec — different data shape, not merged here).
• Any change to the primary-side pending-facts queue (the commit engine) itself — this feature is an additional caller into it, never a modification of it.
• Delta-based / session-token re-attach optimization for rejoin in general (separate, deferred, explicit YAGNI).
• Coupling the Requirement 3.7 reporting window to commit-batch-window's (separately parked) commit-urgency tuning — disclaimed in requirements.md's Boundary Context and not referenced anywhere in this design.
• Recovering a reconciliation-assigned job's original resource allocation (see Data Models — this is a real, explicitly-documented limitation, not a silent gap).

---

Boundary Commitments

This Spec Owns

• The wire protocol for the reconciliation exchange: two new opcodes, OPCode_jobStateQuery (controller → worker) and OPCode_jobStateReport (worker → controller), both paginated.
• Controller-side: deciding when to trigger reconciliation (Requirement 1); tracking one reconciliation round per worker (queried job IDs, round token, deadline); applying every reported outcome via the job table's existing idempotent fact path, plus two new narrow fact-recording functions for the two outcomes with no existing analog (reconcile-to-pending, reconcile-assign); gating new dispatch on reconciliation-outstanding state; notifying the scheduler through the existing inbound notify path. Reconciliation is modeled as extra steps that run strictly after the worker-join RPC (xce_join/xcecli_join) has already completed — never as an extension of that call's own blocking wait (see Out of Boundary).
• Worker-side: a new short-lived retention of recently-finished job outcomes (jobid, exit status, real finish time), bounded by a configurable window; answering the controller's query from that retention plus the still-running job list; reporting extra jobs outside the query; a new local gate flag that delays the worker's own resumption of normal post-join operation until its report has finished sending (Requirement 3.8).
• Two new shared-config fields (the Requirement 3.7 reporting window; the Requirement 5.3 reconciliation timeout).

Out of Boundary

• apiserver/scheduler resync-on-reconnect content (component-rejoin-resync).
• The commit engine / pending-facts queue mechanism itself (commit.c) — used, never modified.
• Delta/session-token rejoin optimization.
• commit-batch-window's commit-urgency tuning — no dependency, no reference.
• Recovering job->allocs (the scheduler's resource allocation) for a job reconciled via Requirement 4.8 that the controller never held a dispatch record for — the data was never known to this controller and cannot be reconstructed from the worker's report. Converges via the existing reserved/actual accounting policy on the job's next rusage report (see Data Models), same as any legitimately-unreserved job today.
• Fixing WAL_JOB_DISPATCH's own pre-existing, separately-tracked job->allocs field-completeness gap (persistence.md's APPLY_DIVERGENT section; roadmap.md's raft-apply-decoupling item #4) — this design does not make that gap worse, but does not fix it either.
• Extending xce_join()/xcecli_join()'s own blocking wait (cbuf_deadline_10s, xceapi.c) to also cover config push or job-state reconciliation. That call is a shared, role-agnostic API (worker, apiserver, scheduler, and xce_test/xceadmin clients all use it) and already returns as soon as the join reply arrives — well before on_worker_join (config push) even runs, unchanged by this feature. Reconciliation has its own, independent, server-side timeout (resync.c's delaymgr trigger against workerResyncTimeoutMs) and is not gated on, or gating of, the join RPC's own deadline in any way.

Allowed Dependencies

• rejoin_worker / apply_join_state (worker-join-refactor, implemented) — the plug-in point for triggering reconciliation; this design must not regress its existing attribute/endpoint-update/admin-bit-preservation behavior.
• The job table's commit-then-apply seam (record_job_*_fact, apply_job_*_transition, job_state_is_terminal, job_state_is_backward, apply_committed_entry) — reused, never duplicated.
• XCE_CONTROLLER_HOOK_JOB fan-out, sched.c's hook callbacks, and notify() (dataex.c:399) — the existing controller→scheduler notify path.
• xce_reply_get/xce_push_data_req's {pos, count, n_total} pagination idiom (request.c) — the template for the new opcode pair.
• commit.c's COMMIT_WAIT_RESERVED admission-reservation pattern — the template for any bounded wait on the worker's report.
• config-design.md's shared-config classification rule, applied to the two new config fields.

Revalidation Triggers

• Any change to worker_t.host.status bitmask semantics or dispatch_a_job_'s connected-state check (job.c:1511-1512) — the dispatch-gating reuse (Requirement 5.1) depends on both staying exactly as observed.
• Any change to apply_job_update_transition / job_state_is_backward's RUN↔PEND handling.
• Any change to the XCE_CONTROLLER_HOOK_JOB fan-out shape or the NotifyData capnp union.
• Any change to JoinData's capnp union arms (this design deliberately does not touch it, but a future change there should re-check this design's choice to use a separate opcode pair instead).
• component-rejoin-resync's eventual design, if it ever proposes touching the join/rejoin wire exchange mechanics this design also touches.

---

Architecture

Existing Architecture Analysis

rejoin_worker/apply_join_state (worker.c:403, :321) already run under the per-worker write lock at the exact moment w->ep is set to the new live endpoint and XCE_WORKER_UNAVAIL is cleared — this is the moment the host becomes "connected" again, and the natural trigger point for Requirement 1's "does the controller hold non-terminal jobs for this host" check. handle_worker_join (join.c:48) commits the join durably (persist_commit_barrier) before on_worker_join runs — a second, later point, after the join itself can no longer be rolled back, where the reconciliation query can be sent.

The job table's existing commit-then-apply split is the load-bearing idempotency mechanism this feature must reuse (per requirements.md's Boundary Context): record_job_*_fact() functions append a WAL fact (durable, no mutation), and apply_job_*_transition() functions are the guarded mutation core, invoked both as a peek at receipt time and later by apply_committed_entry once the fact commits. job_state_is_terminal/job_state_is_backward reject duplicate or backward (terminal→non-terminal) transitions — this is the exact "no duplicate facts, no clobbering newer state" guarantee Requirement 4 asks for, already built and tested (test_guard_replay_finish_equals_single_application).

Dispatch gating (dispatch_a_job_, job.c:1278) already refuses to dispatch when w->ep == NULL || w->host.status != 0 — any nonzero status bit blocks it. A new bit reuses this check with zero changes to dispatch_a_job_ itself.

The controller→scheduler notification path is a hook→notify→wire→dequeue pipeline: job-state-change code fires XCE_CONTROLLER_HOOK_JOB (job.c), sched.c's registered callback (sched_job_hook_fun) dispatches by HOOK_DETAIL_* to sched_job_finish_fun/sched_job_fail_fun/sched_job_update_fun, each of which calls the shared notify() (dataex.c:399) with a distinct NotifyData_data_sched* event over c->sched.ep. On the scheduler side, sched_handle_notify() (handler.c:525) is the inbound receipt switch that calls sched_job_state_change(); this is architecturally distinct from scheduler.c/driver.c's own outbound decision-publish direction, which a standing test (test_no_worker_resync_symbols_in_scheduler_publish_path) mechanically guards against ever referencing "resync." This fixes the only permitted integration seam: handler.c's inbound switch.

The worker daemon retains no finished-job state today: cleanup_worker_list() (xcecd_worker_utils.c:385) deletes both the in-memory entry and its sqlite row on the pass immediately after a job exits, and neither xcecd_worker_t nor xcecd_job_t carries a timestamp field. JoinData's capnp union (nodata/attrs) carries no job content and cannot be extended in place without forcing the exchange through xce_join()'s single-shot synchronous reply, which cannot paginate.

Architecture Pattern & Boundary Map

graph TB
    subgraph Controller
        JC[handle_worker_join / join.c]
        RW[rejoin_worker / apply_join_state / worker.c]
        RS[Reconciliation state: worker_t.resync]
        RQ[Reconciliation query+timeout / new resync.c]
        JOBC[job.c: record_job_finish_fact / record_job_fail_fact<br/>record_job_reconcile_pending_fact NEW<br/>record_job_reconcile_assign_fact NEW]
        COMMIT[commit.c / apply_committed_entry — unchanged, reused]
        HOOK[XCE_CONTROLLER_HOOK_JOB fan-out]
        SCHEDC[sched.c hook callbacks]
        NOTIFY[notify / dataex.c]
    end
    subgraph Worker[xcecd]
        WJOIN[Join / rejoin]
        RETAIN[NEW: finished-job retention]
        RUNNING[Existing: xcecd_worker_t running list]
        QHANDLE[NEW: job-state query handler]
    end
    subgraph Scheduler
        HANDLER[handler.c sched_handle_notify calls sched_job_state_change]
        PUBLISH[scheduler.c / driver.c: outbound decision publish — NOT touched]
    end

    JC --> RW --> RS
    RS -->|non-terminal jobs for host?| RQ
    RQ -->|OPCode_jobStateQuery, paginated| WJOIN
    WJOIN --> QHANDLE
    QHANDLE --> RETAIN
    QHANDLE --> RUNNING
    QHANDLE -->|OPCode_jobStateReport, paginated| RQ
    RQ --> JOBC
    JOBC --> COMMIT --> HOOK --> SCHEDC --> NOTIFY --> HANDLER
    HANDLER -.->|never referenced from| PUBLISH

Architecture Integration:

• Selected pattern: additive extension of an existing daemon-to-daemon RPC + existing fact/apply pipeline — no new subsystem, no new process.
• Domain/feature boundaries: wire protocol (new opcode pair) / controller reconciliation bookkeeping (new, small) / job-table fact recording (two new functions, rest reused) / worker retention (new) / scheduler notify (zero new code — reused as-is).
• Existing patterns preserved: commit-then-apply idempotency, {pos,count,n_total} pagination, hook→notify scheduler path, per-worker write-lock discipline.
• New components rationale: reconciliation bookkeeping and worker retention have no existing analog to extend; everything else is reuse.
• Steering compliance: async-I/O thread-role contract (bounded wait via admission reservation, not raw blocking); persistence per-field completeness review (see Data Models); config-design.md classification (see Data Models).

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
Wire protocol	Cap'n Proto, src/schema/controller.capnp	New OPCode_jobStateQuery/OPCode_jobStateReport, new payload structs	Mirrors JobData/WorkerData/ResourceVals's existing {ntotal,pos,count} shape
Controller	C (gnu17), src/controller/	Reconciliation trigger, bookkeeping, fact recording, notify	worker.c, job.c, new resync.c, cluster.c (opcode dispatch)
Worker daemon	C (gnu17), src/worker/	Finished-job retention, query handling	xcecd_worker_utils.c/h, new xcecd_resync.c
Async I/O	iomgr/vtaskmgr/delaymgr, src/lib/	Non-blocking query send/receive; bounded wait via admission reservation	Reuses commit.c's COMMIT_WAIT_RESERVED pattern
Shared config	Cap'n Proto ClusterConfig	Reporting window (Req 3.7); reconciliation timeout (Req 5.3)	Alongside existing heartbeatInterval/inactiveHeartbeatCount

---

File Structure Plan

Modified Files

• src/schema/controller.capnp — add OPCode_jobStateQuery, OPCode_jobStateReport to the OPCode enum; add JobStateQuery{jobids: List(UInt64), pos, count, ntotal}, JobStateReport{entries: List(JobStateEntry), pos, count, ntotal}, JobStateEntry{jobid, status: JobStateStatus, exitCode, finishTimeMs, queried: Bool} (queried=false marks a Requirement 3.6 extra job); add both as Request/Reply union arms; add workerResyncWindowMs/workerResyncTimeoutMs fields to ClusterConfig.
• src/controller/worker.c — rejoin_worker/apply_join_state: after clearing XCE_WORKER_UNAVAIL, check for non-terminal jobs assigned to this host (Requirement 1) and, if any, set XCE_WORKER_RECONCILING and initialize the per-worker reconciliation record before releasing the write lock. No change to admin-bit preservation or attribute/endpoint update behavior. on_worker_join (worker.c:623) — extended, not handle_worker_join: it already calls worker_push_config as its one existing step (the config the worker needs — hosttype/hostmodel/resource variables — is foundational baseline state, and its own stub comment, /* consider to push jobs on hosts to */, already anticipates this exact follow-on). The reconciliation query send is added right after it, in that fixed order (config push, then query) — both sent fire-and-forget, no new ack/reply invented for this. This ordering is safe, not just conventional: the worker's incoming-request dispatch (myreq_handler, controller_connector.c:1563) is a plain synchronous switch invoked one message at a time off its single long-lived client connection (xcecli_ctx) — confirmed by reading it, not assumed — so the two requests are not just sent in order, they are handled to completion in that same order on the worker side; no thread-pool fan-out is in the path (the worker's own thread_pool_t is for job-forking work, unrelated to request dispatch). The actual wait in this design belongs at the end of the sequence, not in the middle: the controller does not consider the worker's rejoin/reconciliation complete until the job-state report arrives (or the Requirement 5.3 timeout fires) — see Error Handling.
• src/controller/join.c — no change. handle_worker_join itself is untouched; the reconciliation query is sent from the (extended) on_worker_join, which handle_worker_join already calls unconditionally after the join reply.
• src/controller/job.c — two new functions: record_job_reconcile_pending_fact() (Requirement 4.6, routes through the existing WAL_JOB_UPDATE/apply_job_update_transition seam), record_job_reconcile_assign_fact() (Requirement 4.8, mirrors dispatch_a_job_'s assignment logic for host/state only — see Data Models for the allocation limitation). Existing record_job_finish_fact/record_job_fail_fact are called as-is for the finished-success/finished-failure cases.
• src/controller/cluster.c — new case OPCode_jobStateReport: in myreq_handler's dispatch switch (runs after ha_dispatch_gate); route to the new reconciliation-apply logic.
• src/include/xce/types.h — add XCE_WORKER_RECONCILING to the worker status bitmask; add a resync_state_t (queried job-ID set, round token, deadline) referenced from worker_t.
• src/lib/request.c — xce_job_state_query_req()/xce_job_state_report_req() encode/decode helpers, following xce_reply_get/xce_push_data_req's pagination loop shape.

New Files

• src/controller/resync.c (+ src/controller/include/resync.h) — per-worker reconciliation bookkeeping: arm/clear XCE_WORKER_RECONCILING, track queried job IDs and round token (Requirement 6.3's late-report discard), the delaymgr-triggered timeout (Requirement 5.3), and the glue that maps each JobStateEntry outcome to the correct job.c call.
• src/worker/src/xcecd_resync.c (+ header) — handles incoming OPCode_jobStateQuery: looks up each queried job ID against the running list (xcecd_worker_t) and the new finished-job retention; builds and sends paginated OPCode_jobStateReport replies, followed by extra-job pages (Requirement 3.6).
• src/controller/testsuite/test_worker_resync.c — controller-side unit tests (trigger, query construction, per-outcome fact application, dispatch gating, timeout, discard-on-disconnect), following test_worker_join.c's fixture conventions.
• src/worker/testsuite/ — new directory; src/worker has no testsuite/ today (per structure.md/testing.md). This feature is the first worker-side C test coverage; add test_xcecd_resync.c and the minimal harness scaffolding it needs (mirroring controller/testsuite/test.c's runner pattern).

Modified Worker Files

• src/worker/include/xcecd_worker_utils.h / src/worker/src/xcecd_worker_utils.c — add a finished-job retention structure (jobid, exit status, real finish time, retain-until) populated at job exit before cleanup_worker_list() would otherwise reap the entry; cleanup_worker_list() retains an entry in this new structure instead of discarding all trace of it, until the configured window elapses.

---

System Flows

sequenceDiagram
    participant W as Worker
    participant JC as ControllerJoin
    participant WC as ControllerWorker
    participant R as ControllerResync
    participant JOBC as ControllerJob
    participant SCHED as Scheduler

    W->>JC: OPCode_join (reconnect or failover reconnect)
    JC->>WC: rejoin_worker(...)
    WC->>WC: clear UNAVAIL, set ep (per-worker WRITE lock)
    WC->>R: non-terminal jobs assigned to this host?
    alt none
        R-->>WC: no reconciliation needed
    else one or more
        R->>WC: arm XCE_WORKER_RECONCILING, init round token + deadline
    end
    WC-->>JC: idx
    JC->>JC: persist_commit_barrier (join durable)
    JC->>W: xce_reply (join success reply, unchanged)
    JC->>JC: on_worker_join
    JC->>W: OPCode_push (config push, unchanged worker_push_config call)
    JC->>W: OPCode_jobStateQuery (NEW — sent right after, same on_worker_join)
    Note over W: single connection, synchronous per-message dispatch<br/>(myreq_handler in controller_connector.c line 1563) — config push<br/>is handled to completion before the query is even read as<br/>a distinct message, no ack or reply needed to enforce this

    loop each query page
        W->>JC: OPCode_jobStateReport (queried=true entries)
    end
    W->>JC: OPCode_jobStateReport (queried=false entries — extras, Req 3.6)
    Note over W: worker holds off entering its normal post-join operating<br/>mode (resuming periodic background reporting) until its full<br/>report has finished sending (Req 3.8) — a new gate flag,<br/>mirroring the existing needs_rejoin pattern in<br/>controller_connector.c, not a new blocking wait

    loop each reported entry
        alt finished success
            JC->>JOBC: record_job_finish_fact (existing, reused)
        else finished failure
            JC->>JOBC: record_job_fail_fact (existing, reused)
        else still running
            JC->>R: clear this jobid from outstanding set (no-op fact)
        else unrecognized
            JC->>JOBC: record_job_reconcile_pending_fact (NEW)
            JOBC->>SCHED: notify scheduler of new PEND state (Req 7.2) via the same HOOK_DETAIL_JOB_UPDATE path
        else unqueried extra job
            JC->>JOBC: record_job_reconcile_assign_fact (NEW)
        end
        JOBC->>JOBC: commit-then-apply, existing guards, unchanged
        JOBC->>SCHED: existing HOOK_DETAIL_JOB event notify path, unchanged
    end

    R->>WC: outstanding set empty? clear XCE_WORKER_RECONCILING
    Note over R: if deadline fires first (delaymgr trigger),<br/>resolve outstanding as answered/timed out (Req 5.3)<br/>rather than withholding dispatch indefinitely

Gating conditions: dispatch_a_job_ already refuses dispatch whenever w->host.status != 0 (job.c:1511) — XCE_WORKER_RECONCILING reuses this unchanged. A dispatch attempt to a disconnected worker (w->ep == NULL) fails immediately and returns the job to pending (Requirement 5.4) via the same new reconcile-to-pending fact path used for an unrecognized job, rather than the existing retry-then-terminal-fail branch (whose retry semantics — retry the dispatch itself — don't fit "the worker isn't even connected").

Discard-on-disconnect (Requirement 6): if the worker disconnects before its report is fully processed, rejoin_worker's next invocation (on the next join) finds the prior round's token stale and starts a fresh one; any report that later arrives carrying the stale token is discarded (Requirement 6.3) rather than applied against the wrong round.

---

Requirements Traceability

Requirement	Summary	Components	Interfaces	Flows
1.1, 1.2, 1.3	Trigger reconciliation iff non-terminal jobs exist for the host, regardless of join type	rejoin_worker, resync.c	trigger check	Sequence: rejoin → trigger
2.1, 2.2	Query lists exactly the non-terminal jobs by unique ID	resync.c, wire JobStateQuery	xce_job_state_query_req	Sequence: OPCode_jobStateQuery
3.1–3.5	Worker reports definitive status + real finish time per queried job	worker xcecd_resync.c, retention	JobStateEntry	Sequence: report loop
3.6	Worker reports extra unqueried jobs	worker xcecd_resync.c	JobStateEntry.queried=false	Sequence: extras page
3.7	Bounded, cluster-configurable reporting window	ClusterConfig.workerResyncWindowMs, worker retention	config field	Data Models
3.8	Worker delays resuming normal post-join operation until its report has finished sending	worker xcecd_resync.c, new gate flag	resync-in-progress flag	Sequence: post-report note
4.1–4.5	Apply via existing idempotent commit-then-apply; no-op duplicates; never clobber newer/terminal state	job.c (existing fns), apply_job_*_transition	existing	Sequence: apply per outcome
4.6	Unrecognized → return to pending, eligible for scheduling	record_job_reconcile_pending_fact (NEW)	new fn	Sequence: unrecognized branch
4.7, 4.8	Reconcile jobs outside the query; assign if no existing non-terminal record	record_job_reconcile_assign_fact (NEW)	new fn	Sequence: extras branch; Data Models limitation
5.1, 5.2	Gate dispatch while reconciliation outstanding; restore eligibility on completion	XCE_WORKER_RECONCILING bit, dispatch_a_job_ (reused)	status bitmask	Architecture: dispatch gating reuse
5.3	Bounded timeout resolves outstanding reconciliation	resync.c delaymgr trigger, ClusterConfig.workerResyncTimeoutMs	timeout	Sequence: deadline note
5.4	Dispatch to disconnected worker fails fast, returns to pending	new dispatch-fail classification, record_job_reconcile_pending_fact (NEW, reused)	—	Error Handling
6.1–6.3	Disconnect mid-round discards and re-triggers; idempotent re-application; stale-round reports discarded	resync.c round token	round token	Sequence: discard-on-disconnect note
7.1–7.3	Reconciled changes reach the scheduler via the existing notify path	XCE_CONTROLLER_HOOK_JOB, sched.c, handler.c (all reused, unchanged)	notify()	Sequence: apply → notify
8.1–8.3	Works for any job count; paginated multi-message exchange	JobStateQuery/JobStateReport pagination	{pos,count,ntotal}	Sequence: loop each query/report page

---

Components and Interfaces

Component	Domain/Layer	Intent	Req Coverage	Key Dependencies	Contracts
resync.c (new)	Controller	Per-worker reconciliation bookkeeping + timeout	1, 5.3, 6	worker.c (P0), commit.c admission pattern (P1)	State, Batch
job.c additions	Controller	Two new idempotent fact-recording functions	4.6, 4.8, 5.4	existing commit-then-apply seam (P0)	Service
xcecd_resync.c (new)	Worker	Answer job-state query from running list + retention	3, 8	xcecd_worker_utils.c (P0)	Batch, Event
xcecd_worker_utils.c additions	Worker	Short-lived finished-job retention	3.5, 3.7	none new	State
Wire opcode pair	Protocol	Paginated query/report exchange	2, 3, 8	xce_reply_get/xce_push_data_req pattern (P0)	API/Batch

Controller: resync.c

Field	Detail
Intent	Own the reconciliation-outstanding lifecycle for one worker: arm on trigger, track queried IDs + round token, resolve on completion or timeout
Requirements	1.1–1.3, 5.1–5.3, 6.1–6.3

Responsibilities & Constraints

• Called from rejoin_worker/apply_join_state under the per-worker write lock already held there — must not introduce a new lock ordering.
• Owns the round token that distinguishes the current reconciliation attempt from a stale/superseded one (Requirement 6.3).
• Schedules the Requirement 5.3 timeout via a delaymgr one-shot trigger (non-blocking fire, per the async-I/O thread-role contract) — never a blocking wait on iomgr or delaymgr.

Dependencies

• Inbound: worker.c (P0) — trigger call site
• Outbound: job.c (P0) — per-outcome fact application
• Outbound: commit.c's admission-reservation pattern (P1) — template for any bounded wait, if a synchronous variant is ever needed instead of the pure delaymgr-trigger design above

Contracts: Batch [x] / State [x]

Batch / Job Contract

• Trigger: rejoin_worker finds ≥1 non-terminal job for the rejoining host
• Input / validation: the controller's own non-terminal job set for that host (Requirement 2.1) — no external input to validate
• Output / destination: job.c fact-recording calls, one per reported entry
• Idempotency & recovery: round token discards stale reports (6.3); a fresh round replaces a discarded one on the next join (6.1); applying a report twice for the same round is a no-op by construction of the underlying job.c guards (6.2)

State Management

• State model: one resync_state_t per worker_t — {round_token, queried_jobids[], deadline}; absent/cleared when no reconciliation is outstanding
• Persistence & consistency: not itself persisted — it is in-memory reconciliation-in-progress bookkeeping, not a durable fact; the durable facts it produces are the existing job-table WAL entries. A controller crash mid-reconciliation loses this bookkeeping, which is correct: the next rejoin (post-failover) re-triggers reconciliation from scratch (Requirement 1.3), so nothing needs to survive a crash here.
• Concurrency strategy: guarded by the same per-worker write lock rejoin_worker already holds

Controller: job.c — record_job_reconcile_pending_fact, record_job_reconcile_assign_fact

Field	Detail
Intent	The two reconciliation outcomes with no existing idempotent fact analog
Requirements	4.6, 4.7, 4.8, 5.4

Responsibilities & Constraints

• record_job_reconcile_pending_fact(cluster_t*, job_t*) — submits a WAL_JOB_UPDATE fact targeting JOB_STAT_PEND, through the existing apply_job_update_transition/job_state_is_backward guard (RUN→PEND is non-terminal→non-terminal, already permitted); fires the existing HOOK_DETAIL_JOB_UPDATE path exactly as any other job-state-update fact would. Also used for Requirement 5.4 (dispatch-to-disconnected-worker fail-fast).
• record_job_reconcile_assign_fact(cluster_t*, job_t*, worker_t*, xce_jobstate_report_t*) — mirrors dispatch_a_job_'s own host/state assignment logic (job.c:1686-1734) for a job the controller has no existing non-terminal record of; does not attempt to populate job->allocs (see Data Models).
• Neither function introduces a new guard — both submit through the existing apply_job_*_transition seam.

Contracts: Service [x]

Service Interface (C)

int record_job_reconcile_pending_fact(cluster_t* c, job_t* job);
int record_job_reconcile_assign_fact(cluster_t* c, job_t* job,
                                     worker_t* w, xce_jobstate_report_t* report);

• Preconditions: caller holds job's write lock (matching every other record_job_*_fact sibling); job is non-terminal (pending case) or has no existing non-terminal assignment (assign case)
• Postconditions: a WAL fact is durably appended; mutation happens later via apply_committed_entry, identical to every other fact type
• Invariants: never called for a job the controller already knows is terminal or more-final (Requirement 4.4/4.5 — enforced by the existing guard, not by new logic here)

Worker: xcecd_resync.c (new)

Field	Detail
Intent	Answer the controller's job-state query from the running list and the new finished-job retention; report extras; gate the worker's own post-join wake-up on completion
Requirements	3.1–3.8, 8.2

Responsibilities & Constraints

• For each queried job ID: check the running list (xcecd_worker_t) first; if not running, check the new retention; if in neither, report unrecognized (Requirement 3.1, 3.3). Every job ID present in the query gets an entry in the reply — none are silently skipped (Requirement 3.2).
• Include the real finish time from retention, not the time the report is sent (Requirement 3.5).
• After answering all queried IDs, walk the running list and retention for anything not in the query and report it too (Requirement 3.6), using the same JobStateEntry shape with queried=false.
• Send both in paginated OPCode_jobStateReport messages (Requirement 8.2).
• Sets a new local resync_pending flag (worker-side, mirroring controller_connector.c's existing needs_rejoin boolean pattern) when OPCode_jobStateQuery is received; clears it only once every report page (queried and extra) has finished sending. Whatever the worker's own post-join sequence does next (resuming periodic background reporting, entering normal operating mode) checks this flag first (Requirement 3.8). This is a local gate on the worker's own follow-on behavior, not a new wire-level ack and not an extension of xce_join()'s blocking wait (see Boundary Commitments' Out of Boundary).

Dependencies

• Inbound: query receipt (P0)
• Outbound: xcecd_worker_utils.c's running list + new retention (P0)

Contracts: Batch [x] / Event [x]

Batch / Job Contract

• Trigger: OPCode_jobStateQuery received
• Input / validation: job IDs in the query — no validation needed beyond matching against local state
• Output / destination: OPCode_jobStateReport, one or more pages
• Idempotency & recovery: re-sending the same query (e.g. after the worker's own reconnect mid-exchange) produces the same answers — the worker's local state is the read-only source of truth for this exchange

Worker: xcecd_worker_utils.c additions — finished-job retention

Field	Detail
Intent	Retain a job's true outcome (status, exit code, real finish time) for a bounded window after exit, instead of discarding it at the next cleanup_worker_list() pass
Requirements	3.4, 3.5, 3.7

Responsibilities & Constraints

• Populated at the same point a job's exit is currently detected, before cleanup_worker_list() would otherwise delete the entry.
• Bounded by ClusterConfig.workerResyncWindowMs (Requirement 3.7) — entries older than the window are dropped, not retained indefinitely.
• Does not change cleanup_worker_list()'s existing behavior for the in-flight xcecd_worker_t/sqlite bookkeeping — it adds a second, smaller, timestamped structure fed at the same moment, not a rework of the existing one.

Contracts: State [x]

State Management

• State model: a small bounded list/ring of {jobid, exit_status, exit_code, finish_time_ms}, in-memory only
• Persistence & consistency: not persisted — a worker restart loses this retention, which is acceptable: a restarted worker has no running jobs of its own to report on rejoin either way (its child processes died with it), so the retention's only job is to survive a controller-side disconnect/reconnect window, not a worker restart
• Concurrency strategy: same locking discipline already used for xcecd_worker_t's list

---

Data Models

Wire/Protocol Model

JobStateEntry {
  jobid: UInt64
  status: JobStateStatus (RUNNING | FINISHED_OK | FINISHED_FAIL | UNRECOGNIZED)
  exitCode: Int32           # meaningful only for FINISHED_FAIL
  finishTimeMs: UInt64      # meaningful only for FINISHED_OK/FINISHED_FAIL; worker-reported, per Req 3.5
  queried: Bool             # false = Requirement 3.6 extra, not part of the controller's original query
}

JobStateQuery { jobids: List(UInt64), pos: UInt32, count: UInt32, ntotal: UInt32 }
JobStateReport { entries: List(JobStateEntry), pos: UInt32, count: UInt32, ntotal: UInt32 }

Controller-Side In-Memory Model

resync_state_t {          // new, attached to worker_t
  round_token: uint64      // discards stale/superseded reports (Req 6.3)
  queried_jobids: set<uint64>
  deadline: timestamp      // Req 5.3
}

Persistence Field-Completeness Review

Per design-principles.md's Persistence Review requirement and persistence.md's APPLY_DIVERGENT/bug-1678 precedent, every fact this feature introduces or reuses is enumerated below:

Outcome	Fact path	New fields?	Both apply seams?
Finished success	record_job_finish_fact (existing, reused as-is)	None — existing complete fact shape (already carries finish_time, exit info); the reconciliation caller passes the worker-reported finishTimeMs (Requirement 3.5) as that fact's finish time, never the time the controller processed the report (Requirement 4.2)	Already covered by existing apply_committed_entry/standby-catch-up handling — unchanged
Finished failure	record_job_fail_fact (existing, reused as-is)	None — same Requirement 4.2 worker-reported-time rule applies	Unchanged, existing
Still running	No fact — pure no-op against the outstanding set	N/A	N/A
Unrecognized → pending	record_job_reconcile_pending_fact (NEW)	None beyond the existing WAL_JOB_UPDATE state field	Must confirm at implementation time that both apply_committed_entry (primary boot replay) and the standby catch-up apply path decode WAL_JOB_UPDATE identically for an arbitrary target state — expected yes, since this is the same fact type as any other state update, just a new call site, but this is a mechanical check to perform, not assume
Unqueried job → assign	record_job_reconcile_assign_fact (NEW)	jobid, host assignment, state — explicitly does NOT include job->allocs (see below)	Both seams must decode/adopt jobid+host+state identically — new call site into what is otherwise the same assignment shape dispatch_a_job_ already produces

Documented scope decision — job->allocs on a reconciliation-assigned job (Requirement 4.8): when the controller reconciles a job it never held any prior record of (e.g. it was dispatched by a controller instance before a failover whose dispatch fact was itself lost), it has no way to learn what resource allocation the scheduler originally computed for that job — the worker's report carries the job's execution outcome, not its original resource request. This design deliberately does not attempt to reconstruct job->allocs for this case. This is not a new problem: persistence.md's APPLY_DIVERGENT section already documents WAL_JOB_DISPATCH itself not carrying job->allocs through replay as a tracked, separate gap (roadmap.md's raft-apply-decoupling item #4); this design does not worsen it. The practical consequence is bounded by resource-accounting.md's existing policy: a job with an unknown/zero reservation is charged max(reserved=0, actual) — i.e., its real usage — the moment its next rusage tick arrives, identical to how a legitimately-unreserved job is already handled today. The job's state is correct immediately; its resource accounting converges within one rusage cycle, not instantly. No fabricated allocation value is ever written.

Configuration Model

Per config-design.md's classification rules, applied explicitly (not left implicit):

Field	Rule 0 (bootstrap)	Independence	Frequency	Verdict	Why
workerResyncWindowMs (Req 3.7)	passes — not needed to obtain config	passes — one worker's reporting window has no cross-node correctness dependency	passes — rarely changed	Shared, via operational-uniformity override	Structurally identical to heartbeatInterval/inactiveHeartbeatCount (controller→worker liveness domain): no correctness invariant demands agreement, but operators want uniform, predictable reconciliation behavior across the cluster rather than per-node drift — same documented exception, same rationale
workerResyncTimeoutMs (Req 5.3)	passes	passes — one worker's reconciliation deadline has no cross-node dependency	passes	Shared, via operational-uniformity override	Same reasoning as above; kept alongside workerResyncWindowMs in ClusterConfig rather than splitting the override rationale across two places

Both fields are added to the existing ClusterConfig capnp struct, distributed via the existing shared-config push mechanism — no new configuration surface or distribution channel.

---

Error Handling

Error Strategy

Every reconciliation outcome resolves definitively — there is no "leave it ambiguous" outcome. A worker that cannot determine a job's true status reports UNRECOGNIZED rather than guessing (Requirement 3.3), which the controller treats as "return to pending," never as a fabricated finish.

Error Categories and Responses

• Worker disconnects mid-round (Requirement 6.1): the outstanding round is discarded (not applied partially); the next join re-triggers a fresh round via the same Requirement 1 check.
• Stale/superseded report arrives (Requirement 6.3): discarded by round-token mismatch in resync.c — never applied against a round that's already been superseded.
• Reconciliation report never arrives / no reply within the bound (Requirement 5.3): a delaymgr-scheduled trigger (non-blocking fire, per the async-I/O thread-role contract) resolves the outstanding round — accepted-as-answered, timed out, or left for the next rejoin to retry — rather than withholding that worker from dispatch indefinitely. This must not be a blocking wait anywhere in the iomgr/delaymgr call chain; if a synchronous wait style is preferred at implementation time instead of a pure trigger-and-resolve design, it must follow commit.c's COMMIT_WAIT_RESERVED admission-reservation pattern (bounded blocking under an explicit cap on the vtask pool), never a raw wait.
• Dispatch attempted to a disconnected worker (Requirement 5.4): fails immediately (not queued for later delivery) and returns the job to pending via record_job_reconcile_pending_fact — a distinct classification from the existing retry-then-terminal-fail branch in dispatch_a_job_, whose retry semantics (retry the dispatch attempt itself against the same worker) don't fit "the worker isn't even connected."
• New opcode reaches a non-primary controller: per api-standards.md's "Adding a New Opcode" checklist (bug-1700 precedent) — verify at implementation time whether OPCode_jobStateQuery/OPCode_jobStateReport need an explicit redirect-reply path. Expected answer: no, since a worker only ever holds a connection to the current primary (connection-lifecycle-tied-to-primary-status, per the HA connection-ownership model), so this opcode pair should never reach a standby in practice — but this must be confirmed, not assumed, given bug-1700's documented history of this exact class of gap recurring across two other opcodes.

Monitoring

Reuses existing job-state-change logging (job.c's existing log call sites for finish/fail/update facts) — no new logging framework. A reconciliation-driven fact should log which outcome and round produced it, for operator diagnosis of a rejoin event, matching the existing style (structured, operator-visible on the ERROR/WARN path per error-handling.md; INFO for normal reconciliation completion).

---

Testing Strategy

Unit Tests (Controller, src/controller/testsuite/test_worker_resync.c)

• Reconciliation triggers iff the host has ≥1 non-terminal job at rejoin (Requirement 1.1/1.2), for all three join circumstances (first join, network reconnect, failover reconnect) without the code needing to distinguish them (Requirement 1.3).
• Query construction lists exactly the controller's non-terminal jobs for that host by ID (Requirement 2).
• Each of the five reported outcomes (running, finished-success, finished-failure, unrecognized, unqueried-extra) applies through the correct existing/new fact function exactly once, and a repeated report for the same round is a no-op (Requirement 4, mirroring test_guard_replay_finish_equals_single_application's existing style).
• XCE_WORKER_RECONCILING blocks dispatch_a_job_ and clears once the round resolves (Requirement 5.1/5.2).
• Timeout resolves an outstanding round without a blocking call (Requirement 5.3).
• Disconnect mid-round discards and re-triggers; a stale-round report is discarded (Requirement 6).
• A reconciled state change reaches the scheduler via the existing notify path, asserted the same way existing finish/fail notify tests assert it (Requirement 7).

Unit Tests (Worker, new src/worker/testsuite/test_xcecd_resync.c)

• Query answered correctly from the running list, from retention, and as UNRECOGNIZED when in neither (Requirement 3.1–3.3).
• Real finish time reported, not send time (Requirement 3.5).
• Extras reported for jobs outside the query (Requirement 3.6).
• Retention window boundary: an entry just inside/outside workerResyncWindowMs is/isn't reported (Requirement 3.7).
• resync_pending gate: set on receiving OPCode_jobStateQuery, stays set across multiple report pages, and only clears after the last page (queried and extra) has been sent — the worker's post-join wake-up path must observe it set throughout, never cleared early (Requirement 3.8).

Integration Tests

• Full rejoin-reconciliation round trip against a live controller+worker pair, mirroring test_worker_join.c's fixture conventions (setup_test_cluster/teardown_test_cluster, direct calls, one test per behavior).
• Failover-during-job: a job's finish notify is lost across a primary failover; the new primary's rejoin reconciliation recovers it (this is the feature's core bug-1741 regression scenario).
• Burst/array scenario resembling bug-1741's original reproduction (many near-simultaneous finish reports lost on one host) — resync must recover all of them, unbounded by count (Requirement 8).

Performance/Load

• Reconciliation exchange for a worker running an order of magnitude more jobs than a single message can carry completes correctly across multiple pages, not truncated (Requirement 8.1–8.3).

---

Security Considerations

No new authentication/authorization surface — the new opcode pair travels over the same authenticated controller↔worker connection as every other daemon message, subject to the same ha_dispatch_gate role-based admission already governing every opcode. No new external input validation beyond matching reported job IDs against the controller's own records (an unrecognized ID is simply itself information, per Requirement 3.3/4.5 — never treated as an error condition to reject).

Supporting References

Deep file:line citations, the alternatives considered for the wire-protocol shape, and the full rationale for each design decision above live in research.md.