myxce/.kiro/specs/worker-rejoin-resync/requirements.md

Requirements Document

Introduction

Today, a worker's job start/finish/fail report to the controller is a single fire-and-forget send (XCECLI_FF_CALL, src/lib/client_internal.h) with no retry and no redelivery. If that send is lost — because the worker briefly disconnects, or because the controller it was talking to fails over to a new primary mid-flight — the fact is gone forever: the job's true outcome is never recorded, and the job's state in the cluster stays wherever it last was (typically RUN) permanently, even though the worker's own OS process already exited. This was confirmed live (bug-1741): a burst of near-simultaneous job completions on one worker host lost 6 of 7 finish reports, leaving those jobs stuck in RUN forever while their siblings all completed normally, and no controller resync ever corrected them because nothing in the cluster still knew they had actually finished.

This feature closes that gap for the specific case of a worker reconnecting to the controller (after a network disconnect, or after the controller it was talking to fails over and the worker rejoins the new primary): on every such rejoin, the controller asks the worker directly what really happened to the jobs the controller believes are running there, and reconciles the worker's answer into its own job records through the existing idempotent commit-then-apply path. This does not attempt to prevent the underlying send loss (a separate concern); it recovers from it after the fact, the next time the worker and controller re-establish contact.

Boundary Context

• In scope: A worker reconnecting to the controller (network disconnect + reconnect, or reconnecting to a new primary after controller failover) triggers a job-state reconciliation exchange, initiated by the controller, covering jobs the controller currently believes are running on that specific worker host. The controller applies the worker's answer through the existing idempotent commit-then-apply path, with no duplicate facts and no clobbering of state the controller already knows is newer or more final.
• Out of scope: Preventing the original notification loss (the fire-and-forget send itself is unchanged). Resyncing anything other than job execution facts on worker rejoin — apiserver rejoin content, scheduler rejoin content, and any other cluster/queue/job list state resync are separate, deferred concerns and not covered here. Any change to the controller's existing pending-facts queue (which already retries facts it received but could not commit while the worker stayed connected) — this feature complements that mechanism for the case where the fact never arrived at all, it does not modify it. Delta-based or session-token re-attach optimizations for rejoin in general. Coupling the recently-finished reporting window (Requirement 3, Acceptance Criterion 7) to commit-batch-window's commit-urgency tuning — the window's sizing is driven by disconnect/reconnect coverage, and any mitigation it happens to provide for batched-but-uncommitted facts is incidental, not a designed dependency; commit-batch-window remains a separately parked feature.
• Adjacent expectations: This feature depends on the controller's existing commit-then-apply idempotency guards (already in place) to apply reconciled facts safely. It does not introduce a second, competing mechanism for deciding a job's state — reconciled facts are applied as ordinary job-state facts through that same path.
• Multi-host jobs: the Controller dispatches and tracks a multi-host job as a single record owned by that job's head/primary host — it never dispatches separately to the job's other hosts. Job-state reconciliation for a multi-host job therefore happens through that same head host, consistent with how the job is already dispatched and tracked; no separate handling is needed for the job's other hosts. This is unrelated to bug-1741's own documented gap (per-host resource-usage/rusage attribution for a multi-host job's individual tasks), which this feature does not touch.

Requirements

Requirement 1: Reconciliation Trigger

Objective: As the Controller, I want to start job-state reconciliation whenever a joining worker host has jobs I still believe are running on it, so that I don't need to reliably distinguish a worker's first-ever join from a reconnect.

Acceptance Criteria

1. When a worker host joins the cluster and the Controller currently records one or more non-terminal jobs assigned to that host, the Controller shall initiate job-state reconciliation for that worker as part of that join.
2. When a worker host joins the cluster and the Controller currently records no non-terminal jobs assigned to that host, the Controller shall not initiate job-state reconciliation for that worker.
3. The Controller shall apply this same trigger condition regardless of whether the join is the host's first-ever registration, a reconnect after a network disconnect, or a reconnect to a new primary after controller failover — the Controller shall not need to separately determine which of these a given join is.

Requirement 2: Controller-Initiated Job-State Query

Objective: As the Controller, I want to tell a joining worker exactly which jobs I currently believe are running on it, so that the worker only needs to answer about jobs I have an open question about.

Acceptance Criteria

1. When job-state reconciliation is initiated for a worker (Requirement 1), the Controller shall send that worker a list of the jobs the Controller currently records as assigned to that worker host and not yet in a terminal state.
2. The Controller shall identify each job in the query unambiguously (by the job's unique identifier) so the worker can match it to its own local job records.

Requirement 3: Worker Job-Status Report

Objective: As the Controller, I want the worker to tell me the real, current outcome of every job I asked about, so that I can correct any job-state fact I lost while we were disconnected.

Acceptance Criteria

1. When a worker receives a job-state query from the Controller, the Worker shall report back, for every job identifier in the query, one of: still running, finished successfully, finished with a failure/exit code, or unrecognized (the worker has no record of ever running that job).
2. The Worker shall report a status for every job identifier present in the query it received.
3. If the worker cannot determine a job's true status (for example, the job's process state is inconclusive at the time of the query), the Worker shall report that job as unrecognized/indeterminate rather than guessing a definite outcome.
4. If a job in the query has already finished before the rejoin completes, the Worker shall report the same finish outcome (success/failure and exit code) it would have reported through the ordinary job-finish notification.
5. When the Worker reports a job as finished (successfully or with a failure), the Worker shall include the time the job actually finished, not the time the report itself is sent.
6. The Worker shall also report any job it is currently running or has recently finished that was not included in the Controller's query, so the Controller can recover a job assignment its own records are missing.
7. The Worker shall bound "recently finished" (Acceptance Criterion 6) to a configurable time window, exposed as a cluster configuration parameter, so operators can tune how long a completed job remains eligible for this extra reporting without unbounding the exchange.
8. The Worker shall delay resuming its normal post-join operation (for example, periodic background reporting) until it has finished sending its complete job-state report (Acceptance Criteria 1-6) for the current join, so that the worker's other activity does not begin or interleave with the reconciliation exchange it is still completing.

Requirement 4: Controller Reconciliation

Objective: As the Controller, I want to apply a worker's job-status report as authoritative, idempotent corrections to my own job records, so that a job's recorded state converges to the truth without duplicating or corrupting existing facts.

Acceptance Criteria

1. When the Controller receives a worker's job-status report, the Controller shall apply each reported job outcome through the same idempotent commit-then-apply path used for ordinary job-state facts.
2. When the Controller reconciles a job's finish outcome, the Controller shall record the finish time the Worker reported (Requirement 3, Acceptance Criterion 5) as the job's finish time, rather than the time the Controller processed the reconciliation.
3. If a reported job outcome would duplicate a fact the Controller has already committed for that job, the Controller shall apply it as a no-op rather than recording a duplicate.
4. If the Controller has already recorded a more final or newer state for a job than the worker's report implies (for example, the Controller already recorded the job as finished by another means), the Controller shall not overwrite that state with a less final or older report.
5. When a worker reports a queried job as unrecognized, the Controller shall treat this as evidence the job's true outcome could not be confirmed by the worker and shall not fabricate a finish outcome for it.
6. When a worker reports a queried job as unrecognized, the Controller shall return that job to pending state, making it eligible for the normal scheduling path to assign it again, rather than leaving it permanently in its prior non-terminal state.
7. When a worker reports a job that was not part of the Controller's query (Requirement 3, Acceptance Criterion 6), the Controller shall reconcile that job's state through the same idempotent commit-then-apply path and precedence rules used for queried jobs (Acceptance Criteria 1, 3, 4), rather than discarding the report.
8. When the Controller reconciles a job reported outside its original query and the Controller has no existing non-terminal record of that job assigned elsewhere, the Controller shall record the job as assigned to the reporting worker, matching what the worker reported.

Requirement 5: Dispatch Gating During Resync

Objective: As the Controller, I want to withhold new job dispatch to a rejoining worker until its job-state reconciliation completes, so that I never assign new work to a worker whose true resource usage I don't yet know.

Acceptance Criteria

1. While a worker's job-state reconciliation is outstanding, the Controller shall not dispatch new jobs to that worker.
2. When a worker's job-state reconciliation completes (all queried jobs have a reported and reconciled outcome), the Controller shall make that worker eligible for new job dispatch again.
3. If a worker's job-state report does not arrive within a bounded time after the query is sent, the Controller shall resolve the outstanding reconciliation (accepting the query as answered, timed out, or reattempted on a subsequent rejoin) rather than withholding that worker from dispatch indefinitely.
4. If the Controller attempts to dispatch a job to a worker that is not currently connected, the Controller shall fail that dispatch attempt immediately and return the job to pending state, making it eligible for the normal scheduling path to assign it again, rather than holding the dispatch for delivery once the worker reconnects.

Requirement 6: Resync Failure and Repeated Rejoin

Objective: As the Controller, I want repeated or interrupted rejoin attempts to stay safe, so that a worker that keeps disconnecting during reconciliation never leaves job state permanently ambiguous or duplicated.

Acceptance Criteria

1. If a worker disconnects again before its job-state report is fully processed, the Controller shall discard the incomplete reconciliation and shall re-initiate job-state reconciliation on the worker's next rejoin.
2. The Controller shall produce the same reconciled result whether a worker's job-state report is applied once or is, due to a retry, presented to the Controller more than once for the same rejoin.
3. If a worker's job-state report arrives for a reconciliation round the Controller has already discarded or superseded with a newer one (Acceptance Criterion 1), the Controller shall discard that report rather than apply it.

Requirement 7: Scheduler Notification of Reconciled State

Objective: As the Controller, I want every job-state change produced by reconciliation to reach the Scheduler, so that the Scheduler's view of a job converges to the same truth as the Controller's, not just the Controller's own job table.

Acceptance Criteria

1. When reconciliation changes a job's recorded state, the Controller shall notify the Scheduler of that change through the same notification path used for an ordinary (non-reconciliation) job-state change.
2. When reconciliation returns a job from running back to pending state (Requirement 4, Requirement 5), the Controller shall notify the Scheduler of the job's new pending state, so the Scheduler makes the job available for scheduling again rather than continuing to treat it as running.
3. The Controller shall not consider reconciliation of a job complete until the resulting state change, if any, has been handed to the same notification path the Scheduler already relies on for ordinary job-state changes.

Requirement 8: Bounded Job-State Exchange Size

Objective: As an Operator, I want job-state reconciliation to complete reliably even for a worker running many jobs at once, so that a busy host is never unable to rejoin the cluster simply because it has a lot of jobs to reconcile.

Acceptance Criteria

1. The Controller shall be able to query and reconcile job state for a worker regardless of how many non-terminal jobs are currently assigned to that worker, without the reconciliation exchange failing solely because of the number of jobs involved.
2. The Worker shall be able to report status for every job in the Controller's query, and any additional jobs it is running (Requirement 3, Acceptance Criterion 6), without the report failing solely because of the number of jobs involved.
3. Where the number of jobs to reconcile for a worker exceeds what the Controller and Worker exchange in a single message, the exchange shall complete correctly across multiple messages rather than truncating the list or silently dropping jobs beyond a fixed limit.