myxce/.kiro/specs/worker-rejoin-resync/tasks.md

Implementation Plan

• 1. Foundation: wire protocol schema and shared reconciliation types

• 1.1 Define the wire protocol for the reconciliation exchange

  • Add the controller→worker job-state query and worker→controller job-state report as new opcodes, each carrying a paginated list of per-job entries (job ID, status, exit code, worker-reported finish time, and whether the job was part of the original query)
  • Add the two new cluster-wide configuration fields: the recently-finished reporting window and the reconciliation timeout
  • Observable completion: the schema builds cleanly and the new message/opcode/config-field definitions are visible in the generated bindings
  • Requirements: 2.1, 2.2, 3.5, 3.7, 5.3, 8.1, 8.2, 8.3

• 1.2 (P) Add a worker reconciling status and round-tracking state

  • Add a new worker status value meaning "job-state reconciliation outstanding for this host"
  • Add a per-worker reconciliation-round structure capable of holding a round identifier, the set of queried job IDs, and a deadline
  • Observable completion: the new status value and round structure compile and are reachable from the worker record, unused by any caller yet
  • Requirements: 5.1, 6.3
  • Boundary: shared types

• 1.3 (P) Establish worker-side C test infrastructure

  • Add the first test-runner harness for the worker daemon, mirroring the controller's existing runner conventions
  • Observable completion: an empty/placeholder worker-side test program builds and runs successfully via the existing test build target
  • Requirements: 3.1, 3.2, 3.3
  • Boundary: worker test infrastructure
  • Note: satisfied by pre-existing infrastructure — see Implementation Notes.

• 1.4 (P) Add wire encode/decode helpers for the reconciliation exchange

  • Add request/reply encode and decode helpers for the new query and report messages, following the existing paginated-list pattern already used for job/worker/resource listings
  • Observable completion: a round-trip encode-then-decode of a query and of a report (including an empty page and a multi-page case) reproduces the original data exactly
  • Requirements: 8.1, 8.2, 8.3
  • Boundary: wire codec
  • Depends: 1.1

• 1.5 (P) Wire the new config fields into xceadmin's config-build pipeline

  • Add the two new cluster-config fields (the recently-finished reporting window, the reconciliation timeout) to xceadmin's YAML validation schema and its YAML-to-binary field mapping, following the existing heartbeatInterval/inactiveHeartbeatCount pattern exactly, so an operator can actually set them through xceadmin config build — added to the wire schema by task 1.1, but with no configuration path into it until this task
  • Regenerate the checked-in golden config-build fixtures to match the wire schema's current field set (task 1.1 grew it; the fixtures predate that and no longer match a fresh build)
  • Observable completion: a YAML config supplying values for both new fields produces a binary that decodes back to those exact values, and the existing golden-fixture regression tests pass again
  • Requirements: 3.7, 5.3
  • Boundary: xceadmin config-build
  • Depends: 1.1

• 2. Core: controller job-table reconciliation outcomes

• 2.1 Return an unrecognized job to pending state, releasing its resource allocation

  • When a worker reports a queried job as unrecognized, transition that job back to pending using the existing state-transition guard so a terminal or already-more-final job is never overwritten
  • Release any resource allocation the job was carrying, so a job returned to pending never carries a stale allocation from a prior dispatch into its next dispatch attempt
  • Durably record the transition and notify the scheduler of the new pending state through the existing notification path, without waiting for a later replay to discover the change
  • Observable completion: a job previously dispatched (real allocation committed) that reconciles to unrecognized ends the call with pending state, no resource allocation, and the scheduler notified — verified before any commit round-trip completes
  • Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 4.6, 7.1, 7.2, 7.3
  • Boundary: job.c additions

• 2.2 Assign an unqueried job to the worker that reported it

  • When a worker reports a job the controller has no existing non-terminal record of, record that job as assigned to the reporting worker and host, matching the worker's reported status, through the same idempotent path used for a normal dispatch
  • Do not fabricate a resource allocation for this job — leave the allocation unset, consistent with the controller genuinely not knowing what the scheduler originally computed
  • Notify the scheduler of the new assignment through the existing notification path
  • Observable completion: a job the controller never previously tracked, reported running by a worker, ends the call recorded as running on that host with the scheduler notified, and durably survives a restart replay with the same result
  • Requirements: 4.1, 4.3, 4.4, 4.7, 4.8, 7.1, 7.2, 7.3
  • Boundary: job.c additions

• 2.3 Fail a dispatch to a disconnected worker immediately, without retry or job failure

  • When the controller attempts to dispatch a job to a worker it is not currently connected to, fail that specific dispatch attempt right away and return the job to pending, reusing the pending-and-release behavior from task 2.1, rather than retrying the attempt or exhausting a retry count toward permanent failure
  • Leave the existing behavior for every other reason a worker might not be ready (locked, busy, closed, or mid-reconciliation) completely unchanged — confirm those cases still retry and eventually fail exactly as before
  • Observable completion: dispatching to a disconnected worker returns the job to pending on the first attempt with the job's failure-retry count untouched and no permanent failure recorded, while dispatching to a worker that is connected but otherwise not ready still follows the pre-existing retry-then-fail behavior unchanged
  • Requirements: 5.1, 5.4
  • Boundary: job.c additions
  • Depends: 2.1

• 3. Core: controller reconciliation-round bookkeeping

• 3.1 (P) Track one reconciliation round per worker with a bounded timeout

  • Given a worker host with one or more non-terminal jobs currently assigned to it, construct the exact list of those jobs' unique IDs as the round's query set
  • Track a round identifier that distinguishes the current reconciliation attempt from any prior one for the same worker
  • Schedule a bounded timeout for the round using a non-blocking trigger; when it fires before the round otherwise resolves, resolve the round rather than leaving the worker withheld from dispatch indefinitely
  • Observable completion: arming a round for a worker with known non-terminal jobs produces a query set matching exactly those job IDs, a fresh round identifier, and an active timeout; letting the timeout elapse resolves the round without any blocking call
  • Requirements: 1.1, 1.2, 1.3, 2.1, 2.2, 5.3, 6.1
  • Boundary: resync.c

• 3.2 Apply each reported job outcome through the correct idempotent path

  • For each entry in a worker's job-state report: a still-running job is cleared from the round's outstanding set with no fact recorded; a finished-success or finished-failure job is applied through the existing finish/fail recording used for an ordinary report; an unrecognized queried job is applied through task 2.1's pending-and-release path; a reported job outside the original query is applied through task 2.2's assignment path
  • Discard a report that arrives for a round already superseded by a newer one for the same worker, rather than applying it
  • When the round's outstanding set becomes empty, resolve the round so the worker becomes eligible for dispatch again
  • Applying the same report twice for the same round must produce the same result as applying it once
  • Observable completion: a full report covering all five outcome kinds leaves the job table in the correct state for each, a duplicate re-application of the same report changes nothing further, and a report carrying a stale round identifier is dropped without touching the job table
  • Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 6.1, 6.2, 6.3
  • Boundary: resync.c
  • Depends: 2.1, 2.2, 3.1

• 4. Core: worker-side job-state reporting

• 4.1 (P) Retain a job's true outcome for a bounded window after it exits

  • At the moment a job's exit is detected, before its record would otherwise be discarded, retain its exit status, exit code, and true finish time for a configurable bounded window
  • Observable completion: a job's outcome remains queryable for the configured window after it exits and is no longer available once the window elapses
  • Requirements: 3.4, 3.5, 3.7
  • Boundary: worker retention

• 4.2 Answer the controller's job-state query, including extra unqueried jobs

  • For every job ID in the controller's query, report exactly one outcome: still running (from the live running list), finished successfully or with failure and its true finish time (from the new retention), or unrecognized if found in neither — every queried ID gets an answer, none are silently skipped, and an indeterminate case is reported as unrecognized rather than guessed
  • After answering the query, also report any job the worker is currently running or has recently finished that was not part of the query
  • Send both the query answers and the extra jobs in paginated reports, correct regardless of how many jobs are involved
  • Observable completion: a query covering a mix of running, finished, and unknown job IDs produces one correctly-classified entry per ID plus a separate page of extras for jobs outside the query, with no job silently dropped even when the total exceeds one page
  • Requirements: 3.1, 3.2, 3.3, 3.6, 8.2, 8.3
  • Boundary: xcecd_resync.c

• 4.3 Delay resuming normal post-join operation until the report has fully sent

  • Set a local gate the moment a job-state query is received; keep it set across every page of the report; clear it only once the full report, including any extra-job pages, has finished sending
  • Make whatever the worker does next after rejoining (resuming its own periodic background reporting) check this gate first, so that activity never begins or interleaves while the report is still being sent
  • Observable completion: the worker's post-join activity is observed to start only after the last report page has been sent, never before, across a report spanning multiple pages
  • Requirements: 3.8
  • Boundary: xcecd_resync.c

• 5. (P) Core: exclude a reconciling host from scheduler dispatch candidacy

  • Extend the scheduler's existing candidate-host filter so a host currently undergoing job-state reconciliation is excluded from dispatch candidacy the same way an already-unavailable host is
  • When the reconciling status clears, the same filter must stop excluding that host without any separate action
  • Introduces no new reconciliation-protocol name or concept into the scheduler's outbound decision-publish code — this is an ordinary status-bit read alongside the existing unavailable-host check, not a new symbol; the standing regression test guarding that publish path against ever referencing reconciliation concepts must stay green, unmodified
  • Observable completion: a host flagged as reconciling is never selected as a dispatch candidate while the flag is set, and becomes selectable again as soon as it clears, with no change to how any other exclusion reason behaves, and the publish-path symbol-guard test still passes unmodified
  • Requirements: 5.1, 5.2
  • Boundary: scheduler candidate filter

• 6. Integration: cross-daemon wiring

• 6.1 Trigger reconciliation on worker rejoin and send the query

  • At the moment a worker host becomes connected again (first join, network reconnect, or reconnect after controller failover, treated identically), check whether the controller currently holds non-terminal jobs for that host and, if so, arm a reconciliation round for it
  • Send the round's query to the worker as a follow-up exchange after the join reply and the existing configuration push have both already been sent, not as an extension of the join call itself
  • Observable completion: rejoining a worker with known non-terminal jobs results in a reconciliation round being armed and the corresponding query being sent to that worker, in that order, after the join reply; rejoining a worker with none results in no round and no query
  • Requirements: 1.1, 1.2, 1.3, 2.1, 2.2
  • Depends: 1.4, 3.1

• 6.2 Route an incoming job-state report to reconciliation processing

  • Dispatch an incoming job-state report message to the reconciliation-processing logic, correctly reassembling a report that arrives across multiple pages before it is applied
  • Confirm whether the two new message kinds need an explicit redirect response when received by a controller that is not currently primary, matching how every other primary-only message already behaves for a standby recipient — do not assume the existing generic rejection path already covers this without checking, given this exact class of gap has recurred for other message kinds before
  • Observable completion: a report sent across several pages is fully reassembled and applied exactly once, with the same result as an equivalent single-page report; and a query or report sent to a non-primary controller receives the same redirect behavior an equivalent primary-only message already receives, not a bare unrecoverable error
  • Requirements: 8.1, 8.3
  • Depends: 1.4, 3.2

• 7. Validation: end-to-end and regression coverage

• 7.1 Full rejoin-reconciliation round trip

  • Exercise a real rejoin against a live controller and worker pair: a worker with a mix of still-running, finished-success, finished-failure, and unrecognized jobs reconnects, and every job ends up in the correct final state with the scheduler notified
  • Observable completion: the round trip produces the correct end state for every job kind in one live test run, matching the per-outcome behavior verified in isolation by tasks 2.1-2.2 and 3.2
  • Requirements: 1.1, 1.2, 1.3, 2.1, 2.2, 3.1, 3.2, 3.3, 3.6, 4.1, 4.2, 4.3, 4.4, 4.5, 4.6, 4.7, 4.8, 7.1, 7.2, 7.3
  • Depends: 6.1, 6.2, 4.3, 5

• 7.2 Failover-during-job recovery

  • Reproduce the feature's core motivating scenario: a job's finish report is lost across a controller primary failover, and the new primary's rejoin reconciliation with that worker recovers the job's true outcome
  • Observable completion: a job whose finish notification never reached any controller instance ends up correctly finished, not stuck running, once the worker rejoins the new primary
  • Requirements: 4.1, 4.2, 4.6
  • Depends: 7.1

• 7.3 Burst/array recovery unbounded by job count

  • Reproduce a burst of many near-simultaneous job completions on one worker host where most of the individual finish reports are lost, and confirm reconciliation recovers all of them on the worker's next rejoin regardless of count
  • Observable completion: every job in the burst is correctly finished after reconciliation, with none left stuck and none duplicated
  • Requirements: 8.1, 8.2, 8.3
  • Depends: 7.1

• 7.4 Large job-count pagination correctness

  • Drive the reconciliation exchange for a worker running enough jobs to require multiple pages of both the query and the report, confirming the exchange completes correctly rather than truncating or silently dropping jobs past a single page
  • Observable completion: a job count an order of magnitude larger than a single page holds is fully queried and fully reported, with every job accounted for
  • Requirements: 8.1, 8.2, 8.3
  • Depends: 7.1

Implementation Notes

• Task 1.3: this task's premise (src/worker has no testsuite/) is FALSE as of implementation time — real, tracked worker test infrastructure already exists (src/worker/testsuite/{meson.build,test_*.c}, 5 passing tests, suite: 'worker' registered in meson, /xce-build worker already runs it), built by component-failover-adoption (commits ce9f734a/331fa024/619b119e, 2026-08-11) weeks before this spec was written. .kiro/steering/structure.md, testing.md:55, and build-deployment.md:16 are all stale on this point — worth a /kiro-steering sync, out of this feature's boundary to fix. Confirmed directly (fresh build + meson test -C src/xce.build --suite worker: 5/5 OK) rather than trusting the finding secondhand. No files changed for this task — closed as satisfied by pre-existing infrastructure.
• Task 4.2 must follow the EXISTING worker testsuite convention, not the controller aggregator-runner pattern the original task 1.3 text pointed at: each test file is its own standalone Unity executable (own UNITY_BEGIN()/RUN_TEST/UNITY_END(), white-box #includes the production .c file under test — see test_resource_sync.c's shape) with its own explicit executable() + test(..., suite: 'worker', ...) block added to src/worker/testsuite/meson.build, mirroring test_controller_connector.c's registration exactly. Do not introduce a second, competing test-runner shape.
• Task 1.5: xceadmin config dump's _format_cluster_config (build.py) does not print workerResyncWindowMs/workerResyncTimeoutMs — an operator can set them via config build but can't verify the built value via config dump. Confirmed genuinely out of task 1.5's boundary (config-build, not config-dump) and not required by Requirements 3.7/5.3 as written, but a real, minor observability gap worth a small follow-up task if this becomes a live operator pain point. Not blocking; not added to this plan.
• Task 2.2 found and fixed bug-1746, a real pre-existing defect in persist.c (part of bug-1678's original fix, predates this feature): capnp_encode_joballoc() unconditionally dereferenced a NULL alloc via encode_JobAlloc() instead of the already-existing NULL-safe encode_JobAlloc_ptr() wrapper — never triggered before since dispatch_a_job_ (the only prior caller) always passes non-NULL. This directly contradicted design.md's claim that the NULL-alloc case was "tested, backward-compatible" on both apply seams — true for decode, never actually true for encode until this task exercised it. Fixed with a minimal guard producing a true zero-length buffer (not a capnp-null-pointer encoding — verified against apply_committed_entry's alloc_len > 0 decode gate, which a non-zero-length "null" encoding would have wrongly satisfied). design.md corrected in 4 places. Any future task calling persist_submit_job_dispatch_fact with a NULL alloc can now trust this path — it wasn't safe before task 2.2, it is now.
• Task 2.2 (self-correction): a coordinator-applied one-line comment fix to job.c accidentally closed the enclosing /* */ comment block early, leaving the comment's remaining lines as bare (uncompilable) code — caught immediately by re-running the build before marking the task complete, not left for review to find. Reinforces why fresh-evidence re-verification applies even to "trivial, obviously-safe" doc-only edits made outside the TDD subagent loop.