myxce/.kiro/specs/scheduler-dispatch-commit/tasks.md

Implementation Plan

• 1. Foundation: controller-side dispatch-failure classification and durable transition record

• 1.1 (P) Define the dispatch-failure classification used to decide whether a failed dispatch attempt should retry or fail permanently

  • Enumerate each distinct way a dispatch attempt to a worker can fail today (worker lookup, worker unavailable, worker not ready, invalid allocation, queue configuration lookup, the dispatch request itself failing), and classify each as either retryable (transient/environmental) or terminal (a locally invalid decision that retrying will not fix)
  • The classification is a pure decision with no side effects, independently testable against every known failure case without needing a live dispatch attempt
  • Observable completion: a standalone test exercises every classified failure case and asserts the expected retryable/terminal outcome
  • Requirements: 4.1, 4.2
  • Boundary: Controller Dispatch-Failure Classification (site enum + classify function)

• 1.2 (P) Add a durable record of a job's dispatch-time transition to a running state

  • Introduce a new durable fact type recording that a job has transitioned to running state at dispatch-commit time, reusing the existing shape used for other job-state facts
  • Wire this new fact type into both the normal live-commit apply path and the recovery/restart replay path, so the transition is recovered identically whether observed live or replayed after a restart
  • Observable completion: appending this fact and then replaying it (simulating a restart) leaves the job's state as running, without requiring the separate worker-confirmed-start fact to also be present
  • Requirements: 2.4
  • Boundary: WAL Dispatch Fact

• 1.3 (P) Track how many consecutive times a job's dispatch has failed for a retryable reason

  • Add an in-memory counter on the controller's job record that increments on each retryable dispatch failure and resets to zero on a successful dispatch
  • This counter does not need to survive a controller failover — resetting to zero after a failover is an acceptable, safe degradation
  • Observable completion: the counter can be incremented, read, and reset independently of any actual dispatch attempt
  • Requirements: 4.3
  • Boundary: Controller Dispatch-Failure Classification (retry-counter data model)

• 2. Core: scheduler transitions a job to running state at dispatch-commit time

  • No dependency on Task 1 — scheduler-side files only, independent of the controller-side foundation work above.

• 2.1 (P) Scheduler marks a job as running in the same operation that commits its dispatch decision

  • When the scheduler commits a decision to dispatch a job, the job's state becomes running immediately, in the same operation that already records the job's resource allocation — not waiting for any later confirmation
  • The scheduler's existing resource-in-use accounting, which only counts jobs already in a running state, requires no changes — this task provides the input it was missing
  • Observable completion: a job dispatched in one round appears in resource-in-use totals starting that same round, and continues to appear in every subsequent round regardless of how many rounds pass before any confirmation arrives
  • Requirements: 1.1, 1.2, 1.3
  • Boundary: Scheduler Dispatch Commit

• 2.2 (P) Scheduler processes a reported dispatch failure by returning the job to a pending state

  • When the scheduler is told a job's dispatch failed, it returns that job to a pending state instead of silently ignoring the report as it does today
  • Returning a job to pending correctly releases the resource-allocation data that was tentatively committed for it, reusing the scheduler's existing behavior for any other transition away from a running state
  • Observable completion: a job whose dispatch failure is reported to the scheduler is again eligible for the scheduler's own job-selection process in a subsequent round, with no dangling resource-allocation data left behind
  • Requirements: 3.1, 3.2, 4.4
  • Boundary: Scheduler Dispatch-Failure Handler

• 3. Core: controller transitions a job to running state only once dispatch to the worker succeeds, and resolves failures

  • Sub-tasks 3.1-3.3 all restructure the same function and share its per-job lock scope, so none of them are marked parallel to each other, regardless of how disjoint their individual concerns look.

• 3.1 Controller commits a job's resource allocation and running-state transition together, only on a successful dispatch

  • The controller's existing (currently unconditional) commit of a job's resource-allocation data becomes conditional on the dispatch request to the worker actually succeeding; the job's state transition to running happens in that same successful outcome, protected against concurrent access to the job record
  • Record the new durable dispatch-transition fact (Task 1.2) as part of this same successful outcome
  • Reset the consecutive-retryable-failure counter (Task 1.3) to zero as part of this same successful outcome, so a job that later fails again starts counting from zero rather than retaining count from before its last success
  • The worker's own later confirmation that the job actually started continues to be recorded exactly as it already is today, unaffected by this change
  • Observable completion: querying a job's status immediately after a successful dispatch shows it as running, and that status is unaffected by whether the worker has separately confirmed the job started yet; a job that fails, succeeds, then fails again is evaluated against the bound as if it had only failed once
  • Requirements: 2.1, 2.2, 2.3, 2.4, 5.1
  • Depends: 1.2, 1.3

• 3.2 Controller resolves a terminal (locally-invalid) dispatch failure

  • When a dispatch attempt fails for a reason the classification (Task 1.1) marks terminal, the controller resolves the job to a permanently failed state immediately, protected against concurrent access to the job record, and releases any resource-allocation data tentatively associated with the attempt
  • This covers the two failure sites that already release the job reference they hold today — confirm that behavior is preserved, not duplicated, by this new resolution step
  • Observable completion: a terminally-classified dispatch failure leaves the job permanently failed, with no partial allocation data left behind and the job's reference count unchanged from before the attempt
  • Requirements: 4.2, 4.5
  • Depends: 1.1

• 3.3 Controller resolves a retryable (transient/environmental) dispatch failure, with bounded escalation

  • When a dispatch attempt fails for a reason the classification (Task 1.1) marks retryable, the controller increments the consecutive-failure counter (Task 1.3) under the same protected access; if the incremented count is still within a bounded limit, it reports the failure so the job can be retried, otherwise it resolves the job to a permanently failed state exactly as Task 3.2 does for a terminal failure
  • This covers the four failure sites that do not currently release the job reference they hold — fix each to release it as part of this same resolution step (Task 3.2's two sites already release theirs correctly and are unaffected)
  • Observable completion: a retryable dispatch failure below the bound leaves the job eligible for another attempt with its reference count restored to the pre-attempt value; a retryable failure that pushes the count past the bound resolves the job to permanently failed instead
  • Requirements: 4.1, 4.3, 4.4, 4.5
  • Depends: 1.1, 1.3, 3.2

• 4. Integration: caller no longer duplicates dispatch-outcome reporting

• 4.1 Remove the caller's own dispatch-outcome reporting now that resolution owns it

  • The function that invokes a dispatch attempt for each job in a batch no longer separately reports the outcome itself — reporting is now entirely owned by the dispatch resolution added in Task 3, preventing a failed dispatch from ever producing two different, conflicting outcome reports for the same job
  • Observable completion: a dispatch failure that resolves as permanent produces exactly one outcome report, not two conflicting ones
  • Requirements: 3.1, 4.1, 4.2
  • Depends: 3.1, 3.2, 3.3

• 5. Validation: end-to-end behavior and regression

• 5.1 End-to-end test: resource-in-use accounting survives a multi-round gap between dispatch and confirmation

  • Reproduce bug-1624 directly: dispatch a job, let several rounds pass before the worker confirms it started, and confirm resource-in-use totals reflect the job continuously from the round it was dispatched, matching what a same-round confirmation would have produced
  • Observable completion: the regression scenario passes, and fails against the pre-change code path if temporarily reverted
  • Requirements: 1.1, 1.2, 1.3, 2.1, 2.3
  • Depends: 2.1, 3.1

• 5.2 End-to-end test: retryable and bounded-retry dispatch failures resolve correctly

  • Simulate a transient dispatch failure and confirm the job becomes eligible for another attempt; simulate enough consecutive transient failures to exceed the bound and confirm the job resolves to permanently failed instead of retrying indefinitely; simulate a failure, a success, then another failure and confirm the bound is evaluated as if only one failure had occurred
  • Observable completion: all three scenarios produce the documented outcome, with the counter's bound value exercised at exactly the boundary (one under, exactly at, and one over)
  • Requirements: 3.1, 3.2, 4.1, 4.3, 4.4, 4.5
  • Depends: 2.2, 3.3

• 5.3 End-to-end test: terminal dispatch failure resolves without duplicate reporting or leaked references

  • Simulate a locally-invalid dispatch decision and confirm the job resolves to permanently failed, that no retry-eligible report is also produced for the same failure, and that the job record's reference count returns to its pre-attempt value
  • Observable completion: the assertions above all pass in the same test run
  • Requirements: 4.2, 4.5
  • Depends: 3.2, 4.1

• 5.4 Full regression: existing suites and HA soak

  • Run the existing scheduler and controller test suites end to end, updating only the assertions that directly depended on the old confirmation-gated timing; run a full sequential and chaotic HA soak
  • Observable completion: both test suites pass and the soak shows no new job loss, state divergence, or liveness stall attributable to this feature
  • Requirements: 5.2, 5.3, 5.4
  • Depends: 5.1, 5.2, 5.3

Implementation Notes

• Task 1.1: design.md's "Controller Dispatch-Failure Classification" Service Interface Preconditions text claims a default-less switch over xce_dispatch_fail_site_t gives "compiler-enforced exhaustiveness... fails to compile" for an unhandled case — verified false for plain C without -Werror (this project sets none, confirmed in src/meson.build/src/controller/meson.build). Implemented instead with a default: arm that ERROR-logs and returns a safe XCE_DISPATCH_FAIL_TERMINAL, with assert(0 && ...) gated to #ifdef XCE_TESTING only — mirrors the established, verified-matching precedent persist.c's wal_classify() already uses (persist.c:6030-6037) for the same "never risk silent misbehavior on an unrecognized case" concern. Any later task/spec touching this same design.md section should correct its wording rather than re-trusting the "fails to compile" claim.
• Tasks 3.2/3.3 (retryable/terminal resolution, both call xce_classify_dispatch_fail()) should follow the same fail-loud default: convention if they add their own exhaustive switches (e.g. over xce_dispatch_fail_class_t's two values).
• Operational note (not spec-relevant): implementer/reviewer subagents in this run consistently ended their turn mid-build (after starting xce-build.sh controller, before it finished) rather than blocking on it, requiring one resume-and-nudge each time. Future task dispatches in this run should explicitly instruct agents to poll/block synchronously until the build genuinely completes before composing their status/verdict block, to save round-trips.
• Task 1.2: apply_wal_entry() (persist.c) is confirmed to have ZERO production call sites today — persist_recover()'s actual boot-time replay path (replay_wal()) and the standby catch-up path (ha_standby_advance_committed_and_apply(), post the 2026-07-10 ha_apply_single_entry refactor) both go through apply_committed_entry() exclusively. design.md's "WAL Dispatch Fact" component (its "Preconditions"-adjacent prose) still characterizes apply_wal_entry() as "the recovery/replay-time path reading WAL segments from disk after restart" — that's stale. Adding WAL_JOB_DISPATCH to both functions' fall-through groups was still correct (matches every sibling WAL_JOB_* type's identical dual-registration, harmless dead-code parity) and is NOT something to undo — but design.md's wording should be corrected before any later task cites apply_wal_entry()'s role from that doc without re-verifying.
• Task 3.1: xce_request()/every xce_*_req() wrapper (src/lib/request.c) is genuinely non-blocking fire-and-forget (capnp-encode + endpoint_enq() onto ep->outq, returns immediately — never waits for a reply). endpoint_enq() succeeds unconditionally against a zeroed endpoint_t with ep->iomgr == NULL. Net: testing a controller-side dispatch_a_job_() success path needs no live peer/socketpair/round-trip thread — a zeroed static endpoint_t (.fd = -1, no iomgr) is sufficient, matching test_commit_barrier_reply.c's own established g_wkr_disp_ep fixture precedent. Reused directly for test_dispatch_commit.c; any later task exercising dispatch_a_job_()/xce_startjob_req() in a test should use the same pattern rather than building real network mocking.
• Task 3.1: test_commit_barrier_reply.c's test_jobdispatch_receipt_ack_no_wal_append (a different, ha-commit-barrier-rollout-owned pinned test) asserts a successful dispatch cycle appends ZERO WAL entries — now false, since a successful dispatch appends exactly one WAL_JOB_DISPATCH fact (Req 2.4). Confirmed by both task 3.1's implementer and its independent reviewer; this is explicitly the kind of stale assertion Requirement 5.3 / task 5.4 exists to update, deliberately left unfixed here (out of 3.1's boundary, and that file's tests were unreachable in a full-suite run anyway due to the bug-1429 hang below, registered earlier in controller_test.c's main()). Task 5.4 must update this assertion before claiming full-suite regression-clean.
• Task 3.1: reconfirmed bug-1429 (controller_unit_test TIMEOUT/hang, pre-existing, unrooted) a 3rd time, now hanging inside test_commit_barrier_reply.c's test_jobsubmit_repeated_truncation_forced_failures_no_phantom_jobs — a 3rd distinct hang location across 3 sessions, confirmed unrelated to dispatch_a_job_() (that test never calls it) and independently reproduced twice more (standalone binary run past the meson harness's 300s cap, and a fully separate rebuild). Because this hang sits earlier in controller_test.c's registration order than most of this feature's own new test files, a full un-filtered meson test run will NOT reach them — task 5.4's full-suite validation must run with the hanging group filtered out (controller_unit_test -t <group>) or budget for the 300s timeout, not treat a bare meson test TIMEOUT as this feature's own failure.
• Task 3.1: confirmed pre-existing (not introduced by this task, not in this task's boundary to fix) — dispatch_a_job_()'s success path never calls dereference(j) on the reference taken at lookup; identical in the diff's before/after, and the sole caller handle_jobdispatch() never dereferences either. A candidate future bug-log entry, out of scope for this spec.
• Task 3.1: when validating a build against this suite's ASAN leak output, always replicate the project's own ASAN_OPTIONS=detect_leaks=0 test env (meson.build:229) — a standalone binary run without it produces spurious leak reports (a red herring, not a regression) that can waste review time.
• Task 3.2: job_t's real reference count lives in jobj.c's file-local job_private_t, with no accessor exposed anywhere in the codebase — a test proving no reference-count regression can't do a direct numeric comparison. The working pattern (used by test_dispatch_fail_terminal.c) is a "canary reference": take one extra manual reference() on the job immediately before the call under test (not after — arming it after only relies incidentally on other assertions crashing first, which doesn't independently prove anything), then dereference() it after the call and confirm the job is still safely readable — an over-release during the call frees the job while the canary is still held, so the post-call touch is a real ASAN-caught heap-use-after-free.
• Task 3.2: reconfirmed bug-1429's hang again (still test_commit_barrier_reply.c, still unrelated to any change in this feature) — third+ reproduction, still not root-caused. Continues to require controller_unit_test -t <group> filtering for a clean isolated regression signal.
• Task 3.3: bug-1627 (the 4-site reference leak on dispatch_a_job_()'s retryable failure paths) is now FIXED — dereference(j) added at all four retryable sites, closing it as a direct consequence of this task's resolution restructuring. .wolf/buglog.json updated accordingly.
• Task 3.3: notify_dispatch() is static and defined AFTER dispatch_a_job_() in job.c with no forward declaration — needed one added near the top (mirroring mark_job_fail's existing forward-decl precedent at job.c:24) so the new retryable-resolution code could call it. Any later task in this file needing to call a same-file function defined later in the file should check for/add a forward declaration the same way, rather than reordering definitions.
• Task 3.3: capturing a site's natural error code (rc/errno) into a stable local must happen BEFORE any intervening call once more logic is inserted before the eventual return — errno is not guaranteed stable across lock()/ERROR()/free_launchinfo() etc. This bit the worker-null and worker-not-ready sites specifically (both originally read errno directly at the return statement with nothing in between; task 3.3 had to capture it immediately after SET_ERROR()/at branch entry, before inserting the new lock/classify/resolve sequence).
• Task 3.3: test_commit_barrier_reply.c (a file otherwise owned by the ha-commit-barrier-rollout spec) is this binary's sole host of the notify() stub — any future task needing to observe notify_dispatch()'s calls in a controller unit test must add accessors there too (purely additive, read-only _ext-suffixed pass-throughs to its existing static capture globals — do not modify the stub body itself or its existing pinned tests). Confirmed as legitimate shared test infrastructure, not a boundary violation, by task 3.3's independent review.
• Task 3.3: the bound constant is XCE_DISPATCH_FAIL_RETRY_BOUND = 3 (design.md's stated default), defined once near job.c's top and used at all four retryable sites — no magic numbers. 3 consecutive failures still retry; the 4th resolves terminal.
• Task 4.1: test_commit_barrier_reply.c's test_jobdispatch_receipt_ack_no_wal_append is now stale for a SECOND, independent reason, on top of task 3.1's already-documented WAL-append-count staleness: its 2-job batch (one real-worker success, one nhosts==0 terminal failure — no retryable job) now produces ZERO notify_dispatch() calls total post-task-4.1 (the success path never called it even before task 4.1, per task 3.1's design; the terminal path resolves via mark_job_fail()'s schedFailJob hook chain, not notify_dispatch()/jobDispatchFail, per task 3.2's design) — breaking its pinned g_notify_count == 2 / both-jobDispatchFail-event assertions (lines ~1991-2003). Task 5.4 must update BOTH the WAL-count assertion (task 3.1's finding) and the notify-count/event assertions (this finding) in the same test before claiming full-suite regression-clean; the test's own scenario (1 success + 1 terminal, zero retryable jobs) may need to change too since it can no longer meaningfully exercise notify_dispatch() at all with today's cast of outcomes — consider adding a retryable-classified job to the batch if the intent is still to prove notify_dispatch()'s wiring through handle_jobdispatch().
• Task 4.1: confirmed handle_jobdispatch()'s only caller (cluster.c:471, inside myreq_handler) uses its return value only as the generic RPC-handler rc, and handle_jobdispatch() itself always returns 0 unconditionally regardless of any job's outcome — nothing depends on the removed per-job notification behavior at the caller level.
• Task 5.1: a "same-round confirmation" scenario for resource-in-use accounting is legitimately modeled as nrounds==1 rather than by simulating a real schedStartJob notify event — design.md documents (and the reviewer independently re-verified against core_events.c) that re-applying JOB_STAT_RUN to an already-RUN job via the confirmation path is a confirmed no-op, so resInUse depends solely on JOB_IS_RUN(job->state), never on a separate confirmation flag. Reviewer's own finding: the resulting same-round-vs-multi-round comparison test is honest and correctly fails on revert, but adds little independent detection power beyond a per-round-asserting loop test (which is the one genuinely load-bearing proof) — worth knowing if a future task considers deduplicating these two tests, but not something to act on now.
• Task 5.1: no controller-side test was added for this task — Requirements 2.1/2.3 (controller transitions to RUN independent of confirmation) are already covered by task 3.1's test_dc_success_commits_alloc_state_and_resets_counter. No live scheduler+controller wire-protocol integration harness exists anywhere in this codebase; building one was judged disproportionate to what this task verifies, and the reviewer agreed.
• Task 5.2: bug-1631 (logged, NOT fixed, deliberately deferred) — dispatch_a_job_()'s four retryable-failure branches (job.c, tasks 3.1/3.3) never revert job.state/job.allocs when a retryable failure occurs while the job's controller-side state is CURRENTLY RUN (they were only ever designed for a job whose every dispatch attempt starts from PEND, which is all that's reachable today). A kiro-debug investigation (HIGH confidence), independently re-verified even more exhaustively by this task's own reviewer (full trace of every pendJobs-reinsertion and every controller-side job.state-mutation call site), confirmed this scenario is genuinely UNREACHABLE via any real caller in the current, fully-implemented system: the scheduler's picker permanently removes a job from its PEND candidate pool the instant it dispatches it (task 2.1), and nothing already-implemented ever re-adds a successfully-dispatched (RUN) job back to that pool. The only mechanism design.md itself anticipates for that is scheduler-preemption, confirmed still unimplemented (requirements-generated, no approvals). Revisit bug-1631 and add the state/allocs reversal to dispatch_a_job_()'s four retryable branches only once scheduler-preemption (or an equivalent mechanism) actually creates a real RUN→retryable-redispatch path — not before, since there's no reachable caller to regression-test against today. test_rpc_failure_bound_resets_after_intervening_success (test_dispatch_fail_retry.c) proves Requirement 4.3's actual, literal ask (the counter genuinely resets on an intervening success) without asserting the unreachable state/allocs behavior.
• Task 5.2: design.md's "vacuously by construction" reasoning (Controller Dispatch Commit and Controller Dispatch-Failure Classification sections — "on any of the six classified failure paths there is nothing to release") silently assumes every dispatch_a_job_() invocation starts from a job that's never had anything committed (i.e., a first-ever dispatch attempt) — true today (see bug-1631 above) but not stated as a scoped assumption in the doc text itself. Worth correcting design.md's wording if it's ever revisited for the scheduler-preemption follow-up.
• Task 5.3: test_dispatch_fail_terminal.c (task 3.2) and test_jobdispatch_no_double_notify.c (task 4.1) each covered PART of this task but not the full combination in one run — 3.2's test bypasses handle_jobdispatch() entirely (via dispatch_a_job_test()), so it can't exercise the double-notify race at all; 4.1's test only proves an aggregate batch notify count, not a direct per-job zero-notify proof. test_handle_jobdispatch_terminal_no_notify_no_leak (extending 4.1's file, the one with the right entry point) closes this by combining EXIT resolution + a direct single-job zero-notify assertion + the canary-reference no-leak proof, all through the real handle_jobdispatch() entry point in one test. Pattern worth remembering: a batch-aggregate test and a bypassed-entry-point test can each look like they cover a scenario while neither actually does, in combination, what a later task's literal wording asks for ("all pass IN THE SAME TEST RUN").
• Task 5.4 (final task — spec complete): fixed test_commit_barrier_reply.c's test_jobdispatch_receipt_ack_no_wal_append (owned by ha-commit-barrier-rollout), the last remaining stale assertion from this feature's own changes. WAL-count delta corrected from "must be zero" to lsn_before + 1 (the batch's successful dispatch now appends one WAL_JOB_DISPATCH fact; mark_job_fail() was traced and confirmed to append nothing via any path). Notify-count: rather than degrading to "must be zero" (which would also silently kill the test's own reply-before-notify ordering assertion, since there'd be nothing left to order against), the batch's terminal-failure job was replaced with a genuinely retryable one (WORKER_NULL site, unregistered host, new job id) — preserves the test's ability to meaningfully exercise notify_dispatch()'s wiring through handle_jobdispatch(). Independently reviewed and approved, including scrutiny of this specific choice.
• Task 5.4: discovered and logged bug-1632 (scheduler suite) — a new sibling instance of the already-known bug-1598 pattern (unjoined sim_picker_thread_fn background thread reading a stack-local fixture after frame reuse), this time in test_sim_verify.c's own separate run_sim_and_capture_stdout() helper (bug-1598's original fix only covered 3 stack-locals in test_sim_event_loop.c, not this one). Confirmed via git log/git diff as untouched by this feature and confirmed non-deterministic (multiple clean re-runs). Pre-existing, unrelated, NOT fixed — a candidate for whoever next touches test_sim_verify.c's fixture pattern.
• Task 5.4: bug-1429's hang did NOT reproduce during this task's full-suite runs (multiple clean 1133-test passes) — consistent with its own buglog history describing it as non-deterministic/timing-dependent, not evidence it's resolved. Still treat any future meson test TIMEOUT on test_commit_barrier_reply/test_commit_barrier_rollout as this known, pre-existing, unrooted issue rather than a new regression, per all of tasks 3.1-5.3's own prior confirmations.
• Feature complete: all tasks (1.1-1.3, 2.1-2.2, 3.1-3.3, 4.1, 5.1-5.4) implemented, independently reviewed and approved, and committed. Final regression: scheduler suite at documented baseline (51/52, 1 pre-existing unrelated failure), controller suite clean (1133 tests, only 2 pre-existing bug-1561 failures), sequential HA soak PASS (10 rounds, zero job loss, all 3 nodes led), chaotic HA soak PASS (10 rounds, committed_divergence=0 committed_job_loss=0) — independently re-verified by task 5.4's reviewer via its own rebuild plus a 5-round spot-check of both soak modes. bug-1624 and bug-1625 (this spec's originating bugs) are fixed by tasks 3.1/2.1 and 2.2 respectively; .wolf/buglog.json should be updated to reflect FIXED status as a follow-up.