myxce/.kiro/specs/scheduler-dispatch-commit/tasks.md

Implementation Plan

• 1. Foundation: controller-side dispatch-failure classification and durable transition record

• 1.1 (P) Define the dispatch-failure classification used to decide whether a failed dispatch attempt should retry or fail permanently

  • Enumerate each distinct way a dispatch attempt to a worker can fail today (worker lookup, worker unavailable, worker not ready, invalid allocation, queue configuration lookup, the dispatch request itself failing), and classify each as either retryable (transient/environmental) or terminal (a locally invalid decision that retrying will not fix)
  • The classification is a pure decision with no side effects, independently testable against every known failure case without needing a live dispatch attempt
  • Observable completion: a standalone test exercises every classified failure case and asserts the expected retryable/terminal outcome
  • Requirements: 4.1, 4.2
  • Boundary: Controller Dispatch-Failure Classification (site enum + classify function)

• 1.2 (P) Add a durable record of a job's dispatch-time transition to a running state

  • Introduce a new durable fact type recording that a job has transitioned to running state at dispatch-commit time, reusing the existing shape used for other job-state facts
  • Wire this new fact type into both the normal live-commit apply path and the recovery/restart replay path, so the transition is recovered identically whether observed live or replayed after a restart
  • Observable completion: appending this fact and then replaying it (simulating a restart) leaves the job's state as running, without requiring the separate worker-confirmed-start fact to also be present
  • Requirements: 2.4
  • Boundary: WAL Dispatch Fact

• 1.3 (P) Track how many consecutive times a job's dispatch has failed for a retryable reason

  • Add an in-memory counter on the controller's job record that increments on each retryable dispatch failure and resets to zero on a successful dispatch
  • This counter does not need to survive a controller failover — resetting to zero after a failover is an acceptable, safe degradation
  • Observable completion: the counter can be incremented, read, and reset independently of any actual dispatch attempt
  • Requirements: 4.3
  • Boundary: Controller Dispatch-Failure Classification (retry-counter data model)

• 2. Core: scheduler transitions a job to running state at dispatch-commit time

  • No dependency on Task 1 — scheduler-side files only, independent of the controller-side foundation work above.

• 2.1 (P) Scheduler marks a job as running in the same operation that commits its dispatch decision

  • When the scheduler commits a decision to dispatch a job, the job's state becomes running immediately, in the same operation that already records the job's resource allocation — not waiting for any later confirmation
  • The scheduler's existing resource-in-use accounting, which only counts jobs already in a running state, requires no changes — this task provides the input it was missing
  • Observable completion: a job dispatched in one round appears in resource-in-use totals starting that same round, and continues to appear in every subsequent round regardless of how many rounds pass before any confirmation arrives
  • Requirements: 1.1, 1.2, 1.3
  • Boundary: Scheduler Dispatch Commit

• 2.2 (P) Scheduler processes a reported dispatch failure by returning the job to a pending state

  • When the scheduler is told a job's dispatch failed, it returns that job to a pending state instead of silently ignoring the report as it does today
  • Returning a job to pending correctly releases the resource-allocation data that was tentatively committed for it, reusing the scheduler's existing behavior for any other transition away from a running state
  • Observable completion: a job whose dispatch failure is reported to the scheduler is again eligible for the scheduler's own job-selection process in a subsequent round, with no dangling resource-allocation data left behind
  • Requirements: 3.1, 3.2, 4.4
  • Boundary: Scheduler Dispatch-Failure Handler

• 3. Core: controller transitions a job to running state only once dispatch to the worker succeeds, and resolves failures

  • Sub-tasks 3.1-3.3 all restructure the same function and share its per-job lock scope, so none of them are marked parallel to each other, regardless of how disjoint their individual concerns look.

• 3.1 Controller commits a job's resource allocation and running-state transition together, only on a successful dispatch

  • The controller's existing (currently unconditional) commit of a job's resource-allocation data becomes conditional on the dispatch request to the worker actually succeeding; the job's state transition to running happens in that same successful outcome, protected against concurrent access to the job record
  • Record the new durable dispatch-transition fact (Task 1.2) as part of this same successful outcome
  • Reset the consecutive-retryable-failure counter (Task 1.3) to zero as part of this same successful outcome, so a job that later fails again starts counting from zero rather than retaining count from before its last success
  • The worker's own later confirmation that the job actually started continues to be recorded exactly as it already is today, unaffected by this change
  • Observable completion: querying a job's status immediately after a successful dispatch shows it as running, and that status is unaffected by whether the worker has separately confirmed the job started yet; a job that fails, succeeds, then fails again is evaluated against the bound as if it had only failed once
  • Requirements: 2.1, 2.2, 2.3, 2.4, 5.1
  • Depends: 1.2, 1.3

• 3.2 Controller resolves a terminal (locally-invalid) dispatch failure

  • When a dispatch attempt fails for a reason the classification (Task 1.1) marks terminal, the controller resolves the job to a permanently failed state immediately, protected against concurrent access to the job record, and releases any resource-allocation data tentatively associated with the attempt
  • This covers the two failure sites that already release the job reference they hold today — confirm that behavior is preserved, not duplicated, by this new resolution step
  • Observable completion: a terminally-classified dispatch failure leaves the job permanently failed, with no partial allocation data left behind and the job's reference count unchanged from before the attempt
  • Requirements: 4.2, 4.5
  • Depends: 1.1

• 3.3 Controller resolves a retryable (transient/environmental) dispatch failure, with bounded escalation

  • When a dispatch attempt fails for a reason the classification (Task 1.1) marks retryable, the controller increments the consecutive-failure counter (Task 1.3) under the same protected access; if the incremented count is still within a bounded limit, it reports the failure so the job can be retried, otherwise it resolves the job to a permanently failed state exactly as Task 3.2 does for a terminal failure
  • This covers the four failure sites that do not currently release the job reference they hold — fix each to release it as part of this same resolution step (Task 3.2's two sites already release theirs correctly and are unaffected)
  • Observable completion: a retryable dispatch failure below the bound leaves the job eligible for another attempt with its reference count restored to the pre-attempt value; a retryable failure that pushes the count past the bound resolves the job to permanently failed instead
  • Requirements: 4.1, 4.3, 4.4, 4.5
  • Depends: 1.1, 1.3, 3.2

• 4. Integration: caller no longer duplicates dispatch-outcome reporting

• 4.1 Remove the caller's own dispatch-outcome reporting now that resolution owns it

  • The function that invokes a dispatch attempt for each job in a batch no longer separately reports the outcome itself — reporting is now entirely owned by the dispatch resolution added in Task 3, preventing a failed dispatch from ever producing two different, conflicting outcome reports for the same job
  • Observable completion: a dispatch failure that resolves as permanent produces exactly one outcome report, not two conflicting ones
  • Requirements: 3.1, 4.1, 4.2
  • Depends: 3.1, 3.2, 3.3

• 5. Validation: end-to-end behavior and regression

• 5.1 End-to-end test: resource-in-use accounting survives a multi-round gap between dispatch and confirmation

  • Reproduce bug-1624 directly: dispatch a job, let several rounds pass before the worker confirms it started, and confirm resource-in-use totals reflect the job continuously from the round it was dispatched, matching what a same-round confirmation would have produced
  • Observable completion: the regression scenario passes, and fails against the pre-change code path if temporarily reverted
  • Requirements: 1.1, 1.2, 1.3, 2.1, 2.3
  • Depends: 2.1, 3.1

• 5.2 End-to-end test: retryable and bounded-retry dispatch failures resolve correctly

  • Simulate a transient dispatch failure and confirm the job becomes eligible for another attempt; simulate enough consecutive transient failures to exceed the bound and confirm the job resolves to permanently failed instead of retrying indefinitely; simulate a failure, a success, then another failure and confirm the bound is evaluated as if only one failure had occurred
  • Observable completion: all three scenarios produce the documented outcome, with the counter's bound value exercised at exactly the boundary (one under, exactly at, and one over)
  • Requirements: 3.1, 3.2, 4.1, 4.3, 4.4, 4.5
  • Depends: 2.2, 3.3

• 5.3 End-to-end test: terminal dispatch failure resolves without duplicate reporting or leaked references

  • Simulate a locally-invalid dispatch decision and confirm the job resolves to permanently failed, that no retry-eligible report is also produced for the same failure, and that the job record's reference count returns to its pre-attempt value
  • Observable completion: the assertions above all pass in the same test run
  • Requirements: 4.2, 4.5
  • Depends: 3.2, 4.1

• 5.4 Full regression: existing suites and HA soak

  • Run the existing scheduler and controller test suites end to end, updating only the assertions that directly depended on the old confirmation-gated timing; run a full sequential and chaotic HA soak
  • Observable completion: both test suites pass and the soak shows no new job loss, state divergence, or liveness stall attributable to this feature
  • Requirements: 5.2, 5.3, 5.4
  • Depends: 5.1, 5.2, 5.3

Implementation Notes

• Task 1.1: design.md's "Controller Dispatch-Failure Classification" Service Interface Preconditions text claims a default-less switch over xce_dispatch_fail_site_t gives "compiler-enforced exhaustiveness... fails to compile" for an unhandled case — verified false for plain C without -Werror (this project sets none, confirmed in src/meson.build/src/controller/meson.build). Implemented instead with a default: arm that ERROR-logs and returns a safe XCE_DISPATCH_FAIL_TERMINAL, with assert(0 && ...) gated to #ifdef XCE_TESTING only — mirrors the established, verified-matching precedent persist.c's wal_classify() already uses (persist.c:6030-6037) for the same "never risk silent misbehavior on an unrecognized case" concern. Any later task/spec touching this same design.md section should correct its wording rather than re-trusting the "fails to compile" claim.
• Tasks 3.2/3.3 (retryable/terminal resolution, both call xce_classify_dispatch_fail()) should follow the same fail-loud default: convention if they add their own exhaustive switches (e.g. over xce_dispatch_fail_class_t's two values).
• Operational note (not spec-relevant): implementer/reviewer subagents in this run consistently ended their turn mid-build (after starting xce-build.sh controller, before it finished) rather than blocking on it, requiring one resume-and-nudge each time. Future task dispatches in this run should explicitly instruct agents to poll/block synchronously until the build genuinely completes before composing their status/verdict block, to save round-trips.
• Task 1.2: apply_wal_entry() (persist.c) is confirmed to have ZERO production call sites today — persist_recover()'s actual boot-time replay path (replay_wal()) and the standby catch-up path (ha_standby_advance_committed_and_apply(), post the 2026-07-10 ha_apply_single_entry refactor) both go through apply_committed_entry() exclusively. design.md's "WAL Dispatch Fact" component (its "Preconditions"-adjacent prose) still characterizes apply_wal_entry() as "the recovery/replay-time path reading WAL segments from disk after restart" — that's stale. Adding WAL_JOB_DISPATCH to both functions' fall-through groups was still correct (matches every sibling WAL_JOB_* type's identical dual-registration, harmless dead-code parity) and is NOT something to undo — but design.md's wording should be corrected before any later task cites apply_wal_entry()'s role from that doc without re-verifying.