myxce/.kiro/specs/scheduler-dispatch-commit/tasks.md

# Implementation Plan

- [ ] 1. Foundation: controller-side dispatch-failure classification and durable transition record
- [x] 1.1 (P) Define the dispatch-failure classification used to decide whether a failed dispatch attempt should retry or fail permanently
  - Enumerate each distinct way a dispatch attempt to a worker can fail today (worker lookup, worker unavailable, worker not ready, invalid allocation, queue configuration lookup, the dispatch request itself failing), and classify each as either retryable (transient/environmental) or terminal (a locally invalid decision that retrying will not fix)
  - The classification is a pure decision with no side effects, independently testable against every known failure case without needing a live dispatch attempt
  - Observable completion: a standalone test exercises every classified failure case and asserts the expected retryable/terminal outcome
  - _Requirements: 4.1, 4.2_
  - _Boundary: Controller Dispatch-Failure Classification (site enum + classify function)_

- [x] 1.2 (P) Add a durable record of a job's dispatch-time transition to a running state
  - Introduce a new durable fact type recording that a job has transitioned to running state at dispatch-commit time, reusing the existing shape used for other job-state facts
  - Wire this new fact type into both the normal live-commit apply path and the recovery/restart replay path, so the transition is recovered identically whether observed live or replayed after a restart
  - Observable completion: appending this fact and then replaying it (simulating a restart) leaves the job's state as running, without requiring the separate worker-confirmed-start fact to also be present
  - _Requirements: 2.4_
  - _Boundary: WAL Dispatch Fact_

- [x] 1.3 (P) Track how many consecutive times a job's dispatch has failed for a retryable reason
  - Add an in-memory counter on the controller's job record that increments on each retryable dispatch failure and resets to zero on a successful dispatch
  - This counter does not need to survive a controller failover — resetting to zero after a failover is an acceptable, safe degradation
  - Observable completion: the counter can be incremented, read, and reset independently of any actual dispatch attempt
  - _Requirements: 4.3_
  - _Boundary: Controller Dispatch-Failure Classification (retry-counter data model)_

- [ ] 2. Core: scheduler transitions a job to running state at dispatch-commit time
  - No dependency on Task 1 — scheduler-side files only, independent of the controller-side foundation work above.
- [x] 2.1 (P) Scheduler marks a job as running in the same operation that commits its dispatch decision
  - When the scheduler commits a decision to dispatch a job, the job's state becomes running immediately, in the same operation that already records the job's resource allocation — not waiting for any later confirmation
  - The scheduler's existing resource-in-use accounting, which only counts jobs already in a running state, requires no changes — this task provides the input it was missing
  - Observable completion: a job dispatched in one round appears in resource-in-use totals starting that same round, and continues to appear in every subsequent round regardless of how many rounds pass before any confirmation arrives
  - _Requirements: 1.1, 1.2, 1.3_
  - _Boundary: Scheduler Dispatch Commit_

- [x] 2.2 (P) Scheduler processes a reported dispatch failure by returning the job to a pending state
  - When the scheduler is told a job's dispatch failed, it returns that job to a pending state instead of silently ignoring the report as it does today
  - Returning a job to pending correctly releases the resource-allocation data that was tentatively committed for it, reusing the scheduler's existing behavior for any other transition away from a running state
  - Observable completion: a job whose dispatch failure is reported to the scheduler is again eligible for the scheduler's own job-selection process in a subsequent round, with no dangling resource-allocation data left behind
  - _Requirements: 3.1, 3.2, 4.4_
  - _Boundary: Scheduler Dispatch-Failure Handler_

- [ ] 3. Core: controller transitions a job to running state only once dispatch to the worker succeeds, and resolves failures
  - Sub-tasks 3.1-3.3 all restructure the same function and share its per-job lock scope, so none of them are marked parallel to each other, regardless of how disjoint their individual concerns look.
- [x] 3.1 Controller commits a job's resource allocation and running-state transition together, only on a successful dispatch
  - The controller's existing (currently unconditional) commit of a job's resource-allocation data becomes conditional on the dispatch request to the worker actually succeeding; the job's state transition to running happens in that same successful outcome, protected against concurrent access to the job record
  - Record the new durable dispatch-transition fact (Task 1.2) as part of this same successful outcome
  - Reset the consecutive-retryable-failure counter (Task 1.3) to zero as part of this same successful outcome, so a job that later fails again starts counting from zero rather than retaining count from before its last success
  - The worker's own later confirmation that the job actually started continues to be recorded exactly as it already is today, unaffected by this change
  - Observable completion: querying a job's status immediately after a successful dispatch shows it as running, and that status is unaffected by whether the worker has separately confirmed the job started yet; a job that fails, succeeds, then fails again is evaluated against the bound as if it had only failed once
  - _Requirements: 2.1, 2.2, 2.3, 2.4, 5.1_
  - _Depends: 1.2, 1.3_

- [x] 3.2 Controller resolves a terminal (locally-invalid) dispatch failure
  - When a dispatch attempt fails for a reason the classification (Task 1.1) marks terminal, the controller resolves the job to a permanently failed state immediately, protected against concurrent access to the job record, and releases any resource-allocation data tentatively associated with the attempt
  - This covers the two failure sites that already release the job reference they hold today — confirm that behavior is preserved, not duplicated, by this new resolution step
  - Observable completion: a terminally-classified dispatch failure leaves the job permanently failed, with no partial allocation data left behind and the job's reference count unchanged from before the attempt
  - _Requirements: 4.2, 4.5_
  - _Depends: 1.1_

- [x] 3.3 Controller resolves a retryable (transient/environmental) dispatch failure, with bounded escalation
  - When a dispatch attempt fails for a reason the classification (Task 1.1) marks retryable, the controller increments the consecutive-failure counter (Task 1.3) under the same protected access; if the incremented count is still within a bounded limit, it reports the failure so the job can be retried, otherwise it resolves the job to a permanently failed state exactly as Task 3.2 does for a terminal failure
  - This covers the four failure sites that do not currently release the job reference they hold — fix each to release it as part of this same resolution step (Task 3.2's two sites already release theirs correctly and are unaffected)
  - Observable completion: a retryable dispatch failure below the bound leaves the job eligible for another attempt with its reference count restored to the pre-attempt value; a retryable failure that pushes the count past the bound resolves the job to permanently failed instead
  - _Requirements: 4.1, 4.3, 4.4, 4.5_
  - _Depends: 1.1, 1.3, 3.2_

- [ ] 4. Integration: caller no longer duplicates dispatch-outcome reporting
- [x] 4.1 Remove the caller's own dispatch-outcome reporting now that resolution owns it
  - The function that invokes a dispatch attempt for each job in a batch no longer separately reports the outcome itself — reporting is now entirely owned by the dispatch resolution added in Task 3, preventing a failed dispatch from ever producing two different, conflicting outcome reports for the same job
  - Observable completion: a dispatch failure that resolves as permanent produces exactly one outcome report, not two conflicting ones
  - _Requirements: 3.1, 4.1, 4.2_
  - _Depends: 3.1, 3.2, 3.3_

- [ ] 5. Validation: end-to-end behavior and regression
- [ ] 5.1 End-to-end test: resource-in-use accounting survives a multi-round gap between dispatch and confirmation
  - Reproduce bug-1624 directly: dispatch a job, let several rounds pass before the worker confirms it started, and confirm resource-in-use totals reflect the job continuously from the round it was dispatched, matching what a same-round confirmation would have produced
  - Observable completion: the regression scenario passes, and fails against the pre-change code path if temporarily reverted
  - _Requirements: 1.1, 1.2, 1.3, 2.1, 2.3_
  - _Depends: 2.1, 3.1_

- [ ] 5.2 End-to-end test: retryable and bounded-retry dispatch failures resolve correctly
  - Simulate a transient dispatch failure and confirm the job becomes eligible for another attempt; simulate enough consecutive transient failures to exceed the bound and confirm the job resolves to permanently failed instead of retrying indefinitely; simulate a failure, a success, then another failure and confirm the bound is evaluated as if only one failure had occurred
  - Observable completion: all three scenarios produce the documented outcome, with the counter's bound value exercised at exactly the boundary (one under, exactly at, and one over)
  - _Requirements: 3.1, 3.2, 4.1, 4.3, 4.4, 4.5_
  - _Depends: 2.2, 3.3_

- [ ] 5.3 End-to-end test: terminal dispatch failure resolves without duplicate reporting or leaked references
  - Simulate a locally-invalid dispatch decision and confirm the job resolves to permanently failed, that no retry-eligible report is also produced for the same failure, and that the job record's reference count returns to its pre-attempt value
  - Observable completion: the assertions above all pass in the same test run
  - _Requirements: 4.2, 4.5_
  - _Depends: 3.2, 4.1_

- [ ] 5.4 Full regression: existing suites and HA soak
  - Run the existing scheduler and controller test suites end to end, updating only the assertions that directly depended on the old confirmation-gated timing; run a full sequential and chaotic HA soak
  - Observable completion: both test suites pass and the soak shows no new job loss, state divergence, or liveness stall attributable to this feature
  - _Requirements: 5.2, 5.3, 5.4_
  - _Depends: 5.1, 5.2, 5.3_

## Implementation Notes
- Task 1.1: design.md's "Controller Dispatch-Failure Classification" Service Interface Preconditions text claims a default-less `switch` over `xce_dispatch_fail_site_t` gives "compiler-enforced exhaustiveness... fails to compile" for an unhandled case — verified false for plain C without `-Werror` (this project sets none, confirmed in `src/meson.build`/`src/controller/meson.build`). Implemented instead with a `default:` arm that ERROR-logs and returns a safe `XCE_DISPATCH_FAIL_TERMINAL`, with `assert(0 && ...)` gated to `#ifdef XCE_TESTING` only — mirrors the established, verified-matching precedent `persist.c`'s `wal_classify()` already uses (`persist.c:6030-6037`) for the same "never risk silent misbehavior on an unrecognized case" concern. Any later task/spec touching this same design.md section should correct its wording rather than re-trusting the "fails to compile" claim.
- Tasks 3.2/3.3 (retryable/terminal resolution, both call `xce_classify_dispatch_fail()`) should follow the same fail-loud `default:` convention if they add their own exhaustive switches (e.g. over `xce_dispatch_fail_class_t`'s two values).
- Operational note (not spec-relevant): implementer/reviewer subagents in this run consistently ended their turn mid-build (after starting `xce-build.sh controller`, before it finished) rather than blocking on it, requiring one resume-and-nudge each time. Future task dispatches in this run should explicitly instruct agents to poll/block synchronously until the build genuinely completes before composing their status/verdict block, to save round-trips.
- Task 1.2: `apply_wal_entry()` (persist.c) is confirmed to have ZERO production call sites today — `persist_recover()`'s actual boot-time replay path (`replay_wal()`) and the standby catch-up path (`ha_standby_advance_committed_and_apply()`, post the 2026-07-10 `ha_apply_single_entry` refactor) both go through `apply_committed_entry()` exclusively. design.md's "WAL Dispatch Fact" component (its "Preconditions"-adjacent prose) still characterizes `apply_wal_entry()` as "the recovery/replay-time path reading WAL segments from disk after restart" — that's stale. Adding `WAL_JOB_DISPATCH` to both functions' fall-through groups was still correct (matches every sibling `WAL_JOB_*` type's identical dual-registration, harmless dead-code parity) and is NOT something to undo — but design.md's wording should be corrected before any later task cites `apply_wal_entry()`'s role from that doc without re-verifying.
- Task 3.1: `xce_request()`/every `xce_*_req()` wrapper (`src/lib/request.c`) is genuinely non-blocking fire-and-forget (capnp-encode + `endpoint_enq()` onto `ep->outq`, returns immediately — never waits for a reply). `endpoint_enq()` succeeds unconditionally against a zeroed `endpoint_t` with `ep->iomgr == NULL`. Net: testing a controller-side `dispatch_a_job_()` success path needs no live peer/socketpair/round-trip thread — a zeroed static `endpoint_t` (`.fd = -1`, no iomgr) is sufficient, matching `test_commit_barrier_reply.c`'s own established `g_wkr_disp_ep` fixture precedent. Reused directly for `test_dispatch_commit.c`; any later task exercising `dispatch_a_job_()`/`xce_startjob_req()` in a test should use the same pattern rather than building real network mocking.
- Task 3.1: **`test_commit_barrier_reply.c`'s `test_jobdispatch_receipt_ack_no_wal_append`** (a different, `ha-commit-barrier-rollout`-owned pinned test) asserts a successful dispatch cycle appends ZERO WAL entries — now false, since a successful dispatch appends exactly one `WAL_JOB_DISPATCH` fact (Req 2.4). Confirmed by both task 3.1's implementer and its independent reviewer; this is explicitly the kind of stale assertion Requirement 5.3 / task 5.4 exists to update, deliberately left unfixed here (out of 3.1's boundary, and that file's tests were unreachable in a full-suite run anyway due to the bug-1429 hang below, registered earlier in `controller_test.c`'s `main()`). **Task 5.4 must update this assertion** before claiming full-suite regression-clean.
- Task 3.1: reconfirmed **bug-1429** (`controller_unit_test` TIMEOUT/hang, pre-existing, unrooted) a 3rd time, now hanging inside `test_commit_barrier_reply.c`'s `test_jobsubmit_repeated_truncation_forced_failures_no_phantom_jobs` — a 3rd distinct hang location across 3 sessions, confirmed unrelated to `dispatch_a_job_()` (that test never calls it) and independently reproduced twice more (standalone binary run past the meson harness's 300s cap, and a fully separate rebuild). Because this hang sits earlier in `controller_test.c`'s registration order than most of this feature's own new test files, a full un-filtered `meson test` run will NOT reach them — **task 5.4's full-suite validation must run with the hanging group filtered out (`controller_unit_test -t <group>`) or budget for the 300s timeout**, not treat a bare `meson test` TIMEOUT as this feature's own failure.
- Task 3.1: confirmed pre-existing (not introduced by this task, not in this task's boundary to fix) — `dispatch_a_job_()`'s success path never calls `dereference(j)` on the reference taken at lookup; identical in the diff's before/after, and the sole caller `handle_jobdispatch()` never dereferences either. A candidate future bug-log entry, out of scope for this spec.
- Task 3.1: when validating a build against this suite's ASAN leak output, always replicate the project's own `ASAN_OPTIONS=detect_leaks=0` test env (`meson.build:229`) — a standalone binary run without it produces spurious leak reports (a red herring, not a regression) that can waste review time.
- Task 3.2: `job_t`'s real reference count lives in `jobj.c`'s file-local `job_private_t`, with no accessor exposed anywhere in the codebase — a test proving no reference-count regression can't do a direct numeric comparison. The working pattern (used by `test_dispatch_fail_terminal.c`) is a "canary reference": take one extra manual `reference()` on the job **immediately before** the call under test (not after — arming it after only relies incidentally on other assertions crashing first, which doesn't independently prove anything), then `dereference()` it after the call and confirm the job is still safely readable — an over-release during the call frees the job while the canary is still held, so the post-call touch is a real ASAN-caught heap-use-after-free.
- Task 3.2: reconfirmed bug-1429's hang again (still `test_commit_barrier_reply.c`, still unrelated to any change in this feature) — third+ reproduction, still not root-caused. Continues to require `controller_unit_test -t <group>` filtering for a clean isolated regression signal.
- Task 3.3: bug-1627 (the 4-site reference leak on `dispatch_a_job_()`'s retryable failure paths) is now FIXED — `dereference(j)` added at all four retryable sites, closing it as a direct consequence of this task's resolution restructuring. `.wolf/buglog.json` updated accordingly.
- Task 3.3: `notify_dispatch()` is `static` and defined AFTER `dispatch_a_job_()` in job.c with no forward declaration — needed one added near the top (mirroring `mark_job_fail`'s existing forward-decl precedent at job.c:24) so the new retryable-resolution code could call it. Any later task in this file needing to call a same-file function defined later in the file should check for/add a forward declaration the same way, rather than reordering definitions.
- Task 3.3: capturing a site's natural error code (`rc`/`errno`) into a stable local must happen BEFORE any intervening call once more logic is inserted before the eventual `return` — `errno` is not guaranteed stable across `lock()`/`ERROR()`/`free_launchinfo()` etc. This bit the worker-null and worker-not-ready sites specifically (both originally read `errno` directly at the `return` statement with nothing in between; task 3.3 had to capture it immediately after `SET_ERROR()`/at branch entry, before inserting the new lock/classify/resolve sequence).
- Task 3.3: `test_commit_barrier_reply.c` (a file otherwise owned by the `ha-commit-barrier-rollout` spec) is this binary's sole host of the `notify()` stub — any future task needing to observe `notify_dispatch()`'s calls in a controller unit test must add accessors there too (purely additive, read-only `_ext`-suffixed pass-throughs to its existing static capture globals — do not modify the stub body itself or its existing pinned tests). Confirmed as legitimate shared test infrastructure, not a boundary violation, by task 3.3's independent review.
- Task 3.3: the bound constant is `XCE_DISPATCH_FAIL_RETRY_BOUND = 3` (design.md's stated default), defined once near job.c's top and used at all four retryable sites — no magic numbers. 3 consecutive failures still retry; the 4th resolves terminal.
- Task 4.1: **`test_commit_barrier_reply.c`'s `test_jobdispatch_receipt_ack_no_wal_append` is now stale for a SECOND, independent reason**, on top of task 3.1's already-documented WAL-append-count staleness: its 2-job batch (one real-worker success, one `nhosts==0` terminal failure — no retryable job) now produces ZERO `notify_dispatch()` calls total post-task-4.1 (the success path never called it even before task 4.1, per task 3.1's design; the terminal path resolves via `mark_job_fail()`'s `schedFailJob` hook chain, not `notify_dispatch()`/`jobDispatchFail`, per task 3.2's design) — breaking its pinned `g_notify_count == 2` / both-`jobDispatchFail`-event assertions (lines ~1991-2003). **Task 5.4 must update BOTH the WAL-count assertion (task 3.1's finding) and the notify-count/event assertions (this finding) in the same test** before claiming full-suite regression-clean; the test's own scenario (1 success + 1 terminal, zero retryable jobs) may need to change too since it can no longer meaningfully exercise `notify_dispatch()` at all with today's cast of outcomes — consider adding a retryable-classified job to the batch if the intent is still to prove `notify_dispatch()`'s wiring through `handle_jobdispatch()`.
- Task 4.1: confirmed `handle_jobdispatch()`'s only caller (`cluster.c:471`, inside `myreq_handler`) uses its return value only as the generic RPC-handler `rc`, and `handle_jobdispatch()` itself always returns 0 unconditionally regardless of any job's outcome — nothing depends on the removed per-job notification behavior at the caller level.