myxce/.kiro/specs/scheduler-dispatch-commit/design.md

Design Document

Overview

Purpose: Move a job's transition to a running state from worker-confirmation time to dispatch-decision-commit time, on both the scheduler and the controller, and build the dispatch-failure retry/terminal-resolution machinery this requires — closing bug-1624 (resource-in-use totals silently drop a dispatched-but-unconfirmed job at the next round boundary) and bug-1625 (the scheduler discards the controller's existing dispatch-failure notification).

Users: Cluster operators who rely on accurate live resource-in-use accounting and correct job status; scheduler/controller maintainers who need dispatch failures to resolve deterministically instead of leaking stale allocation state.

Impact: sched_commit_decision() (scheduler) and dispatch_a_job_() (controller) each gain a state transition alongside the resource-allocation commit they already perform. A new scheduler-side notify handler and a new controller-side classification step are added. No existing component is replaced; both integration points are extended in place.

Goals

• A job's state becomes RUN at the moment its dispatch is committed, on both scheduler and controller, with no window where the two disagree.
• sched_account_jobs()'s existing RUN-state gate — unmodified — naturally includes a dispatched job from the round it was dispatched.
• A dispatch failure resolves deterministically: transient/environmental failures retry the job back to PEND (bounded), local/invalid-decision failures fail the job permanently. Both outcomes release the stale allocation/commitment data from the failed attempt, on both sides.
• The worker's actual confirmed-start fact is still durably recorded, unchanged, for start_time and audit purposes.

Non-Goals

• Changing simulation-mode's own dispatch/commit path — sched_commit_decision() is not called by the simulation driver at all (confirmed in Research Log); no code change is needed there.
• Implementing preemption delivery or victim selection — scheduler-preemption owns that; this design only leaves its cross-reference note intact.
• Changing scheduler-allocation's own allocation-plan logic — this design only changes when a job reaches RUN, not how a plan is chosen.
• Redesigning the WAL entry header format or the raft-apply-decoupling deferred-apply mechanism — this design adds one new WAL fact type using the existing wal_jobstate_t shape and the existing synchronous-mutation discipline job.allocs already follows, not a new mechanism.

Boundary Commitments

This Spec Owns

• The scheduler-side state transition inside sched_commit_decision()'s DISPATCH branch.
• The controller-side state transition inside dispatch_a_job_(), gated on xce_startjob_req() success, correctly synchronized under RTTI_LOCK_WRITE, and the new WAL fact type recording it durably.
• The controller-side dispatch-failure classification (retryable vs. terminal, including the bounded-retry check) and its two resolution paths.
• The new scheduler-side NotifyData_data_jobDispatchFail handler (the actual fix for bug-1625) — a single, narrow case that applies JOB_STAT_PEND.

Out of Boundary

• scheduler.c's sched_update()/sched_job_state_change() — confirmed dead in the live driver (Research Log); not modified, not reused, not removed (removal is a separate cleanup, not required for this feature to be correct).
• core_events.c's XCE_CORE_EVENT_JOB_STATE_CHANGE case — already correct and already idempotent for a repeated RUN application; unmodified.
• builtin_fcfs's picker hooks (on_dispatch, on_job_state_change) — unmodified; their existing idempotency is relied upon, not extended.
• The limit plugin's own usage accounting — confirmed to not share bug-1624's class of gate (Research Log); not investigated further here.
• mark_job_fail()'s own mutate-then-notify body and its existing HOOK_DETAIL_JOB_FAIL → schedFailJob chain — reused as-is for the terminal resolution path; not modified.
• Any change to xcecli/REST API response shapes — job status already reflects job.state directly; no new field is added there.
• Fixing bug-1626 at job_ctrl()'s own pre-existing (and unrelated to dispatch) call to mark_job_fail() (job.c:1083) — this design's new call sites correctly take RTTI_LOCK_WRITE themselves, but job_ctrl()'s own gap is a separate, pre-existing condition this spec does not fix as a drive-by patch.

Allowed Dependencies

• sched_commit_decision() (scheduler.c) and sched_account_jobs()'s existing JOB_IS_RUN gate (scheduler.c) — read, not modified, except for the one new transition line.
• mark_job_fail() (job.c) and its existing hook chain — called, not modified, under a caller-supplied RTTI_LOCK_WRITE (its own documented precondition).
• notify_dispatch() (job.c) — called, not modified (its meaning narrows by convention, not by signature change).
• persist_submit_job_fact_ex() (persist.c) and wal_jobstate_t (wal_types.h) — extended with one new WAL entry type, following the existing _pad-discriminated shape.
• job_t's in-memory-only wrapper fields (xce/types.h) — extended with one new in-memory-only counter, following submit_lsn's existing precedent.
• job_lock/job_unlock (src/lib/jobj.c) via the existing lock(j, RTTI_LOCK_WRITE)/unlock(j) calling convention already used correctly at most other job-mutation sites in job.c — this design's new mutation sites must follow the same convention, not the unlocked pattern dispatch_a_job_() and job_ctrl()'s existing mark_job_fail() call site currently use (bug-1626, pre-existing — see research.md's Addendum).

Revalidation Triggers

• Any change to sched_account_jobs()'s RUN-state gate itself (not just what feeds it).
• Any change to dispatch_a_job_()'s failure-point ordering or return codes — the classification table (Components, below) is keyed to today's exact failure points.
• Any change to mark_job_fail()'s hook chain or to HOOK_DETAIL_JOB_STOP/HOOK_DETAIL_JOB_FAIL's wiring to sched.c's notify translation.
• scheduler-preemption moving to design — its AC6/AC9 pending-revision note depends on this design's RUN-timing conclusion being final.
• Any future fix to bug-1626 (the pre-existing job_ctrl()/mark_job_fail() unlocked call site, out of this spec's boundary) that changes mark_job_fail()'s locking contract itself, as opposed to fixing individual callers.

Architecture

Existing Architecture Analysis

Two independently-maintained job records exist for the same logical job: the scheduler's xce_sched_job_t (sched/types.h, decision-making working copy) and the controller's job_t/xce_job_t (xce/types.h, the durable, user-facing record). They are today synchronized by two one-directional channels: scheduler → controller decision publish (sched_commit_decision() → xcecli_jobdispatch()), and controller → scheduler notify (sched_handle_notify()'s switch, handler.c). This design adds a state transition to each side's own record at its own commit point, rather than introducing a third synchronization channel — the two sides reach the same conclusion (RUN) independently, from the same triggering event (the dispatch decision), with the notify channel used only for the asymmetric case where the controller's downstream attempt fails after the scheduler already committed.

A second, load-bearing discovery (Research Log): scheduler.c's notify-apply path (sched_update()/sched_job_state_change()) is dead code in the current driver — the live path is core_events.c's XCE_CORE_EVENT_JOB_STATE_CHANGE case, reached via xce_sched_driver_translate_pending_events(). All references below to "the existing RUN-confirmation path" mean this live path.

Architecture Pattern & Boundary Map

sequenceDiagram
    participant Picker as Scheduler picker/allocator
    participant SchedCommit as sched_commit_decision
    participant Ctrl as Controller dispatch_a_job_ (owns resolution + notify)
    participant Worker as Worker
    participant Notify as Controller notify (existing)
    participant SchedNotify as Scheduler sched_handle_notify

    Picker->>SchedCommit: DISPATCH decision
    SchedCommit->>SchedCommit: set job-accounts, job-allocs, state=RUN
    SchedCommit-->>Ctrl: publish decision
    Ctrl->>Worker: xce_startjob_req
    alt RPC succeeds
        Worker-->>Ctrl: ok
        Ctrl->>Ctrl: lock j write, commit job-allocs, state=RUN, append WAL_JOB_DISPATCH fact, unlock
        Note over Ctrl: worker later confirms actual start<br/>WAL_JOB_START still recorded separately, unchanged
    else RPC fails, transient
        Ctrl->>Ctrl: increment dispatch_fail_count, stay PEND
        Ctrl->>Notify: jobDispatchFail
        Notify->>SchedNotify: NotifyData_data_jobDispatchFail
        SchedNotify->>SchedNotify: apply JOB_STAT_PEND (new case)
    else RPC fails, terminal or retry bound exceeded
        Ctrl->>Ctrl: lock j write, mark_job_fail (existing), unlock
        Ctrl->>Notify: schedFailJob (existing, unmodified path)
        Notify->>SchedNotify: NotifyData_data_schedFailJob
        SchedNotify->>SchedNotify: apply JOB_STAT_EXIT (existing, unmodified)
    end

Architecture Integration:

• Selected pattern: extend two existing, independent commit points rather than add a synchronization protocol. Reuses the existing asymmetric notify channel only for the failure case, where it is unavoidable (the scheduler cannot know the controller's downstream outcome any other way).
• Domain/feature boundaries: scheduler-side accounting correctness and controller-side dispatch/failure handling remain separately owned, matching the existing component split; neither gains knowledge of the other's internals beyond the existing decision-publish/notify contracts.
• Existing patterns preserved: sched_account_jobs()'s RUN-state gate, mark_job_fail()'s mutate-then-notify shape, wal_jobstate_t's _pad-discriminated WAL fact shape, job_t's in-memory-only wrapper-field convention.
• New components rationale: a classification step and a bounded counter are new because no equivalent existed; a new WAL fact type is new because the transition now needs a durable record that job.allocs's own (non-durable) precedent doesn't provide.
• Steering compliance: no controller self-termination path is introduced; the terminal-failure path stays entirely inside the existing "fail loud, mutate, notify" discipline mark_job_fail() already follows.

File Structure Plan

Modified Files

• src/scheduler/src/scheduler.c — sched_commit_decision()'s DISPATCH branch gains job->state = JOB_STAT_RUN;, alongside the existing job->allocs/job->accounts assignment. No new field.
• src/scheduler/src/handler.c — sched_handle_notify()'s switch gains one new case NotifyData_data_jobDispatchFail:, applying JOB_STAT_PEND through the existing XCE_SCHED_OP_JSTATE_CHANGE enqueue path (same shape as the adjacent schedExitJob/schedFailJob cases).
• src/controller/job.c — two coupled changes:
  • dispatch_a_job_()'s success/failure handling is restructured: job.allocs's commit block (today unconditional, job.c:947-954, run regardless of xce_startjob_req()'s rc) moves inside an if (rc >= 0) branch. On success, the function takes lock(j, RTTI_LOCK_WRITE), commits job.allocs/job.state, appends the WAL_JOB_DISPATCH fact, then unlock(j) — closing bug-1626 for this call site as a direct consequence of restructuring it. On xce_startjob_req() failure (XCE_DISPATCH_FAIL_SITE_RPC_FAILURE, one of the six sites Controller Dispatch-Failure Classification enumerates), the function follows that component's same classify-resolve procedure. At each of the six post-lookup failure points, the function classifies and resolves inline (under its own lock(j, RTTI_LOCK_WRITE)): retryable (four sites, including RPC failure) → increment dispatch_fail_count, unlock(j), dereference(j) (closing bug-1627's leak on these four paths), call notify_dispatch(); terminal (two sites) → call mark_job_fail() under the write lock, unlock(j) after (these two already correctly dereference(j) today, unaffected by bug-1627).
  • handle_jobdispatch()'s loop drops its existing per-job notify_dispatch(c, ep, alloc->jobid, ...) calls (both the success-path and failure-path ones) — notification is now entirely owned by dispatch_a_job_(), preventing the double-notify race found in round 2 of this review (a terminal failure previously would have produced both jobDispatchFail and schedFailJob).
• src/include/xce/types.h — xce_job_t's wrapper job_t (not the wire type itself) gains uint32_t dispatch_fail_count;, following submit_lsn's existing "in-memory only, reset on replay" precedent.
• src/include/wal_types.h — one new wal_entry_type value, WAL_JOB_DISPATCH = 12, reusing the existing wal_jobstate_t{jobid, state, _pad=0} payload shape (the "legacy: state is absolute" discipline, matching job.allocs's own synchronous-commit precedent — see Data Models).
• src/controller/persist.c — WAL_JOB_DISPATCH is added as a new case label to two existing fall-through groups: apply_wal_entry()'s (persist.c:3452-3466; currently has zero production call sites — see the Components section correction below) and apply_committed_entry()'s (the actual live-apply-and-recovery path, persist.c:3700-3706) — both already apply wal_jobstate_t.state as an absolute value for every type in their respective groups; no new function or case body.
• src/controller/job.c (same file, no new header) — xce_classify_dispatch_fail(), xce_dispatch_fail_site_t, and xce_dispatch_fail_class_t are declared static/file-local next to dispatch_a_job_(), its only caller — not in a shared header, since nothing outside this file consumes them (see Controller Dispatch-Failure Classification's placement-correction note).

New/Modified Test Files

• src/scheduler/testsuite/test_scheduler.c (or a new focused file) — sched_commit_decision() DISPATCH-branch RUN-transition.
• src/scheduler/testsuite/test_handler.c (or equivalent) — new jobDispatchFail case applies JOB_STAT_PEND, frees stale accounts/allocs via the existing shared path.
• src/controller/testsuite/test_job_dispatch.c (or equivalent existing dispatch test file) — gated commit on RPC success/failure for all seven failure returns (the jobid-lookup case's two variants unchanged, the six post-lookup cases newly classified); retryable vs. terminal classification; bounded-retry escalation; WAL fact durability across a simulated restart/replay.
• HA soak (/xce-test-ha) — no new file; existing sequential/chaotic regression re-run per Requirement 5.4.

System Flows

See the sequence diagram above (Architecture Pattern & Boundary Map) for the end-to-end flow across both success and both failure branches. No additional flow diagrams are needed — the state-transition logic itself is a simple two-branch decision per side, not a multi-step process.

Requirements Traceability

Requirement	Summary	Components	Interfaces	Flows
1.1, 1.2, 1.3	Scheduler RUN transition at commit time	Scheduler Dispatch Commit	sched_commit_decision	Sequence: SchedCommit branch
2.1, 2.3	Controller RUN transition at dispatch success	Controller Dispatch Commit	dispatch_a_job_	Sequence: RPC succeeds branch
2.2	Confirmed-start fact still recorded separately	Controller Dispatch Commit (unmodified sibling)	record_job_start_fact (unmodified)	n/a
2.4	Durability of the new transition	WAL Dispatch Fact	persist_submit_job_fact_ex, WAL_JOB_DISPATCH apply cases (live + recovery)	Sequence: RPC succeeds branch
3.1, 3.2	Scheduler processes dispatch-failure notify	Scheduler Dispatch-Failure Handler	sched_handle_notify new case	Sequence: transient branch
4.1, 4.2	Retryable vs. terminal classification	Controller Dispatch-Failure Classification	classification table	Sequence: both failure branches
4.3	Bounded retry count	Controller Dispatch-Failure Classification	dispatch_fail_count	Sequence: transient branch
4.4, 4.5	Release stale allocation/commitment data	Scheduler: Scheduler Dispatch-Failure Handler (retry), mark_job_fail (terminal, unmodified). Controller: satisfied vacuously — Controller Dispatch Commit never commits on any failure path, so there is nothing to release	core_events.c's existing free-on-non-RUN logic (scheduler side)	Sequence: both failure branches
5.1	Confirmed-start fact recorded unchanged	Controller Dispatch Commit (unmodified sibling)	n/a	n/a
5.2	Simulation mode unaffected	n/a (no simulation code touched)	n/a	Research Log
5.3, 5.4	Regression / HA soak	n/a (test-only)	n/a	n/a

Components and Interfaces

Component	Domain/Layer	Intent	Req Coverage	Key Dependencies (P0/P1)	Contracts
Scheduler Dispatch Commit	Scheduler / scheduler.c	Transition a job to RUN in the same operation that commits its dispatch decision	1.1-1.3	sched_get_job (P0), sched_charge_account (P0)	State
Scheduler Dispatch-Failure Handler	Scheduler / handler.c	Apply PEND to a job whose dispatch was reported as retryably failed	3.1, 3.2, 4.4	XCE_SCHED_OP_JSTATE_CHANGE enqueue (P0), core_events.c's existing apply case (P0)	Event
Controller Dispatch Commit	Controller / job.c	Transition a job to RUN, and durably record it, only once dispatch to the worker actually succeeds	2.1-2.4, 5.1	xce_startjob_req (P0), WAL Dispatch Fact (P0)	State, Batch
Controller Dispatch-Failure Classification	Controller / job.c	Decide retryable-vs-terminal for a dispatch failure, including the bounded-retry escalation	4.1-4.3	job_t.dispatch_fail_count (P0), mark_job_fail (P0), notify_dispatch (P0)	Service
WAL Dispatch Fact	Controller / persist.c, wal_types.h	Durable record of the dispatch-time RUN transition, replayed on recovery	2.4	persist_submit_job_fact_ex (P0), wal_jobstate_t (P0)	Batch, State

Scheduler (scheduler.c, handler.c)

Scheduler Dispatch Commit

Field	Detail
Intent	Set job->state = JOB_STAT_RUN in the same operation that commits job->allocs/job->accounts
Requirements	1.1, 1.2, 1.3

Responsibilities & Constraints

• Runs inside sched_commit_decision()'s existing XCE_DECISION_DISPATCH branch, after job->accounts is built and before sched_commit_job_alloc() charges it — ordering among these three lines is not load-bearing to each other, but all three must complete before this function returns.
• Does not call plugin_job_state_change()/on_job_state_change — the picker is already notified of this dispatch via the existing, unchanged on_dispatch() call in sched_allocator(). Firing a second hook is unnecessary and was confirmed unneeded via builtin_fcfs_on_dispatch()'s own pendJobs-removal behavior (Research Log).
• The later, now-redundant schedStartJob-triggered application of JOB_STAT_RUN (via the existing, unmodified core_events.c path) is a confirmed-safe no-op against an already-RUN job — no guard is added here to suppress it; suppressing it would be unnecessary defensive code against an already-idempotent path.

Contracts: State [x]

State Management

• State model: job->state transitions PEND → RUN synchronously, in-process, no deferred-apply (the scheduler has no WAL/durability concept — xce_sched_job_t is a working copy, not a system of record).
• Persistence & consistency: none — a scheduler restart already rebuilds xce_sched_job_t from a fresh controller push, which will already reflect whatever the controller's own (durable) state is by then.
• Concurrency strategy: unchanged — sched_commit_decision() already runs single-threaded on the driver's decision-drain loop; no new locking is introduced.

Scheduler Dispatch-Failure Handler

Field	Detail
Intent	The actual fix for bug-1625: process NotifyData_data_jobDispatchFail instead of falling to default: -EINVAL
Requirements	3.1, 3.2, 4.4

Responsibilities & Constraints

• New case NotifyData_data_jobDispatchFail: in sched_handle_notify()'s switch (handler.c), positioned alongside the existing schedStartJob/schedExitJob/schedFailJob cases.
• Calls the same enqueue helper those cases already use (sched_job_state_change(sched, notify->data.job_dispatch_fail->jobid, JOB_STAT_PEND) — the handler.c-local queuing wrapper, not the dead scheduler.c mutator of the same name), so the actual state application flows through the live xce_sched_driver_translate_pending_events() → core_events.c path identically to every other notify-driven state change.
• No new classification logic here — jobDispatchFail's meaning is narrowed (Research Log) to exclusively mean "return this job to pending"; the controller has already decided this is the right resolution before sending it.
• The existing core_events.c XCE_CORE_EVENT_JOB_STATE_CHANGE case's !JOB_IS_RUN(new_state) branch frees job->accounts/job->allocs automatically — this component does not duplicate that logic (Requirement 4.4 is satisfied by the existing, unmodified free-on-non-RUN path, reused rather than reimplemented).

Contracts: Event [x]

Event Contract

• Subscribed events: NotifyData_data_jobDispatchFail (existing capnp/notify type, already sent by notify_dispatch(), currently unhandled).
• Ordering / delivery guarantees: same as every other notify — delivered in-order on the same connection, applied on the next round's xce_sched_driver_translate_pending_events() drain (same latency characteristics as schedStartJob/schedExitJob/schedFailJob today).

Controller (job.c, persist.c)

Controller Dispatch Commit

Field	Detail
Intent	Commit job.allocs and transition to RUN only once xce_startjob_req() succeeds; append a durable fact recording the transition
Requirements	2.1, 2.3, 2.4, 5.1

Responsibilities & Constraints

• dispatch_a_job_()'s existing unconditional j->job.allocs = ... assignment (today placed after the xce_startjob_req() call but not gated on its return code) becomes conditional: only committed when rc >= 0. j->job.state = JOB_STAT_RUN is set in the same conditional block.
• This ordering choice supersedes requirements.md 2.1's literal "before attempting to notify the worker" phrasing (Research Log: gating on RPC success removes an entire reversal-path category that gating on decision-receipt would require). The observable guarantee requirements.md 2.1/2.3 actually asks for — the transition and the allocation commit happen as one atomic operation, and status queries reflect RUN once that operation completes — is preserved exactly; only the specific instant within dispatch_a_job_() moves later, to right after a successful RPC instead of right before the RPC attempt.
• On success, appends one WAL_JOB_DISPATCH fact (see WAL Dispatch Fact, below) recording {jobid, state=JOB_STAT_RUN}, in addition to the synchronous in-memory mutation — mirrors job.allocs's existing synchronous-commit style, not record_job_start_fact()'s deferred-apply style (Research Log: no same-job racing-fact concern exists for the first post-PEND transition).
• Requirement 2.2's worker-confirmed-start fact (record_job_start_fact()/WAL_JOB_START) is entirely unmodified — it continues to fire later, from the existing worker-confirmation code path, and continues to set start_time.
• Requirement 2.3 (status queries reflect RUN) requires no new code — job.state is read directly by every existing status/listing path today; committing it earlier is sufficient.

Contracts: State [x] / Batch [x]

State Management

• State model: PEND → RUN, gated on RPC success; on RPC failure, state is untouched (still PEND — never entered RUN, so no reversal is needed, per the Research Log finding).
• Persistence & consistency: the in-memory mutation and the WAL fact append happen back-to-back under lock(j, RTTI_LOCK_WRITE) (acquired explicitly by this component, held across both the job.allocs/job.state mutation and the persist_submit_job_fact_ex() call, released via unlock(j) immediately after) — _pad = 0 (legacy/absolute), matching wal_jobstate_t's documented convention for a mutation with no staleness window.
• Concurrency strategy: corrects a pre-existing gap (bug-1626), does not merely reuse it. dispatch_a_job_()'s existing locking, traced directly (job.c:831-963), takes only RTTI_LOCK_READ on c->jobmgr/c->workermgr/the worker w — it never takes any lock on the job object j itself, despite j->job.allocs already being mutated today. RTTI_LOCK_READ is a genuine pthread-backed reader lock (my_rwlock_rlock(), confirmed in src/lib/jobj.c:708-725), not a liveness/reference guard, so this is a real gap, not a documentation inaccuracy. This component's mutation (and the pre-existing job.allocs mutation it sits beside, which this restructuring already touches) is brought under RTTI_LOCK_WRITE, matching the convention most other job-mutating call sites in job.c already correctly follow (e.g. job.c:768,1687,1795,2058). job_ctrl()'s own separate, unrelated call to mark_job_fail() (job.c:1083) has the same gap and is intentionally left unfixed here (Out of Boundary) — only this feature's own new/restructured call sites are corrected.

Batch / Job Contract

• Trigger: successful xce_startjob_req() return within dispatch_a_job_().
• Input / validation: jobid, target state (JOB_STAT_RUN) — no new validation beyond what dispatch_a_job_() already performs before reaching this point.
• Output / destination: one WAL_JOB_DISPATCH WAL entry, replicated and applied through the existing commit/apply pipeline.
• Idempotency & recovery: WAL_JOB_DISPATCH's case in both apply_committed_entry() (live) and apply_wal_entry() (recovery replay) applies state as an absolute value (matching every _pad == 0 job-state fact today) — replaying it any number of times is a no-op after the first, identical to how every other absolute job-state fact already behaves.

Controller Dispatch-Failure Classification

Field	Detail
Intent	Decide retryable vs. terminal for one of dispatch_a_job_()'s six post-lookup failure points, including the bounded-retry escalation, and route to the corresponding resolution mechanism — fully inside dispatch_a_job_(), not deferred to its caller
Requirements	4.1, 4.2, 4.3

Responsibilities & Constraints

• A small classification table maps each of dispatch_a_job_()'s six post-lookup failure sites to one of two classes:
  • Retryable (transient/environmental): worker lookup failure, worker not found, worker-not-ready (E_XCE_BAD_WORKER), the xce_startjob_req() RPC call itself failing.
  • Terminal (local/invariant): invalid allocation (nc <= 0), queue-exec-info lookup failure.
  • The seventh failure point (jobid lookup failure, j never resolved) is not part of this table at all — see the dedicated bullet below.
• Placement correction, found in the same round as the site-identity fix above: this table has no consumer outside job.c — nothing on the scheduler side classifies (that assumption was superseded when classification moved to be controller-authoritative). It is declared static in job.c, not in the shared src/include/xce/errnox.h — a shared header is for cross-component contracts, and this classification is an internal implementation detail of one function's restructuring.
• xce_startjob_req()'s own RPC failure (job.c:956-960, the function's final failure return) is the sixth site, XCE_DISPATCH_FAIL_SITE_RPC_FAILURE — see Controller Dispatch Commit for the accompanying job.allocs-gating and dereference(j) fixes at that exact call site.
• Ownership correction found during /kiro-validate-design round 2: classification and resolution happen inside dispatch_a_job_(), at each of its six post-lookup failure points, using the j/job_t* pointer already held there — not in its caller handle_jobdispatch(). Two independent reasons: (1) handle_jobdispatch() only has alloc->jobid after dispatch_a_job_() returns (alloc is freed immediately after), so resolving there would require a second key_find() lookup and reopen a TOCTOU gap between the two; (2) handle_jobdispatch()'s existing loop unconditionally calls notify_dispatch() for every dispatch_a_job_() return, success (code=0) or failure — notify_dispatch() itself always emits NotifyData_data_jobDispatchFail as the event type regardless of code (confirmed by reading its body, job.c:1108-1125). Left unchanged, a terminal failure would produce both a jobDispatchFail notify (from the existing unconditional call) and a schedFailJob notify (from mark_job_fail()'s hook chain) for the same event, racing to apply PEND and EXIT to the same job depending on arrival order.
• Fix: handle_jobdispatch()'s existing per-job notify_dispatch() calls (both the success-path notify_dispatch(c, ep, alloc->jobid, 0) and the failure-path notify_dispatch(c, ep, alloc->jobid, rc1)) are removed from that loop. dispatch_a_job_() now owns notification as part of resolution: it calls notify_dispatch() itself, only on the retryable-failure path, and never on success or on a terminal failure (mark_job_fail()'s own hook chain is the sole notification for that case). handle_jobdispatch()'s loop becomes a thin dispatch-and-free loop with no notification logic of its own.
• Every one of the six post-lookup failure points calls xce_classify_dispatch_fail(site) with its own xce_dispatch_fail_site_t identity (not the raw rc/errno value that point happened to return — see Service Interface for why) — classification is never implicit in which branch executed; the table is the single source of truth, actually called at every site, not merely documentation the branch structure happens to match. This also makes the function genuinely load-bearing (not dead weight the Unit Tests section would otherwise be testing in isolation).
• Single critical section per failure point — deliberately structured to avoid an unlock/relock gap between the count check and the resolution it drives (a dispatch_fail_count read separated from its acting-upon by a lock release would be a fresh race, on top of the one this design already closes). At each of the six post-lookup failure points, inside dispatch_a_job_(), before that point's existing dereference(j)/return (or, for the four sites bug-1627 identifies, where that dereference(j) is being newly added):
  1. lock(j, RTTI_LOCK_WRITE).
  2. Call xce_classify_dispatch_fail(site) with this failure point's own xce_dispatch_fail_site_t value. If XCE_DISPATCH_FAIL_TERMINAL, skip to step 3's terminal branch immediately (no counter involved). If XCE_DISPATCH_FAIL_RETRYABLE, increment job.dispatch_fail_count and capture the resulting value into a local.
  3. Still under the same lock: for a retryable classification whose incremented count is within the configured bound (design default: 3 consecutive failures; exact value is a task-level config decision, not hardcoded here), nothing further needs the lock — this failure resolves retryable. For a retryable classification whose count exceeds the bound, or a classification that was terminal from step 2, this failure resolves terminal without releasing and reacquiring the lock — call mark_job_fail() directly here (already satisfying its RTTI_LOCK_WRITE-held precondition), with a job_failinfo_t describing the dispatch failure (jobid, code = the original error, detail = a short diagnostic string; .stage left at 0, which correctly routes apply_job_fail_transition()'s switch to its default: branch, job->job.state = JOB_STAT_EXIT — confirmed by reading that function directly; the two named .stage cases are for post-exec failures, not applicable to a job that never started).
  4. unlock(j), then dereference(j) (closing bug-1627's leak on the four failure points where it exists today, found during rounds 3-4 of this review — the other two, nc <= 0 and get_queue_exec() failure, already correctly dereference(j), unaffected by that bug — this step is uniform across all six regardless).
  5. Only now, entirely outside the lock: if step 3 resolved retryable, call notify_dispatch(c, ep, jobid, rc) — now the only caller of this function for the dispatch-failure case, so jobDispatchFail unambiguously means "retry." If step 3 resolved terminal (whichever of the two ways it got there), notify_dispatch() is not called — mark_job_fail()'s hook chain (already fired, under the lock, in step 3) is the sole notification.
• job.dispatch_fail_count is reset to 0 by Controller Dispatch Commit's own success path (above, under that component's own RTTI_LOCK_WRITE) — a fresh successful dispatch clears any prior failure history for that job.
• How the controller side satisfies Requirement 4.4/4.5's "release stale allocation/commitment data": not by cleaning up after a failed commit, but by never committing in the first place — job.allocs/job.state are only ever written inside the rc >= 0 success branch (Controller Dispatch Commit), so on any of the six classified failure paths there is nothing to release; the requirement is satisfied vacuously by construction, not by an explicit cleanup step. This is the direct benefit of the RPC-gated commit ordering decision in research.md.
• dispatch_a_job_()'s earliest failure point (jobid lookup failure — two code variants, key_find() itself failing or returning a NULL j, before j is ever resolved) is a distinct case outside this table entirely: there is no job_t* to mark failed or count against. This is not a job-state resolution problem, it is "the job does not exist" — the function's existing return rc/return -errno is sufficient, unchanged. Requirement 4's classification and this component's locking/counting/notification logic apply only to the six failure points reached after j is successfully resolved and referenced.

Contracts: Service [x]

Service Interface

/*
 * Classifies one of dispatch_a_job_()'s six post-lookup failure points as
 * retryable or terminal. Deliberately keyed on WHICH failure point was hit
 * (an explicit site identity), not on the raw errno-style `int rc` that
 * point returns -- key_find() (workermgr) and get_queue_exec() are two
 * distinct, independently-implemented failure sources that are not
 * guaranteed to return disjoint error codes (e.g. both could plausibly
 * return a generic -ENOENT for unrelated reasons), so classifying by rc
 * value alone would be ambiguous whenever two sites happen to collide.
 * Does not itself mutate job state, take any lock, or send notifications --
 * called from within dispatch_a_job_() itself, at each site, which uses the
 * result to choose between notify_dispatch() (retry) and mark_job_fail()
 * (terminal).
 */
typedef enum {
    XCE_DISPATCH_FAIL_SITE_WORKER_LOOKUP    = 0, /* workermgr key_find() rc<0 */
    XCE_DISPATCH_FAIL_SITE_WORKER_NULL      = 1, /* worker not found (w==NULL) */
    XCE_DISPATCH_FAIL_SITE_WORKER_NOT_READY = 2, /* E_XCE_BAD_WORKER */
    XCE_DISPATCH_FAIL_SITE_INVALID_ALLOC    = 3, /* nc <= 0 */
    XCE_DISPATCH_FAIL_SITE_QUEUE_EXEC       = 4, /* get_queue_exec() failure */
    XCE_DISPATCH_FAIL_SITE_RPC_FAILURE      = 5, /* xce_startjob_req() rc<0 */
} xce_dispatch_fail_site_t;

typedef enum {
    XCE_DISPATCH_FAIL_RETRYABLE = 0,
    XCE_DISPATCH_FAIL_TERMINAL  = 1,
} xce_dispatch_fail_class_t;

xce_dispatch_fail_class_t xce_classify_dispatch_fail(xce_dispatch_fail_site_t site);

• Preconditions: site is one of the six named enumerators. Corrected during task 1.1 implementation review: a default-less switch does NOT compel a compile error in plain C without -Werror (this project sets none) — it only emits a non-fatal -Wreturn-type warning and falls through to undefined behavior on an unhandled value. The actual implementation instead follows this codebase's established fail-loud convention (matching persist.c's wal_classify(), persist.c:6030-6037): a default: arm that ERROR-logs and returns a safe XCE_DISPATCH_FAIL_TERMINAL, with assert(0 && ...) gated to #ifdef XCE_TESTING only, so a future unhandled site fails loud in test builds and degrades safely (never silently misclassifies as retryable) in production.
• Postconditions: pure function, no side effects; the caller separately applies the bounded-retry override. WORKER_LOOKUP, WORKER_NULL, WORKER_NOT_READY, RPC_FAILURE classify XCE_DISPATCH_FAIL_RETRYABLE; INVALID_ALLOC, QUEUE_EXEC classify XCE_DISPATCH_FAIL_TERMINAL.
• Invariants: the same site always classifies the same way — no hidden state, no time-dependence, no dependence on the specific rc/errno value that accompanied it.

WAL Dispatch Fact (persist.c, wal_types.h)

WAL Dispatch Fact

Field	Detail
Intent	Durable record of the dispatch-time RUN transition, so a controller failover recovers the same state a live run would have reached
Requirements	2.4

Responsibilities & Constraints

• New wal_entry_type value WAL_JOB_DISPATCH = 12 (next free value in the job-fact range, after WAL_JOB_ARRAY_SUBMIT = 11).
• Reuses the existing wal_jobstate_t{jobid, state, _pad} payload verbatim — no new struct. _pad is always 0 for this type (the "legacy: state is absolute value" convention wal_jobstate_t's own header comment documents), matching the synchronous, non-deferred-apply commit style Controller Dispatch Commit uses.
• Two separate functions, both needing the new case — found by tracing precisely rather than assuming a single switch: apply_wal_entry() (persist.c:3375+) has its own fall-through group at persist.c:3452-3466 (WAL_JOB_START through WAL_JOB_RUSAGE, including WAL_JOB_FAIL — applies every type as an absolute overwrite, unconditionally). apply_committed_entry() (persist.c:3570+, the live path invoked immediately when a WAL entry commits during normal operation — the one raft-apply-decoupling actually introduced the _pad-discriminated routing for) has its own separate fall-through group at persist.c:3700-3706 (WAL_JOB_START, WAL_JOB_FINISH, WAL_JOB_STOP, WAL_JOB_RESUME, WAL_JOB_RUSAGE, WAL_JOB_CKPT, WAL_JOB_PREASON — deliberately excluding WAL_JOB_FAIL/WAL_JOB_UPDATE, which get their own _pad-aware handling immediately after). That group's own comment states the exact rationale this design already relies on: these types' payloads are "deterministic/state-independent... an absolute state baked at receipt time never goes stale" — WAL_JOB_DISPATCH fits this group by the same reasoning (Research Log: no same-job racing-fact concern for the first post-PEND transition). WAL_JOB_DISPATCH joins both fall-through lists — no new apply-time function or standalone case body is introduced in either. Corrected during task 1.2 implementation review: apply_wal_entry() is NOT actually "the recovery/replay-time path" in the current architecture — persist_recover()'s real boot-time replay (replay_wal()) and the standby catch-up path (post the 2026-07-10 ha_apply_single_entry refactor) both call apply_committed_entry() exclusively; apply_wal_entry() has zero production call sites today. Adding the case to both functions is still correct (matches every sibling WAL_JOB_* type's identical dual-registration, harmless dead-code parity, and costs nothing if apply_wal_entry() is ever revived) — only the prose description of what apply_wal_entry() currently is was wrong.
• Does not carry or restore job.allocs/job.accounts — those remain non-durable in-memory fields, unchanged from today's behavior (Boundary: Out of Boundary does not list a new durability guarantee for allocation data itself, only for the state transition).

Contracts: Batch [x] / State [x]

Batch / Job Contract

• Trigger: Controller Dispatch Commit's success path.
• Input / validation: jobid, state = JOB_STAT_RUN.
• Output / destination: standard WAL append → replication → commit pipeline, identical to every other job-state fact.
• Idempotency & recovery: absolute-value apply, safe to replay any number of times.

Data Models

Domain Model

• No new domain entities. job_t.dispatch_fail_count (via xce_job_t's wrapper, not the wire type) is a new scalar field on an existing entity, not a new aggregate. This counter is controller-side only — the scheduler has no equivalent field, since only the controller's copy is read for classification (see Controller Dispatch-Failure Classification); a scheduler-side mirror would be dead weight with no consumer.

Logical Data Model

• job_t (xce/types.h, the {xce_job_t job; uint64_t submit_lsn;} wrapper — not xce_job_t itself): + uint32_t dispatch_fail_count — in-memory only, following submit_lsn's exact precedent ("recomputed... never serialized"). A controller failover resets this to 0, which is a safe, conservative degradation (more retries allowed post-failover, never fewer, and never a misrouted terminal/retry decision).
• wal_entry_type (wal_types.h): + WAL_JOB_DISPATCH = 12, payload = existing wal_jobstate_t.

Error Handling

Error Strategy

Two independent failure surfaces, each already governed by an existing discipline this design does not change: the scheduler-side notify handler follows the same -ENOENT/-EINVAL conventions every sibling case in sched_handle_notify() already uses; the controller-side classification is keyed on an exhaustive, compiler-checked failure-site enum (no default/unrecognized branch is reachable) rather than on a possibly-ambiguous raw return code.

Error Categories and Responses

• Unrecognized dispatch-fail notify jobid (scheduler side): same as every existing case — sched_job_state_change()'s enqueue wrapper does not validate jobid existence at enqueue time; the live apply path's sched_get_job() == NULL branch (already present in core_events.c) logs a WARN and no-ops, unchanged.
• A new post-lookup failure point added to dispatch_a_job_() in the future without updating the classification (controller side): fails to compile — xce_classify_dispatch_fail()'s switch over xce_dispatch_fail_site_t has no default case, so a new enumerator with no matching case is a compiler error, not a silent runtime default.
• Bounded-retry exhaustion: not an error path — a deliberate, expected transition to terminal, using the exact same mark_job_fail() call as any other terminal classification.

Monitoring

No new logging infrastructure — mark_job_fail()'s existing hook-fired logging and sched_handle_notify()'s existing per-case TRACE/WARN conventions already cover both new code paths; only the log lines' content changes to reference the new case names.

Testing Strategy

Unit Tests

• sched_commit_decision()'s DISPATCH branch sets job->state == JOB_STAT_RUN, in addition to the existing accounts/allocs assertions.
• sched_handle_notify()'s new jobDispatchFail case enqueues an XCE_SCHED_OP_JSTATE_CHANGE targeting JOB_STAT_PEND; a follow-up drain via xce_sched_driver_translate_pending_events()/core_events.c confirms job->accounts/job->allocs are freed (reusing the existing free-on-non-RUN assertion pattern from other state-change tests).
• dispatch_a_job_(): for each of the seven failure returns (jobid lookup's two variants, plus the six post-lookup sites), confirm job.state remains PEND and job.allocs is not committed; for a successful xce_startjob_req(), confirm both are committed together with one WAL_JOB_DISPATCH fact appended. For the four retryable-class paths, additionally assert j's reference count returns to its pre-call value (regression test for bug-1627's leak).
• xce_classify_dispatch_fail(): table-driven test covering all six xce_dispatch_fail_site_t enumerators against their documented classification (four retryable, two terminal).
• Bounded-retry escalation: three consecutive retryable failures for the same job route to notify_dispatch(); the fourth (exceeding the design-default bound of 3) routes to mark_job_fail() instead.

Integration Tests

• End-to-end: scheduler commits a DISPATCH decision → controller processes it successfully → scheduler's sched_account_jobs() includes the job's resource consumption from that same round onward, across a simulated multi-round gap before the worker's actual confirmation arrives (this is bug-1624's regression test).
• End-to-end retryable failure: controller's xce_startjob_req() fails (simulated worker-unreachable) → scheduler receives jobDispatchFail → job returns to PEND → scheduler's picker reconsiders it in a later round.
• End-to-end terminal failure: controller's jobid/allocation lookup fails → mark_job_fail() fires → scheduler receives the existing schedFailJob notify (unmodified path) → job resolves to EXIT with accounts/allocs freed. Assert the scheduler does not also receive a jobDispatchFail notify for this same failure (regression test for the round-2 double-notify finding) — handle_jobdispatch()'s loop must not call notify_dispatch() for a failure dispatch_a_job_() already resolved as terminal.
• WAL durability: append a WAL_JOB_DISPATCH fact, simulate a controller restart/replay, confirm job.state == JOB_STAT_RUN is recovered without requiring the (separate, unmodified) WAL_JOB_START fact to also be present.

Performance/Load

• Not applicable — this feature adds O(1) work per dispatch decision and per notify; no new per-round scan or O(n) pass is introduced (sched_account_jobs() itself is unmodified).

Supporting References

• .kiro/specs/scheduler-dispatch-commit/research.md — full discovery findings, the three design decisions this document builds on (dead-code path identification, RPC-gated commit ordering, controller-authoritative classification), and risk mitigations.