# Requirements Document

## Project Description (Input)

**Who has the problem:** scheduler and controller maintainers, and cluster operators who rely on accurate live resource accounting and (eventually) preemption. Surfaced while investigating bug-1611/1613-adjacent behavior and while designing the sibling `scheduler-sim-eager-reconciliation` spec.

**Current situation:** `job->state`'s transition to RUN only happens later, via a separate worker-confirmation event — scheduler's `sched_job_state_change()` (`scheduler.c:963-988`) fires only on the controller's `NotifyData_data_schedStartJob` notify; controller's `job.state` flips to RUN only via `on_job_start()`/`record_job_start_fact()` (`job.c`), driven by the worker's own confirmation. Meanwhile, resource-commitment data is set much earlier and independently: scheduler's `job->accounts`/`job->allocs` are set synchronously inside `sched_commit_decision()`'s DISPATCH branch (`scheduler.c:1441-1446`), and controller's `job.allocs` is set synchronously inside `dispatch_a_job_()` (`job.c:831-963`) — both well before any worker confirmation.

This split (intentional and, on its own, correct — "scheduled/dispatched" and "confirmed running" are legitimately different signals) has a concrete correctness consequence: every scheduler round, `xce_core_begin_session()` (`core_session.c:66-108`) unconditionally zeroes `core->sharedInUse`/`worker->resInUse`, and `sched_account_jobs()` (`scheduler.c:1258-1292`) re-derives those totals only from jobs where `JOB_IS_RUN(job->state)` is true — silently skipping a job that is dispatched (accounts already set) but still PEND (not yet confirmed). Normal-mode rounds run on a 1-second cadence (`main.c:453-460`), so any dispatch-to-confirmation latency over ~1s (ordinary for a real worker RPC + fork/exec) wipes that job's resource charge at the very next round and does not restore it until confirmation lands. The allocator's admission check (`sched_check_host_satisfy_`/`sched_check_shared_satisfy_`, `resource.c:252-370`, called from `seqjob_allocator.c:130`) reads exactly this same `resInUse`/`sharedInUse`, with no separate reserved/committed pool (`xce_sched_job_t` has only `state`/`allocs`/`accounts`, `sched/types.h:96-116`) — so the resource looks free again and can be double-granted to a second job. This is a genuine live over-commit/double-booking risk in normal-mode operation, logged as **bug-1624** (unfixed).

Separately, and necessary for any fix to be safe: the controller already reports dispatch-RPC failure back to the scheduler via `NotifyData_data_jobDispatchFail` (`notify_dispatch()`, `job.c:1108-1125`), but the scheduler's `sched_handle_notify()` switch (`handler.c:499-544`) has zero case for it — it silently falls to `default: -EINVAL`. So today a dispatch failure leaves `job->accounts`/`job->allocs` (scheduler) and `j->job.allocs` (controller) stale forever with no cleanup, logged as **bug-1625** (unfixed). There is also currently no code path anywhere in `job.c` that ever sets a job's state back to `JOB_STAT_PEND` after dispatch — `mark_job_fail()`/`apply_job_fail_transition()` only ever resolves forward onto a terminal EXIT/DONE-ish state; a genuine retry-back-to-PEND path does not exist yet.

**What should change:**
1. Redefine the RUN transition to occur synchronously at DISPATCH-decision-commit time rather than at worker-confirmation time, on BOTH scheduler (inside `sched_commit_decision()`'s DISPATCH branch, alongside where `accounts`/`allocs` already get set) and controller (inside `dispatch_a_job_()`, alongside where `job.allocs` already gets set) — so neither side's internal state diverges from the other for the same job, and `sched_account_jobs()`'s existing `JOB_IS_RUN` gate naturally includes a dispatched job from the round it was dispatched, fixing bug-1624 with no change to that gate's own logic.
2. Build the dispatch-failure handling this redefinition requires to be safe: scheduler-side handling for `NotifyData_data_jobDispatchFail` (fixing bug-1625), a retryable-vs-terminal failure classification (which failure codes/conditions resolve back to PEND for retry vs. forward to EXIT/FAIL permanently — no such classification exists today), and correct reversal of state/accounts/allocs/`job.allocs` on both scheduler and controller when a dispatch is retried back to PEND.
3. Document (not implement delivery for) that a job holding committed-but-unconfirmed resources is a legitimate future preemption victim candidate under the new RUN definition — `scheduler-preemption` (sibling spec, still at requirements-generated) currently assumes in its AC6/AC9 that every RUN victim has a live, signalable process; that assumption is worth revising before that spec proceeds to design. Actually implementing preemption delivery/victim selection remains `scheduler-preemption`'s own boundary.

## Sequencing

The user has directed that this spec land **before `scheduler-allocation`** begins implementation. `scheduler-allocation`'s new iterator-based `try_alloc_on_host()` will read the same `resInUse`/`sharedInUse` totals bug-1624 corrupts, and its planned `XCE_ALLOC_PREEMPT` borrowing mode interacts with what counts as committed/borrowable — building the new allocator on top of today's broken round-boundary accounting would compound rather than fix the problem.

## Boundary Context

- **In scope**: synchronous RUN-state transition at dispatch-decision-commit time on scheduler (`sched_commit_decision()`) and controller (`dispatch_a_job_()`); scheduler-side handling of `NotifyData_data_jobDispatchFail`; retryable-vs-terminal dispatch-failure classification; PEND-reversion of state/accounts/allocs on both scheduler and controller for retryable failures; a documented note (not implementation) that a RUN-but-unconfirmed job is a valid future preemption victim candidate.
- **Out of scope**: `scheduler-controller-msg-gap` (already implementation-complete — bug-1625 is thematically identical but a new, separately-discovered gap; not a reopening of that closed spec); `scheduler-sim-decision-parity` (implementation-complete, do not reopen); `scheduler-sim-eager-reconciliation` (sibling, in-flight, explicitly simulation-mode-only and explicitly excludes normal-mode behavior changes — this spec is the normal-mode counterpart but a distinct spec, not a merge); `scheduler-preemption`'s own preemption delivery/victim-selection mechanics (cross-referenced via a pending-revision note, not merged in); `scheduler-allocation`'s iterator/plan-generation work itself (unrelated axis — HOW allocation plans are computed, not WHEN state/accounting transitions happen); any change to how a job's state is displayed to end users beyond what naturally falls out of the state-transition-timing change; PEND job candidacy/picker-selection logic changes beyond what the retry-to-PEND path requires.
- **Adjacent expectations**: this spec extends `scheduler-sim-decision-parity`/`scheduler-sim-eager-reconciliation`'s and `scheduler-controller-msg-gap`'s existing mechanisms without reopening them. `scheduler-preemption`'s requirements.md received a cross-reference note (2026-08-23) flagging its AC6/AC9 as pending revision once this spec's design is settled. Because simulation mode replays whatever the controller actually recorded (it does not independently decide state-transition timing), a recording captured *after* this feature ships will naturally carry the dispatch decision and the running-state transition in the same round — narrowing, though not eliminating, the cross-round case `scheduler-sim-eager-reconciliation` handles for older or otherwise-delayed recordings. This is a beneficial side effect to note, not a dependency either spec should design around.

## Requirements

### Requirement 1: Synchronous running-state transition on dispatch commit (Scheduler)
**Objective:** As a scheduler maintainer relying on accurate live resource accounting, I want a job's state to become running as soon as the Scheduler commits its dispatch decision, so that per-round resource-in-use totals never lose track of a job's committed resources solely because worker confirmation has not yet arrived.

#### Acceptance Criteria
1. When the Scheduler commits a dispatch decision for a job, the Scheduler shall transition that job to a running state in the same operation that records the job's resource allocation.
2. While a job's dispatch decision has been committed and no dispatch failure has since been reported for it, the Scheduler shall include that job's resource consumption in worker and shared resource-in-use totals from the round it was dispatched onward, regardless of how many rounds elapse before the worker confirms the job has started.
3. The Scheduler shall not change the existing rule that only jobs in a running state contribute to a round's resource-in-use totals; this feature changes only when a job reaches that state, not how totals are computed from it.

### Requirement 2: Synchronous running-state transition on dispatch receipt (Controller), consistent with Scheduler
**Objective:** As a maintainer relying on scheduler-side and controller-side bookkeeping staying consistent for the same job, I want the Controller to mark a job running at the same moment it processes the Scheduler's dispatch decision, so the two components never disagree about whether a job has begun consuming its allocated resources.

#### Acceptance Criteria
1. When the Controller's attempt to notify the assigned worker of a dispatch decision succeeds, the Controller shall transition that job to a running state in the same operation that commits the job's resource allocation. The transition and the allocation commit shall not depend on any later worker confirmation of actual process start.
2. The Controller shall continue to record the worker's confirmation that a job has actually started, including the job's actual start time, as a separate durable fact, independent of and in addition to the state transition performed under Criterion 1.
3. When a status query or job listing is served after a job's dispatch decision has been processed, the Controller shall report that job's state as running, whether or not the worker has yet confirmed the job started.
4. The Controller shall apply the state transition in Criterion 1 through the same durability and failover-consistency discipline it already applies to a job's other state transitions, so the transition is neither lost nor left divergent across a controller failover.

### Requirement 3: Scheduler-side dispatch-failure handling
**Objective:** As a maintainer relying on the Scheduler not silently losing track of failed dispatches, I want the Scheduler to actually process the Controller's existing dispatch-failure notification, so a dispatch that never reaches a worker does not leave stale committed-resource bookkeeping behind indefinitely.

#### Acceptance Criteria
1. When the Scheduler receives a dispatch-failure notification for a job from the Controller, the Scheduler shall process that notification rather than discard it as an unrecognized event.
2. When the Scheduler processes a dispatch-failure notification for a job, the Scheduler shall resolve that job according to the retryable-vs-terminal classification defined in Requirement 4.

### Requirement 4: Retryable-vs-terminal dispatch-failure resolution
**Objective:** As a cluster operator, I want a dispatch failure caused by a transient or environmental condition to be retried automatically, and a dispatch failure caused by a locally-invalid decision to fail immediately without endless retrying, so failed dispatches resolve correctly in both directions.

#### Acceptance Criteria
1. When a dispatch failure is caused by a transient or environmental condition — for example the assigned worker being unreachable or not in a ready state, or the dispatch request itself failing to be delivered — the Scheduler and Controller shall return the job to a pending state, making it eligible for the Scheduler to attempt dispatch again in a subsequent round.
2. When a dispatch failure is caused by a locally-invalid decision — for example the job or its allocation no longer being resolvable by the Controller — the Scheduler and Controller shall resolve the job to a terminal failed state without returning it to a pending state.
3. While a job has accumulated consecutive transient/environmental dispatch failures with no intervening successful dispatch, when the number of such failures reaches a bounded limit, the Scheduler and Controller shall resolve the job to a terminal failed state rather than returning it to pending again. The exact limit is a design-phase decision; this requirement only constrains it to be bounded, never infinite.
4. When a job is returned to a pending state following a dispatch failure, the Scheduler and Controller shall each release the resource-allocation and commitment data associated with the failed dispatch attempt, so a subsequent dispatch decision for that job is not blocked by stale data from the failed attempt.
5. When a job is resolved to a terminal failed state following a dispatch failure, the Scheduler and Controller shall each release the resource-allocation and commitment data associated with the failed dispatch attempt.

### Requirement 5: No regression to existing guarantees
**Objective:** As a maintainer, I want this feature to preserve existing dispatch, accounting, and lifecycle guarantees outside the specific gap it closes.

#### Acceptance Criteria
1. When a job's dispatch succeeds and the worker later confirms the job has started, the Controller shall continue to record the confirmed-start fact exactly as it does today, in addition to (not instead of) the earlier state transition performed under Requirement 2.
2. The Scheduler shall not apply this feature's dispatch-time state transition when operating in simulation/replay mode; simulation mode's own recorded-event replay path is unaffected by this feature.
3. Existing scheduler and controller job-lifecycle test suites shall continue to pass, except for assertions that directly depended on the state transition occurring only at worker-confirmation time, which shall be updated to reflect the new timing.
4. A full HA soak (sequential and chaotic `/xce-test-ha` regression) shall show no new job loss, state divergence, or liveness stall attributable to this feature.