sim_reconcile_unaccounted_jobs() (sim_loop.c) was calling picker->picker_iface->on_dispatch() directly from the sim driver thread, violating plugin.h's documented "allocator thread only" contract for Post-allocation hooks. Dormant in practice only because builtin_fcfs.c happens to lock internally around every hook -- a defensive measure the contract text doesn't actually require of picker plugins. Since scheduler-sim-live-allocation, sim mode spawns the same real sched_job_picker()/sched_allocator() thread pair normal mode uses, so a real allocator thread now exists to route through. Added a reconcile_req/reconcile_ack cbuf pair on xce_sched_core_t and a reserved XCE_SCHED_RECONCILE_WAKE sentinel pushed through jbuf to wake the allocator thread without touching jbuf's pinned capacity-1/item-type invariant (Req 5 AC 7). sched_allocator() services the request and calls on_dispatch() itself; sim_reconcile_invoke_on_dispatch() blocks on the ack. Falls back to a direct call when no allocator thread is live for a given core (standalone unit tests), which is safe by construction. TDD: new test spawns a stand-in allocator-relay thread and asserts on_dispatch() fires there, not on the caller's thread -- confirmed RED against the old direct-call code, GREEN after. Four other test files with hand-rolled sched_allocator() stubs needed the same relay branch (found by an actual hang at meson's timeout). Full scheduler suite: 85/85, zero regressions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
225 lines
30 KiB
Markdown
225 lines
30 KiB
Markdown
# Design Document
|
|
|
|
## Overview
|
|
|
|
**Purpose**: This feature closes the residual gap in `scheduler-sim-decision-parity`'s Recorded-Decision Reconciliation mechanism (`sim_reconcile_unaccounted_jobs()`, `sim_loop.c`) so a simulation-mode replay correctly captures a job's resource allocation regardless of how many rounds separate its recorded DISPATCH decision from its RUN-state confirmation — the realistic case in any real deployment, and the dominant real-world manifestation of bug-1611/bug-1613.
|
|
|
|
**Users**: Scheduler operators debugging or tuning via `xce_scheduler -i -s <replay>` (interactive) or batch `--sim` replay.
|
|
|
|
**Impact**: Changes when `sim_reconcile_unaccounted_jobs()` is willing to search a round's recorded decision stream for a job — from "only if the job is already RUN this round" to "whenever the job has no captured allocation yet, regardless of its current state." No other function changes.
|
|
|
|
### Goals
|
|
- Capture a job's recorded DISPATCH allocation at the round it actually occurred in the original recording, independent of the job's state at that moment.
|
|
- Transition a captured job to a running state synchronously, at the SAME round its dispatch decision is captured -- mirroring normal-mode's own `sched_commit_decision()` semantics (bug-1624) -- so resource-in-use accounting reflects it starting that same round via `sched_account_jobs()`'s existing unchanged `JOB_IS_RUN` gate. (Revised 2026-09-11 -- see below.)
|
|
- Make the dispatch decision observable (printed, recorded, REPL-checked) at the historically accurate round.
|
|
|
|
### Non-Goals
|
|
- Any change to how/when a job transitions to a NON-running state (DONE/EXIT/preemption). (The RUN transition itself is IN scope as of the 2026-09-11 revision below -- see Correction.)
|
|
- Any change to simulation mode's live picker (still PEND-only for freshly-arriving jobs, per `scheduler-allocation`).
|
|
- Any change to `--sim-verify`'s comparison semantics beyond not introducing a new false mismatch.
|
|
- Reopening or modifying `scheduler-sim-decision-parity`'s own requirements, design, or tasks, or any normal-mode (`driver.c`) behavior.
|
|
|
|
**Correction (2026-09-11, user-directed, live replay session):** the original version of this Overview described capture and RUN-state transition as fully decoupled, with `sched_account_jobs()`'s existing `JOB_IS_RUN` gate providing "correct, state-gated" deferral until a job's SEPARATELY-recorded `JOB_STATE_CHANGE` confirmation event replayed -- potentially many rounds after capture. Live use surfaced the resulting operator confusion directly (`info jobs` showing PEND for a job `[sim] DISPATCH` had already printed), and investigation confirmed normal-mode's own `sched_commit_decision()` (`scheduler.c`, bug-1624) does NOT defer this way -- it sets `job->state = JOB_STAT_RUN` synchronously in the same operation that captures the allocation. This design is revised so simulation-mode reconciliation now does the same: `sim_reconcile_unaccounted_jobs()`'s capture step ALSO sets `job->state = JOB_STAT_RUN`, so `sched_account_jobs()`'s unchanged `JOB_IS_RUN` gate now includes the job starting the SAME round as capture, not a later one. The later, separately-recorded `JOB_STATE_CHANGE`/`JOB_UPDATE`-to-RUN event, when it replays, becomes a safe no-op re-affirmation (RUN -> RUN) rather than the sole state-transition trigger. See Requirement 1's own 2026-09-11 revision (requirements.md) and `research.md`'s corresponding Research Log entry for the full rationale.
|
|
|
|
**Correction (2026-09-07, bug-1759/bug-1779):** the "live picker" Non-Goal above was never meant to license `sim_reconcile_unaccounted_jobs()` leaving a reconciled job as a live picker candidate — it means "do not give the live picker real resource-matching allocation logic" (still out of scope, still `scheduler-allocation`'s territory). Whether a reconciled job stays in the picker's own `pendJobs` bookkeeping is a `scheduler-sim-decision-parity` concern (see that spec's design.md, "Integration with `--sim-verify`", which already documented "the picker never re-decides an already-resolved job" as the correct assumption). The implementation had drifted from that documented assumption: `sim_reconcile_unaccounted_jobs()` never called `picker_iface->on_dispatch()`, so a reconciled job stayed a live candidate and was live-observed being re-offered and re-PENDed every subsequent round (bug-1759: 22/50 already-dispatched jobs in one batch; bug-1779: 5/10 array elements). Fixed by adding that missing `on_dispatch()` call, mirroring `sched_allocator()`'s own live-dispatch ordering (`scheduler.c`) exactly — a correction to already-agreed-upon behavior, not a new capability, so it did not need to reopen this spec's own requirements/tasks. `test_sim_verify_no_false_mismatch_across_round_separated_dispatch` (`test_sim_verify.c`) was updated to match: rounds 2-3 of its fixture are now empty `ReplaySession`s (no more picker-echoed PEND for an already-reconciled job) rather than baking in the pre-fix stale-PEND behavior as "expected."
|
|
|
|
**Correction (2026-09-21, bug-1838, superseding the 2026-09-11 revision above):** the 2026-09-11 widening (capture "regardless of the job's current state," including PEND) turned out to be the root cause of a live-observed over-dispatch: a 22-slot host took 44 concurrent single-slot jobs during replay (bug-1836), because a PEND job's recorded DISPATCH decision was force-applied by reconciliation with no capacity check against current resource state. bug-1836 first closed the immediate hole by adding a capacity gate (`sim_reconciled_alloc_fits_capacity()`) to reconciliation itself — but investigation into why a PEND job could ever reach this force-apply path at all found that the underlying justification for the 2026-09-11 widening no longer holds: the since-completed `scheduler-sim-live-allocation` spec gave simulation mode a genuine, capacity-checked live picker/allocator thread pair (`sched_job_picker()`/`sched_allocator()`, `sched_election.c`) that both drivers now spawn identically — "one core, two consumers" (`.kiro/steering/scheduler-driver-equivalence.md`) is now actually true for dispatch, not just aspirational. A freshly-PEND job now correctly reaches that SAME live capacity-checked path simulation mode itself runs, exactly like normal mode; reconciliation force-applying a stale recorded decision for it is redundant at best and a second, competing source of truth at worst. The trigger is therefore narrowed back to RUN-only (`JOB_IS_RUN(job->state)`, mutually excluding DONE/EXIT for free under this codebase's 4-state job model): reconciliation's remaining, genuinely necessary scope is a job already confirmed RUN (via a separately-recorded `JOB_STATE_CHANGE`/`JOB_UPDATE` event) whose allocation data was never captured — structurally invisible to the live picker, since `next_job()` never offers a RUN job. This does NOT reopen the 2026-09-11 revision's OTHER change (synchronous RUN-transition-at-capture for the jobs reconciliation still does handle) — that remains correct and unchanged; only the ELIGIBILITY condition for entering capture at all is revised. See `sim_reconcile_unaccounted_jobs()`'s own updated header comment (`sim_loop.c`) for the complete before/after rationale, and Requirement 1's own 2026-09-21 correction (requirements.md) for the corresponding Acceptance Criteria update.
|
|
|
|
**Correction (2026-09-21, bug-1837, user-directed):** the `on_dispatch()` call bug-1759/bug-1779's own correction above added (2026-09-07) called `picker_iface->on_dispatch()` directly from `sim_reconcile_unaccounted_jobs()`'s own caller thread (`sched_sim_driver_run()`'s driver thread) — but `plugin.h` documents `on_dispatch()`/`on_pend()` as "Post-allocation hooks (MUST) -- allocator thread, after each allocation attempt," a threading-contract violation an `/xce-audit-invariants scheduler` run flagged as dormant/inert (real only because `builtin_fcfs.c`'s own internal locking happened to make every call site safe, a defensive measure plugin.h's contract text does not actually require of picker plugins in general). Now that `scheduler-sim-live-allocation` gives simulation mode the SAME real `sched_job_picker()`/`sched_allocator()` thread pair normal mode spawns (see the 2026-09-21 bug-1838 correction above), a real allocator thread genuinely exists to route this call through instead of calling it directly: `sim_reconcile_invoke_on_dispatch()` (`sim_loop.c`) relays the request to `sched_allocator()` (`sched_election.c`) via two dedicated cbufs on `xce_sched_core_t` (`core.h`) -- `reconcile_req`/`reconcile_ack` -- waking it through a reserved jbuf sentinel (`XCE_SCHED_RECONCILE_WAKE`) rather than widening jbuf's own pinned item type/capacity (`test_core_jbuf_capacity_pin.c`, Req 5 AC 7). A standalone caller with no live allocator thread for its core (e.g. `test_sim_reconcile_unaccounted.c`'s own direct-call unit tests) still gets a direct call, safe by construction since only one thread exists to touch picker state in that case. This is a threading-mechanism correction only -- reconciliation's own trigger condition, capacity gate, and RUN-only eligibility (both corrections above) are unchanged.
|
|
|
|
## Boundary Commitments
|
|
|
|
### This Spec Owns
|
|
- The trigger condition of `sim_reconcile_unaccounted_jobs()` (`sim_loop.c`) — which jobs are eligible to be searched for a matching recorded DISPATCH decision each round.
|
|
- The test coverage proving capture now works across round separation, still respects the existing no-fabrication/no-double-capture guarantees, and does not regress `--sim-verify` or interactive/batch parity.
|
|
|
|
### Out of Boundary
|
|
- `sched_account_jobs()`'s own `JOB_IS_RUN` charging gate (`scheduler.c:1258`) — unchanged; it is exactly the mechanism this feature relies on to turn a captured job's RUN state (now set at capture time, per the 2026-09-11 revision above) into an actual resource charge.
|
|
- The match/capture mechanics once a job is eligible (jobid matching, ownership transfer of `xce_job_alloc_t`, `make_job_account()`, shift-down compaction, decision reporting via `make_dispatch_decision()`/`sim_print_decision()`/`xce_repl_check_decision()`) — all unchanged, all already correct per `scheduler-sim-decision-parity`.
|
|
- Normal-mode (`driver.c`) behavior, simulation mode's live picker, `--sim-verify`'s comparison algorithm itself, and any NON-running-state job transition logic (DONE/EXIT/preemption) — the RUN transition triggered by capture is now IN boundary as of the 2026-09-11 revision (see Overview Correction).
|
|
|
|
### Allowed Dependencies
|
|
- `sim->core.jobs` iteration and `decoded_session->decisions` (both already owned/populated by `sim_reconcile_unaccounted_jobs()`'s existing implementation) — P0.
|
|
- `sched_account_jobs()`'s existing `JOB_IS_RUN` gate as the sole mechanism deferring actual resource charging — P0, must remain unmodified for this design to be correct.
|
|
- `make_job_account()`/`make_dispatch_decision()` (existing, unmodified) — P0.
|
|
|
|
### Revalidation Triggers
|
|
- Any future change to `sched_account_jobs()`'s charging gate (e.g. adding a new precondition beyond `JOB_IS_RUN`) must re-verify this feature's Requirement 1.2/1.3 still hold, since this design relies on that gate — not a new one of its own — for correct deferral.
|
|
- Any future change that makes simulation mode's live picker emit a DISPATCH decision (currently PEND-only, `scheduler-allocation`'s stated scope) must re-verify Requirement 3.3's dedup assumption, which currently holds structurally (no live DISPATCH ever competes with a reconciled capture) rather than via an explicit check in this feature's own code.
|
|
- Any change to `decoded_session`'s per-round decode/free lifecycle (`sim_loop.c:871`/`:1006`) must re-verify this feature's core premise (every round's own decisions are only ever visible during that round).
|
|
|
|
## Architecture
|
|
|
|
### Existing Architecture Analysis
|
|
`sim_reconcile_unaccounted_jobs()` runs unconditionally once per simulation round, strictly between `xce_core_begin_session()` (which zeroes `resInUse`/`sharedInUse`) and `sched_account_jobs()` (which re-sums them from every `JOB_IS_RUN` job with `job->accounts != NULL`) — see `sim_loop.c:912-957`. Its current trigger gate additionally requires `JOB_IS_RUN(job->state)` before considering a job at all:
|
|
|
|
```c
|
|
if ((job == NULL) || !JOB_IS_RUN(job->state) || (job->accounts != NULL)) {
|
|
continue;
|
|
}
|
|
```
|
|
|
|
`decoded_session` (this round's decoded recorded-decision stream) is freed at the end of the same round (`sim_loop.c:1006`) and never persisted — so a job skipped at round N because it was still PEND has no later opportunity to be matched against round N's data, even after it reaches RUN at round N+k.
|
|
|
|
### Architecture Pattern & Boundary Map
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
RoundStart[Round begins] --> Decode[decode_ReplaySession this round]
|
|
Decode --> Begin[xce_core_begin_session resets resInUse]
|
|
Begin --> Reconcile[sim_reconcile_unaccounted_jobs]
|
|
Reconcile --> Gate{job accounts is NULL}
|
|
Gate -- no --> NextJob[skip already-captured job]
|
|
Gate -- yes --> Search[search this rounds decoded decisions for jobid]
|
|
Search --> Found{matching DISPATCH found}
|
|
Found -- yes --> Capture["capture allocs and accounts, set state=RUN synchronously (2026-09-11), report decision"]
|
|
Found -- no --> Retry[leave accounts NULL and state unchanged, retry next round]
|
|
Capture --> Charge[sched_account_jobs]
|
|
Retry --> Charge
|
|
NextJob --> Charge
|
|
Charge --> ChargeGate{job state is RUN and accounts set}
|
|
ChargeGate -- yes --> ResInUse["include in resInUse -- now true the SAME round as capture, not a later one"]
|
|
ChargeGate -- no --> Defer[exclude from resInUse]
|
|
```
|
|
|
|
**Architecture Integration**:
|
|
- Selected pattern: single-condition removal from an existing, already-correct-in-every-other-respect mechanism — not a new component.
|
|
- Domain/feature boundaries: capture (this feature, `Gate`/`Search`/`Capture`/`Retry` above) stays fully decoupled from charging (`sched_account_jobs`, `ChargeGate` above, untouched) — the same separation normal mode already has between `sched_commit_decision()` (capture) and `sched_account_jobs()` (charge).
|
|
- Existing patterns preserved: per-round re-invocation, jobid-scoped matching, ownership-transfer capture, shift-down compaction, three-call decision reporting — all untouched.
|
|
- New components rationale: none — this is a boundary-widening change to one existing condition.
|
|
- Steering compliance: directly closes the gap `.kiro/steering/scheduler-driver-equivalence.md` documents as bug-1611's residual; testing discipline (contention fixture) from that same steering doc drives the Testing Strategy below.
|
|
|
|
### Technology Stack
|
|
|
|
| Layer | Choice / Version | Role in Feature | Notes |
|
|
|-------|------------------|-----------------|-------|
|
|
| Scheduler core (C, gnu17) | Existing `sim_loop.c`/`scheduler.c` | Trigger-condition change only | No new dependency; no schema/wire-format change |
|
|
|
|
## File Structure Plan
|
|
|
|
### Modified Files
|
|
- `src/scheduler/src/sim_loop.c` — remove `!JOB_IS_RUN(job->state)` from `sim_reconcile_unaccounted_jobs()`'s trigger condition (`sim_loop.c:592`); update the function's header doc comment (`sim_loop.c:~505-565`) to describe the new state-independent trigger and reframe the "double-application guard" description to also cover "this is now the sole capture-eligibility gate."
|
|
- `src/scheduler/testsuite/test_sim_reconcile_unaccounted.c` — replace `test_reconcile_skips_non_run_state_job` (asserts the bug: a PEND job with a matching recorded entry must stay uncaptured) with an inverted-expectation test proving a PEND job **is** now captured; add new tests for cross-round separation, `job->state` transitioning after capture, the no-recorded-origin-ever case, and the "captured-but-not-RUN contributes zero to resInUse" case (Requirement 1.2) driven through `sched_account_jobs()`.
|
|
- `src/scheduler/testsuite/test_sim_contention_parity.c` and/or a new contention-focused test file — add a resource-contention fixture (capacity insufficient for all candidate jobs) exercising cross-round dispatch-to-confirmation separation, per `scheduler-driver-equivalence.md`'s explicit testing discipline that a no-contention fixture cannot prove accounting correctness.
|
|
- A `--sim-verify` regression test (existing `test_sim_*` verify-mode file, or a new one if none fits) spanning a multi-round-separated dispatch — proves Requirement 4.2.
|
|
|
|
### No New Files
|
|
This feature introduces no new production source file — see Simplification decision in `research.md`.
|
|
|
|
## System Flows
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant Replay as Replay stream
|
|
participant Recon as sim_reconcile_unaccounted_jobs
|
|
participant Job as job in core jobs
|
|
participant Acct as sched_account_jobs
|
|
|
|
Replay->>Recon: round N decoded decisions include DISPATCH for job X
|
|
Note over Job: job X state is PEND at round N
|
|
Recon->>Job: capture allocs and accounts from DISPATCH; set state=RUN synchronously (2026-09-11)
|
|
Recon->>Recon: report decision at round N
|
|
Acct->>Job: check state is RUN and accounts set
|
|
Acct->>Acct: charge resInUse starting round N (same round as capture)
|
|
Replay->>Job: round N+k replays job X's separately-recorded state-change-to-RUN event
|
|
Note over Job: state already RUN -- idempotent re-affirmation, not a new transition
|
|
Recon->>Job: accounts already set, skip re-search
|
|
```
|
|
|
|
**Key decisions not obvious from the diagram (revised 2026-09-11)**: the decision is reported at round N (the historically accurate round, Requirement 2.1). As of the 2026-09-11 revision, charging ALSO begins at round N, not round N+k — `job->state` is set to RUN synchronously with capture, mirroring normal-mode's own `sched_commit_decision()` exactly (bug-1624), so reporting and charging are no longer decoupled by round. The separately-recorded `JOB_STATE_CHANGE`/`JOB_UPDATE`-to-RUN event at round N+k, when it later replays, is a safe no-op re-affirmation of an already-RUN job (`core_events.c`'s handler simply reassigns `job->state = new_state`, RUN → RUN, with no guard triggered) — it is preserved in the replay stream and still processed, but no longer the operative trigger for the job's own running-state transition or its resource charge.
|
|
|
|
## Requirements Traceability
|
|
|
|
| Requirement | Summary | Components | Interfaces | Flows |
|
|
|-------------|---------|------------|------------|-------|
|
|
| 1.1 | Capture at originating round regardless of state | `sim_reconcile_unaccounted_jobs` | trigger gate | Search/Capture |
|
|
| 1.2 (revised 2026-09-11) | Capture ALSO transitions job to RUN synchronously, same round | `sim_reconcile_unaccounted_jobs` | Capture step (new `job->state = JOB_STAT_RUN` write) | Capture |
|
|
| 1.3 (revised 2026-09-11) | resInUse includes the job starting the SAME round as capture | `sched_account_jobs` (unchanged, relied upon) | `JOB_IS_RUN` charge gate | ChargeGate/ResInUse |
|
|
| 1.4 (new 2026-09-11) | Later separately-recorded confirmation event is an idempotent re-affirmation | `core_events.c`'s `XCE_CORE_EVENT_JOB_STATE_CHANGE` handler (unchanged, verified safe) | n/a | n/a |
|
|
| 1.5 | Matches normal-mode's equivalent-point reporting | `sched_account_jobs` (shared with normal mode) | n/a | n/a |
|
|
| 2.1 | Decision visible at historically accurate round | `sim_reconcile_unaccounted_jobs` (decision reporting, unchanged) | `make_dispatch_decision`/`sim_print_decision`/`xce_repl_check_decision` | Capture step |
|
|
| 2.2 | Original decision type/detail preserved | `make_dispatch_decision` (unchanged) | n/a | Capture step |
|
|
| 2.3 | No duplicate report when live decision also exists | live picker (unchanged, PEND-only — structural invariant) | n/a | n/a |
|
|
| 3.1 | Never capture from a non-originating decision | jobid match (unchanged) | `entry->body.dispatch->alloc->jobid` check | Search |
|
|
| 3.2 | Never re-capture once captured | `job->accounts != NULL` guard (unchanged, now the sole gate) | trigger gate | Gate |
|
|
| 3.3 | Already-captured job not overwritten by a later live decision | structural (live picker never emits DISPATCH) | n/a | n/a |
|
|
| 4.1 | Same-round case still works | trigger gate (RUN is a subset of "any state") | trigger gate | Gate/Search |
|
|
| 4.2 | No new `--sim-verify` mismatch | compaction ordering (unchanged), verified by new test | decoded_session compaction | Capture step |
|
|
| 4.3 | Interactive/batch parity preserved | shared `sched_sim_driver_run()` call site (unchanged) | n/a | n/a |
|
|
| 4.4 | No recorded origin anywhere: report unavailable, keep going | "not found" branch (unchanged) | TRACE log, `job->accounts` stays NULL | Retry |
|
|
| 4.5 | Normal mode unaffected | `driver.c` untouched | n/a | n/a |
|
|
|
|
## Components and Interfaces
|
|
|
|
| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|
|
|-----------|--------------|--------|---------------|---------------------------|-----------|
|
|
| `sim_reconcile_unaccounted_jobs` | Scheduler sim driver | Widen capture eligibility to any not-yet-captured job, regardless of state | 1.1, 2.1, 3.1, 3.2, 4.1, 4.4 | `sched_account_jobs` (P0, relied on unmodified) | Service |
|
|
|
|
### Scheduler sim driver
|
|
|
|
#### `sim_reconcile_unaccounted_jobs`
|
|
|
|
| Field | Detail |
|
|
|-------|--------|
|
|
| Intent | Capture a job's recorded DISPATCH allocation as soon as the replay reaches the round it occurred in, independent of the job's current state |
|
|
| Requirements | 1.1, 2.1, 3.1, 3.2, 4.1, 4.4 |
|
|
|
|
**Responsibilities & Constraints**
|
|
- Owns only the capture-eligibility decision (which jobs are searched each round) and the search/capture/report mechanics already delivered by `scheduler-sim-decision-parity` (unchanged).
|
|
- Has no knowledge of resource charging — that remains `sched_account_jobs()`'s exclusive responsibility.
|
|
- Never fabricates data: a job with no matching recorded decision in a given round is left uncaptured and re-checked the following round (Requirement 4.4).
|
|
|
|
**Dependencies**
|
|
- Inbound: `sched_sim_driver_run()`'s round loop — calls this once per round, unchanged call site and unchanged position relative to `xce_core_begin_session()`/`sched_account_jobs()`/`xce_core_signal_go()` (P0).
|
|
- Outbound: `sched_account_jobs()` — this feature supplies `job->accounts` earlier than before; `sched_account_jobs()`'s own unmodified `JOB_IS_RUN` gate determines when that data starts being charged (P0).
|
|
|
|
**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]
|
|
|
|
##### Service Interface
|
|
```c
|
|
/* Unchanged signature. Behavior change #1 (original): a job is now
|
|
* eligible for capture whenever job->accounts == NULL, regardless of
|
|
* job->state (previously additionally required JOB_IS_RUN(job->state)).
|
|
* Behavior change #2 (2026-09-11 revision): a successful capture ALSO
|
|
* sets job->state = JOB_STAT_RUN synchronously, in the same operation --
|
|
* mirroring scheduler.c's sched_commit_decision() (bug-1624) exactly --
|
|
* so sched_account_jobs()'s own unchanged JOB_IS_RUN gate includes the
|
|
* job starting THIS round, not a later one. This function still performs
|
|
* no charging itself; it now also performs no NON-running state
|
|
* transition (DONE/EXIT/preemption remain exclusively triggered by their
|
|
* own separately-recorded events, unchanged). */
|
|
static void sim_reconcile_unaccounted_jobs(xce_sched_sim_driver_t* sim,
|
|
replay_session_t* decoded_session,
|
|
sched_record_session_ctx_t* record_ctx);
|
|
```
|
|
- Preconditions: unchanged — `sim`/`decoded_session` NULL is a safe no-op; `record_ctx` may legitimately be NULL.
|
|
- Postconditions: unchanged — a matched job has `job->allocs`/`job->accounts` set and its entry removed from `decoded_session->decisions`; an unmatched job is left exactly as it was.
|
|
- Invariants (new): a job is captured **at most once**, at the first round (in replay order) whose decoded stream contains its originating DISPATCH decision — independent of that round's job state.
|
|
|
|
**Implementation Notes**
|
|
- Integration: delete `!JOB_IS_RUN(job->state)` from the trigger condition at `sim_loop.c:592`; no other line in the match/capture/report path changes.
|
|
- Validation: see Testing Strategy — contention fixture, cross-round separation, `--sim-verify`, interactive/batch parity, and the inverted `test_reconcile_skips_non_run_state_job` replacement are all required, not optional.
|
|
- Risks: repeated per-round "not found" TRACE logging for a job with no recorded origin anywhere in the replay (Requirement 4.4 case) — accepted, TRACE-level, see `research.md`'s Risks section.
|
|
- **Verified safe (early capture vs. cleanup)**: capturing `job->accounts` while a job is still PEND is safe against every existing job-cleanup path. `sched_uncharge_account()` has zero production callers anywhere in the scheduler — production resource accounting is a reset-and-resum model (`xce_core_begin_session()` zeroes `resInUse`/`sharedInUse` every round; `sched_account_jobs()` re-sums it from scratch from currently-`JOB_IS_RUN` jobs only). Both places `job->accounts` is freed on a non-RUN transition (`scheduler.c:974-977`, `core_events.c:915-918`) release memory only, with no `resInUse` mutation. A job captured while PEND that later transitions directly to a terminal state without ever passing through RUN is therefore never summed by `sched_account_jobs()` at any round and its accounts are freed with zero accounting impact — early capture cannot under-count `resInUse`. See `research.md`'s corresponding Research Log entry for the full trace.
|
|
|
|
## Error Handling
|
|
|
|
### Error Strategy
|
|
- **No matching recorded decision this round**: not an error — `job->accounts`/`job->allocs` stay `NULL`, a TRACE line is logged, the job is re-checked next round (unchanged behavior, now reached by more jobs per round since the gate widened).
|
|
- **No recorded origin anywhere in the replay** (Requirement 4.4): the same "not found" path applies every round for the rest of the replay; the scheduler continues processing without terminating, and that job's resource usage is reported as unavailable (never fabricated) wherever it is inspected (e.g. `info worker`).
|
|
- **`decoded_session` decode failure**: unchanged — already handled upstream of this function (`sim_loop.c:874-879`) by skipping verification for that round; this feature does not alter that path.
|
|
|
|
## Testing Strategy
|
|
|
|
### Unit Tests (`test_sim_reconcile_unaccounted.c`)
|
|
- Replace `test_reconcile_skips_non_run_state_job` with `test_reconcile_captures_non_run_state_job_when_dispatch_recorded` — a `JOB_STAT_PEND` job with a matching recorded DISPATCH entry now gets `job->allocs`/`job->accounts` captured and the entry consumed (Requirement 1.1, 3.1).
|
|
- New: `test_reconcile_still_skips_already_captured_job_regardless_of_state` — a job with `job->accounts` already set (any state) is never re-searched, proving the state-widened gate did not weaken the existing no-double-capture guard (Requirement 3.2).
|
|
- New: `test_reconcile_no_match_leaves_pend_job_uncaptured_and_retries` — a PEND job with no matching entry this round stays uncaptured and is eligible again next round (Requirement 4.4, state-independent retry).
|
|
- New: `test_reconcile_multiple_jobs_captured_independently_across_different_rounds` — several PEND jobs simultaneously outstanding, each with its own recorded DISPATCH decision spread across different rounds (some matching this round, some not yet recorded, some already captured from an earlier round); assert each job's `job->accounts` reflects only its own jobid's allocation data, captured at its own correct round, with zero cross-job interference from the wider per-round scan (Requirement 1.1, 3.1 under concurrent load — the single-job tests above cannot distinguish "correct" from "correct by coincidence, only one candidate available").
|
|
- Existing `test_reconcile_reconstructs_allocs_and_accounts_from_matching_dispatch`, `test_reconcile_ignores_non_dispatch_entry_with_matching_jobid`, `test_reconcile_compacts_via_shift_down_preserving_order`, `test_reconcile_is_safe_to_recheck_on_a_later_round`, `test_reconcile_then_release_via_job_state_change_is_asan_clean`, `test_reconcile_null_args_are_a_safe_noop` — re-run unmodified as regression coverage (Requirement 4.1); no expected behavior change for an already-RUN job.
|
|
|
|
### Integration Tests (round-boundary + charging interaction)
|
|
- **Revised 2026-09-11**: `test_captured_pend_job_excluded_from_resinuse_until_run` — RENAMED/INVERTED to `test_captured_job_included_in_resinuse_same_round_as_capture`. Drive `sim_reconcile_unaccounted_jobs()` for a PEND job with a matching recorded DISPATCH entry, then assert `job->state == JOB_STAT_RUN` immediately (Requirement 1.2) and, after running `sched_account_jobs()` in the SAME round, that `w->resInUse`/`core->sharedInUse` already reflect it (Requirement 1.3) — NOT deferred to a later round. Add a companion assertion that a later, separately-recorded `JOB_STATE_CHANGE`-to-RUN event for the same job (applied via the normal `XCE_CORE_EVENT_JOB_STATE_CHANGE` path) is a safe no-op that does not double-charge or corrupt `resInUse` (Requirement 1.4).
|
|
- New (resource-contention fixture, per `scheduler-driver-equivalence.md`'s explicit discipline): a worker with capacity for N jobs, more than N candidate jobs recorded, dispatch decisions and RUN confirmations separated across multiple rounds — assert the live allocator's own capacity check (`sched_check_host_satisfy_`) sees correct `resInUse` starting at the CAPTURE round (revised from "the confirmation round"), not permanently-zero capacity. A no-contention fixture is explicitly insufficient here (`scheduler-driver-equivalence.md`).
|
|
- New: `test_sim_verify_no_false_mismatch_across_round_separated_dispatch` — a `--sim-verify` run over a replay where dispatch and RUN confirmation are several rounds apart reports zero mismatches (Requirement 4.2).
|
|
- Existing `test_repl_equivalence.c`-style interactive/batch parity coverage re-run unmodified — this feature's call site is shared between both modes, so no new test is needed to prove Requirement 4.3, only confirmation the existing suite still passes.
|