diff --git a/.kiro/specs/scheduler-sim-eager-reconciliation/design.md b/.kiro/specs/scheduler-sim-eager-reconciliation/design.md index b0841f48..27d65b63 100644 --- a/.kiro/specs/scheduler-sim-eager-reconciliation/design.md +++ b/.kiro/specs/scheduler-sim-eager-reconciliation/design.md @@ -10,15 +10,17 @@ ### Goals - Capture a job's recorded DISPATCH allocation at the round it actually occurred in the original recording, independent of the job's state at that moment. -- Preserve correct, state-gated resource-in-use accounting: a captured-but-not-yet-running job still contributes zero to `resInUse` until it genuinely transitions to RUN. +- Transition a captured job to a running state synchronously, at the SAME round its dispatch decision is captured -- mirroring normal-mode's own `sched_commit_decision()` semantics (bug-1624) -- so resource-in-use accounting reflects it starting that same round via `sched_account_jobs()`'s existing unchanged `JOB_IS_RUN` gate. (Revised 2026-09-11 -- see below.) - Make the dispatch decision observable (printed, recorded, REPL-checked) at the historically accurate round. ### Non-Goals -- Any change to `job->state` transition timing or triggers. +- Any change to how/when a job transitions to a NON-running state (DONE/EXIT/preemption). (The RUN transition itself is IN scope as of the 2026-09-11 revision below -- see Correction.) - Any change to simulation mode's live picker (still PEND-only for freshly-arriving jobs, per `scheduler-allocation`). - Any change to `--sim-verify`'s comparison semantics beyond not introducing a new false mismatch. - Reopening or modifying `scheduler-sim-decision-parity`'s own requirements, design, or tasks, or any normal-mode (`driver.c`) behavior. +**Correction (2026-09-11, user-directed, live replay session):** the original version of this Overview described capture and RUN-state transition as fully decoupled, with `sched_account_jobs()`'s existing `JOB_IS_RUN` gate providing "correct, state-gated" deferral until a job's SEPARATELY-recorded `JOB_STATE_CHANGE` confirmation event replayed -- potentially many rounds after capture. Live use surfaced the resulting operator confusion directly (`info jobs` showing PEND for a job `[sim] DISPATCH` had already printed), and investigation confirmed normal-mode's own `sched_commit_decision()` (`scheduler.c`, bug-1624) does NOT defer this way -- it sets `job->state = JOB_STAT_RUN` synchronously in the same operation that captures the allocation. This design is revised so simulation-mode reconciliation now does the same: `sim_reconcile_unaccounted_jobs()`'s capture step ALSO sets `job->state = JOB_STAT_RUN`, so `sched_account_jobs()`'s unchanged `JOB_IS_RUN` gate now includes the job starting the SAME round as capture, not a later one. The later, separately-recorded `JOB_STATE_CHANGE`/`JOB_UPDATE`-to-RUN event, when it replays, becomes a safe no-op re-affirmation (RUN -> RUN) rather than the sole state-transition trigger. See Requirement 1's own 2026-09-11 revision (requirements.md) and `research.md`'s corresponding Research Log entry for the full rationale. + **Correction (2026-09-07, bug-1759/bug-1779):** the "live picker" Non-Goal above was never meant to license `sim_reconcile_unaccounted_jobs()` leaving a reconciled job as a live picker candidate — it means "do not give the live picker real resource-matching allocation logic" (still out of scope, still `scheduler-allocation`'s territory). Whether a reconciled job stays in the picker's own `pendJobs` bookkeeping is a `scheduler-sim-decision-parity` concern (see that spec's design.md, "Integration with `--sim-verify`", which already documented "the picker never re-decides an already-resolved job" as the correct assumption). The implementation had drifted from that documented assumption: `sim_reconcile_unaccounted_jobs()` never called `picker_iface->on_dispatch()`, so a reconciled job stayed a live candidate and was live-observed being re-offered and re-PENDed every subsequent round (bug-1759: 22/50 already-dispatched jobs in one batch; bug-1779: 5/10 array elements). Fixed by adding that missing `on_dispatch()` call, mirroring `sched_allocator()`'s own live-dispatch ordering (`scheduler.c`) exactly — a correction to already-agreed-upon behavior, not a new capability, so it did not need to reopen this spec's own requirements/tasks. `test_sim_verify_no_false_mismatch_across_round_separated_dispatch` (`test_sim_verify.c`) was updated to match: rounds 2-3 of its fixture are now empty `ReplaySession`s (no more picker-echoed PEND for an already-reconciled job) rather than baking in the pre-fix stale-PEND behavior as "expected." ## Boundary Commitments @@ -28,9 +30,9 @@ - The test coverage proving capture now works across round separation, still respects the existing no-fabrication/no-double-capture guarantees, and does not regress `--sim-verify` or interactive/batch parity. ### Out of Boundary -- `sched_account_jobs()`'s own `JOB_IS_RUN` charging gate (`scheduler.c:1258`) — unchanged; it is exactly the correct deferral mechanism this feature relies on, not something it modifies. +- `sched_account_jobs()`'s own `JOB_IS_RUN` charging gate (`scheduler.c:1258`) — unchanged; it is exactly the mechanism this feature relies on to turn a captured job's RUN state (now set at capture time, per the 2026-09-11 revision above) into an actual resource charge. - The match/capture mechanics once a job is eligible (jobid matching, ownership transfer of `xce_job_alloc_t`, `make_job_account()`, shift-down compaction, decision reporting via `make_dispatch_decision()`/`sim_print_decision()`/`xce_repl_check_decision()`) — all unchanged, all already correct per `scheduler-sim-decision-parity`. -- Normal-mode (`driver.c`) behavior, simulation mode's live picker, `--sim-verify`'s comparison algorithm itself, and any `job->state` transition logic. +- Normal-mode (`driver.c`) behavior, simulation mode's live picker, `--sim-verify`'s comparison algorithm itself, and any NON-running-state job transition logic (DONE/EXIT/preemption) — the RUN transition triggered by capture is now IN boundary as of the 2026-09-11 revision (see Overview Correction). ### Allowed Dependencies - `sim->core.jobs` iteration and `decoded_session->decisions` (both already owned/populated by `sim_reconcile_unaccounted_jobs()`'s existing implementation) — P0. @@ -66,13 +68,13 @@ flowchart TD Gate -- no --> NextJob[skip already-captured job] Gate -- yes --> Search[search this rounds decoded decisions for jobid] Search --> Found{matching DISPATCH found} - Found -- yes --> Capture[capture allocs and accounts, report decision] - Found -- no --> Retry[leave accounts NULL, retry next round] + Found -- yes --> Capture["capture allocs and accounts, set state=RUN synchronously (2026-09-11), report decision"] + Found -- no --> Retry[leave accounts NULL and state unchanged, retry next round] Capture --> Charge[sched_account_jobs] Retry --> Charge NextJob --> Charge Charge --> ChargeGate{job state is RUN and accounts set} - ChargeGate -- yes --> ResInUse[include in resInUse] + ChargeGate -- yes --> ResInUse["include in resInUse -- now true the SAME round as capture, not a later one"] ChargeGate -- no --> Defer[exclude from resInUse] ``` @@ -111,26 +113,26 @@ sequenceDiagram Replay->>Recon: round N decoded decisions include DISPATCH for job X Note over Job: job X state is PEND at round N - Recon->>Job: capture allocs and accounts from DISPATCH + Recon->>Job: capture allocs and accounts from DISPATCH; set state=RUN synchronously (2026-09-11) Recon->>Recon: report decision at round N - Acct->>Job: check state is RUN - Note over Acct: state is PEND, skip charging - Replay->>Job: round N+k replays job X state change to RUN - Recon->>Job: accounts already set, skip re-search Acct->>Job: check state is RUN and accounts set - Acct->>Acct: charge resInUse from round N+k onward + Acct->>Acct: charge resInUse starting round N (same round as capture) + Replay->>Job: round N+k replays job X's separately-recorded state-change-to-RUN event + Note over Job: state already RUN -- idempotent re-affirmation, not a new transition + Recon->>Job: accounts already set, skip re-search ``` -**Key decisions not obvious from the diagram**: the decision is reported once, at round N (the historically accurate round, Requirement 2.1) — not at round N+k when charging actually begins. Charging timing (round N+k) and reporting timing (round N) are intentionally decoupled, mirroring how normal mode's own `job->state` display and `resInUse` accounting are already decoupled from `sched_commit_decision()`'s synchronous capture. +**Key decisions not obvious from the diagram (revised 2026-09-11)**: the decision is reported at round N (the historically accurate round, Requirement 2.1). As of the 2026-09-11 revision, charging ALSO begins at round N, not round N+k — `job->state` is set to RUN synchronously with capture, mirroring normal-mode's own `sched_commit_decision()` exactly (bug-1624), so reporting and charging are no longer decoupled by round. The separately-recorded `JOB_STATE_CHANGE`/`JOB_UPDATE`-to-RUN event at round N+k, when it later replays, is a safe no-op re-affirmation of an already-RUN job (`core_events.c`'s handler simply reassigns `job->state = new_state`, RUN → RUN, with no guard triggered) — it is preserved in the replay stream and still processed, but no longer the operative trigger for the job's own running-state transition or its resource charge. ## Requirements Traceability | Requirement | Summary | Components | Interfaces | Flows | |-------------|---------|------------|------------|-------| | 1.1 | Capture at originating round regardless of state | `sim_reconcile_unaccounted_jobs` | trigger gate | Search/Capture | -| 1.2 | Captured-but-not-RUN excludes from resInUse | `sched_account_jobs` (unchanged, relied upon) | `JOB_IS_RUN` charge gate | ChargeGate/Defer | -| 1.3 | RUN transition includes in resInUse from that point | `sched_account_jobs` (unchanged, relied upon) | `JOB_IS_RUN` charge gate | ChargeGate/ResInUse | -| 1.4 | Matches normal-mode's equivalent-point reporting | `sched_account_jobs` (shared with normal mode) | n/a | n/a | +| 1.2 (revised 2026-09-11) | Capture ALSO transitions job to RUN synchronously, same round | `sim_reconcile_unaccounted_jobs` | Capture step (new `job->state = JOB_STAT_RUN` write) | Capture | +| 1.3 (revised 2026-09-11) | resInUse includes the job starting the SAME round as capture | `sched_account_jobs` (unchanged, relied upon) | `JOB_IS_RUN` charge gate | ChargeGate/ResInUse | +| 1.4 (new 2026-09-11) | Later separately-recorded confirmation event is an idempotent re-affirmation | `core_events.c`'s `XCE_CORE_EVENT_JOB_STATE_CHANGE` handler (unchanged, verified safe) | n/a | n/a | +| 1.5 | Matches normal-mode's equivalent-point reporting | `sched_account_jobs` (shared with normal mode) | n/a | n/a | | 2.1 | Decision visible at historically accurate round | `sim_reconcile_unaccounted_jobs` (decision reporting, unchanged) | `make_dispatch_decision`/`sim_print_decision`/`xce_repl_check_decision` | Capture step | | 2.2 | Original decision type/detail preserved | `make_dispatch_decision` (unchanged) | n/a | Capture step | | 2.3 | No duplicate report when live decision also exists | live picker (unchanged, PEND-only — structural invariant) | n/a | n/a | @@ -171,12 +173,17 @@ sequenceDiagram ##### Service Interface ```c -/* Unchanged signature. Behavior change: a job is now eligible for capture - * whenever job->accounts == NULL, regardless of job->state (previously - * additionally required JOB_IS_RUN(job->state)). Capturing a job's - * allocation while it is not yet RUN is inert until sched_account_jobs()'s - * own JOB_IS_RUN gate later includes it -- this function performs no - * charging itself. */ +/* Unchanged signature. Behavior change #1 (original): a job is now + * eligible for capture whenever job->accounts == NULL, regardless of + * job->state (previously additionally required JOB_IS_RUN(job->state)). + * Behavior change #2 (2026-09-11 revision): a successful capture ALSO + * sets job->state = JOB_STAT_RUN synchronously, in the same operation -- + * mirroring scheduler.c's sched_commit_decision() (bug-1624) exactly -- + * so sched_account_jobs()'s own unchanged JOB_IS_RUN gate includes the + * job starting THIS round, not a later one. This function still performs + * no charging itself; it now also performs no NON-running state + * transition (DONE/EXIT/preemption remain exclusively triggered by their + * own separately-recorded events, unchanged). */ static void sim_reconcile_unaccounted_jobs(xce_sched_sim_driver_t* sim, replay_session_t* decoded_session, sched_record_session_ctx_t* record_ctx); @@ -208,7 +215,7 @@ static void sim_reconcile_unaccounted_jobs(xce_sched_sim_driver_t* sim, - Existing `test_reconcile_reconstructs_allocs_and_accounts_from_matching_dispatch`, `test_reconcile_ignores_non_dispatch_entry_with_matching_jobid`, `test_reconcile_compacts_via_shift_down_preserving_order`, `test_reconcile_is_safe_to_recheck_on_a_later_round`, `test_reconcile_then_release_via_job_state_change_is_asan_clean`, `test_reconcile_null_args_are_a_safe_noop` — re-run unmodified as regression coverage (Requirement 4.1); no expected behavior change for an already-RUN job. ### Integration Tests (round-boundary + charging interaction) -- New: `test_captured_pend_job_excluded_from_resinuse_until_run` — drive `sim_reconcile_unaccounted_jobs()` then `sched_account_jobs()` back to back for a PEND job with a freshly-captured account; assert `w->resInUse`/`core->sharedInUse` stay zero (Requirement 1.2), then transition the job to RUN and re-run `sched_account_jobs()`; assert `resInUse` now reflects it (Requirement 1.3). -- New (resource-contention fixture, per `scheduler-driver-equivalence.md`'s explicit discipline): a worker with capacity for N jobs, more than N candidate jobs recorded, dispatch decisions and RUN confirmations separated across multiple rounds — assert the live allocator's own capacity check (`sched_check_host_satisfy_`) sees correct `resInUse` at each round, not permanently-zero capacity. A no-contention fixture is explicitly insufficient here (`scheduler-driver-equivalence.md`). +- **Revised 2026-09-11**: `test_captured_pend_job_excluded_from_resinuse_until_run` — RENAMED/INVERTED to `test_captured_job_included_in_resinuse_same_round_as_capture`. Drive `sim_reconcile_unaccounted_jobs()` for a PEND job with a matching recorded DISPATCH entry, then assert `job->state == JOB_STAT_RUN` immediately (Requirement 1.2) and, after running `sched_account_jobs()` in the SAME round, that `w->resInUse`/`core->sharedInUse` already reflect it (Requirement 1.3) — NOT deferred to a later round. Add a companion assertion that a later, separately-recorded `JOB_STATE_CHANGE`-to-RUN event for the same job (applied via the normal `XCE_CORE_EVENT_JOB_STATE_CHANGE` path) is a safe no-op that does not double-charge or corrupt `resInUse` (Requirement 1.4). +- New (resource-contention fixture, per `scheduler-driver-equivalence.md`'s explicit discipline): a worker with capacity for N jobs, more than N candidate jobs recorded, dispatch decisions and RUN confirmations separated across multiple rounds — assert the live allocator's own capacity check (`sched_check_host_satisfy_`) sees correct `resInUse` starting at the CAPTURE round (revised from "the confirmation round"), not permanently-zero capacity. A no-contention fixture is explicitly insufficient here (`scheduler-driver-equivalence.md`). - New: `test_sim_verify_no_false_mismatch_across_round_separated_dispatch` — a `--sim-verify` run over a replay where dispatch and RUN confirmation are several rounds apart reports zero mismatches (Requirement 4.2). - Existing `test_repl_equivalence.c`-style interactive/batch parity coverage re-run unmodified — this feature's call site is shared between both modes, so no new test is needed to prove Requirement 4.3, only confirmation the existing suite still passes. diff --git a/.kiro/specs/scheduler-sim-eager-reconciliation/requirements.md b/.kiro/specs/scheduler-sim-eager-reconciliation/requirements.md index 5a46bb4b..39712067 100644 --- a/.kiro/specs/scheduler-sim-eager-reconciliation/requirements.md +++ b/.kiro/specs/scheduler-sim-eager-reconciliation/requirements.md @@ -10,8 +10,9 @@ What should change: simulation mode's recorded-decision capture should be decoup This is a follow-up to the `scheduler-sim-decision-parity` spec (implementation-complete, all 10 tasks landed and reviewed) -- do not reopen or modify that spec's own requirements/design/tasks. This is its own new spec, building on that feature's existing `sim_reconcile_unaccounted_jobs()` mechanism and round-boundary wiring. ## Boundary Context -- **In scope**: capturing a job's recorded DISPATCH allocation data as soon as simulation-mode replay encounters it in a round's recorded decision stream, regardless of the job's current state at that point; preserving correct, state-gated resource-in-use accounting once capture no longer depends on same-round state coincidence; making the dispatch decision observable at the historically accurate round; preventing double-capture or fabricated data across this wider round span. -- **Out of scope**: any change to how or when a job's state itself transitions (PEND/RUN/DONE/etc.) -- state transition timing and triggers are unchanged, only when allocation data is captured for accounting purposes changes; any change to `--sim-verify`'s comparison semantics beyond not producing a new false mismatch as a side effect; any change to simulation mode's live picker (it remains PEND-only for freshly-arriving jobs, unchanged, out of this feature's scope per `scheduler-allocation`); any change to PEND or PREEMPT decision handling; reopening or modifying `scheduler-sim-decision-parity`'s own requirements, design, or tasks; any change to normal-mode (non-simulation) behavior. +- **In scope**: capturing a job's recorded DISPATCH allocation data as soon as simulation-mode replay encounters it in a round's recorded decision stream, regardless of the job's current state at that point; transitioning that job to a running state synchronously with capture, mirroring normal-mode's own dispatch-commit semantics (see Requirement 1's 2026-09-11 revision below); preserving correct resource-in-use accounting once capture no longer depends on same-round state coincidence; making the dispatch decision observable at the historically accurate round; preventing double-capture or fabricated data across this wider round span. +- **Out of scope**: any change to how or when a job transitions to a NON-running terminal or intermediate state (DONE/EXIT/preemption, etc.) -- those triggers and semantics are unchanged; any change to `--sim-verify`'s comparison semantics beyond not producing a new false mismatch as a side effect; any change to simulation mode's live picker (it remains PEND-only for freshly-arriving jobs, unchanged, out of this feature's scope per `scheduler-allocation`); any change to PEND or PREEMPT decision handling; reopening or modifying `scheduler-sim-decision-parity`'s own requirements, design, or tasks; any change to normal-mode (non-simulation) behavior. + - **Revision (2026-09-11)**: the original version of this bullet excluded ANY change to job->state transition timing, including the RUN transition. That exclusion is REVISED by Requirement 1's 2026-09-11 update below -- the RUN transition itself is now explicitly in scope, triggered by dispatch-decision capture rather than by the separately-recorded confirmation event. Every other state transition (DONE/EXIT/preemption) remains untouched and out of scope, as originally stated. - **Adjacent expectations**: this feature extends `scheduler-sim-decision-parity`'s existing Recorded-Decision Reconciliation mechanism and round-boundary wiring; it does not replace or duplicate that mechanism. It relies on that feature's decision-reporting channel and resource-accounting service remaining exactly as delivered. ## Requirements @@ -19,11 +20,14 @@ This is a follow-up to the `scheduler-sim-decision-parity` spec (implementation- ### Requirement 1: Cross-round resource-accounting accuracy for dispatched jobs **Objective:** As an operator inspecting simulation-mode resource state, I want a job's resource consumption reflected correctly regardless of how many rounds separate its dispatch from its running confirmation, so that what I inspect during a replay is trustworthy even under realistic dispatch-to-confirmation latency. +**Revision (2026-09-11, user-directed, live replay session):** the original version of Acceptance Criteria 2/3 below deferred a captured job's transition to a running state until its separately-recorded `JOB_STATE_CHANGE` confirmation event replayed, potentially many rounds after its dispatch decision was captured -- reproducing a live-observed operator confusion (`info jobs`/`info worker` showing a job as still PEND for multiple sessions after `[sim] DISPATCH` had already printed for it). Investigation confirmed normal-mode execution does NOT work this way: `sched_commit_decision()` (`scheduler.c`, bug-1624) sets `job->state = JOB_STAT_RUN` synchronously, in the SAME operation that captures the dispatch allocation -- from the live scheduler's own perspective, a job IS running the instant the scheduler decides to dispatch it, independent of whatever later, separately-recorded signal confirms the worker actually started it. The Acceptance Criteria below are revised so simulation-mode reconciliation reproduces this same scheduler's-eye-view semantics instead of the worker-confirmation-timeline semantics the original version specified. This is a deliberate, explicit decision to prioritize matching live-mode's own internal state model over preserving the separately-recorded confirmation event's timing as a distinct, independently meaningful signal -- see `research.md`'s corresponding entry for the full rationale trail and the design point that made this ambiguous in the first place. + #### Acceptance Criteria 1. When simulation-mode replay encounters a round containing a job's originating dispatch decision, the scheduler shall capture that decision's resource-allocation data for the job, regardless of the job's current state at that point. -2. While a job's resource-allocation data has been captured but the job has not yet transitioned to a running state, the scheduler shall not include that job's resource consumption in any worker or shared resource-in-use total. -3. When a job whose resource-allocation data was already captured subsequently transitions to a running state, the scheduler shall include that job's resource consumption in its worker's and any shared resource's resource-in-use totals from that point forward, regardless of how many rounds elapsed between capture and the transition. -4. The scheduler shall report the same resource-in-use totals for a dispatched-then-confirmed-later job as normal-mode execution reported at the equivalent point in the original run, regardless of the round separation between dispatch and confirmation. +2. When the scheduler captures a job's originating dispatch decision, the scheduler shall also transition that job to a running state at that same round, synchronously with the capture -- mirroring normal-mode execution's own dispatch-commit semantics (Requirement 1's 2026-09-11 revision above), in which a job is considered running from the scheduler's perspective the instant the scheduler decides to dispatch it. +3. The scheduler shall include a captured job's resource consumption in its worker's and any shared resource's resource-in-use totals starting from the same round its dispatch decision is captured, not deferred to a later round. +4. When a job's dispatch decision and its separately-recorded running-state-confirmation event (`JOB_STATE_CHANGE`/`JOB_UPDATE` to a running state) land in different rounds, replaying that later confirmation event shall re-affirm an already-running job (a safe, idempotent re-application) rather than being the sole trigger for the transition to a running state. +5. The scheduler shall report the same resource-in-use totals for a dispatched job as normal-mode execution reported at the equivalent point in the original run, regardless of the round separation between the dispatch decision and its separately-recorded confirmation event. ### Requirement 2: Decision visibility at the historically accurate round **Objective:** As an operator watching simulation-mode decision output, I want a job's dispatch decision to become visible when the replay reaches the round it was actually decided in, so I have an accurate picture of when scheduling activity happened, not only that it eventually happened. diff --git a/.kiro/specs/scheduler-sim-eager-reconciliation/research.md b/.kiro/specs/scheduler-sim-eager-reconciliation/research.md index 6b01e7d2..143dbdca 100644 --- a/.kiro/specs/scheduler-sim-eager-reconciliation/research.md +++ b/.kiro/specs/scheduler-sim-eager-reconciliation/research.md @@ -52,6 +52,11 @@ - **Findings**: `sim_reconcile_unaccounted_jobs()` never sets `job->state` anywhere — replay-mode state transitions come exclusively from the separately-recorded `XCE_CORE_EVENT_JOB_STATE_CHANGE` event stream, applied via `core_events.c`, independent of whatever the decision stream contains. `sched_commit_decision()`'s `job->state = JOB_STAT_RUN` (bug-1624's fix) is a direct in-memory field write in the *live* scheduler process — not a call into the core's event-push pipeline (`xce_core_push_events()`) — so it is not, by itself, a recordable core event. The recorded `JOB_STATE_CHANGE` event a replay later applies still originates from whatever asynchronous signal produces it for recording purposes (the controller's own worker-confirmation path), independent of bug-1624's live in-memory synchronization. - **Implications**: bug-1624 fixed a live-mode admission-safety window; it does not change what ends up in a recording's event stream or its round timing. This spec's premise — a recorded DISPATCH decision and its recorded RUN-state-change event can land in different replay rounds — holds for recordings made before *and* after bug-1624, and is not something a future reader should mistake as already resolved by that unrelated fix. +- **Revision (2026-09-11, user-directed, live replay session)**: the finding above is factually unchanged (bug-1624 genuinely does not alter what a recording's own event stream contains or when). What changed is the PROJECT's decision about which of two defensible readings of "job->state during replay" to implement, given that factual finding: + - **(a) Scheduler's-own-view** (adopted 2026-09-11): reconstruct what the ORIGINAL live scheduler process's `job->state` field actually was at the round a DISPATCH decision is captured — which, per bug-1624, was already `JOB_STAT_RUN`, synchronously, the instant that decision was originally committed. Under this reading, the separately-recorded `JOB_STATE_CHANGE` confirmation event is a genuinely real signal (the worker's own confirmation, useful for other purposes) but is NOT the thing `job->state` during replay should be tracking, since the live scheduler's own in-memory state had already moved on by the time that confirmation was recorded. + - **(b) Worker-confirmation-timeline** (the design this spec originally shipped): treat the separately-recorded `JOB_STATE_CHANGE` event's own timing as the operative signal for `job->state`, faithfully reproducing exactly when the worker confirmation arrived relative to other replay activity — useful for debugging dispatch-to-confirmation latency itself, but at the cost of `job->state` not matching what the live scheduler's own field actually held at the same point in the original run. + - Live operator use surfaced (b)'s cost directly and confusingly (a job showing PEND for multiple sessions after its own `[sim] DISPATCH` line had already printed), and the explicit decision was made to adopt (a) instead, matching normal-mode's own synchronous semantics exactly rather than the separately-recorded confirmation stream's timing. This is a genuine design trade-off, not a bug fix to a previously-wrong factual claim — see `requirements.md`'s Requirement 1 and `design.md`'s Overview, both revised 2026-09-11 to reflect (a). + ### Existing test locking in the bug - **Context**: identifying what existing coverage this feature must invalidate, not just extend. - **Sources Consulted**: `src/scheduler/testsuite/test_sim_reconcile_unaccounted.c` (grepped test names, read `test_reconcile_skips_non_run_state_job` in full at lines 507-531). diff --git a/.kiro/specs/scheduler-sim-eager-reconciliation/spec.json b/.kiro/specs/scheduler-sim-eager-reconciliation/spec.json index bd8b2f8e..166fba39 100644 --- a/.kiro/specs/scheduler-sim-eager-reconciliation/spec.json +++ b/.kiro/specs/scheduler-sim-eager-reconciliation/spec.json @@ -1,7 +1,7 @@ { "feature_name": "scheduler-sim-eager-reconciliation", "created_at": "2026-08-22T14:50:13Z", - "updated_at": "2026-08-27T06:13:12Z", + "updated_at": "2026-09-11T09:45:00Z", "language": "en", "phase": "implementation-complete", "approvals": { diff --git a/.kiro/specs/scheduler-sim-eager-reconciliation/tasks.md b/.kiro/specs/scheduler-sim-eager-reconciliation/tasks.md index c72dce15..cd958ebb 100644 --- a/.kiro/specs/scheduler-sim-eager-reconciliation/tasks.md +++ b/.kiro/specs/scheduler-sim-eager-reconciliation/tasks.md @@ -51,6 +51,23 @@ - _Requirements: 3.3, 4.1, 4.3, 4.5_ - _Depends: 3.1, 3.2_ +- [x] 4. Revision (2026-09-11, user-directed): align captured job's `job->state` with normal-mode's synchronous dispatch-commit semantics +- [x] 4.1 Set `job->state = JOB_STAT_RUN` synchronously at capture, mirroring `sched_commit_decision()` (bug-1624) + - Add `job->state = JOB_STAT_RUN;` to `sim_reconcile_unaccounted_jobs()`'s FOUND branch (`sim_loop.c`), in the same place `job->allocs`/`job->accounts` are set, so a captured job is RUN from the scheduler's own perspective starting the SAME round as capture -- not deferred to whenever its separately-recorded `JOB_STATE_CHANGE` confirmation event later replays + - Update the function's own header/doc comments to describe the new synchronous state transition alongside the existing state-independent capture-eligibility description + - Observable: `info jobs`/`info worker` show a job as RUN in the same session its `[sim] DISPATCH` line prints, matching what a live scheduler process's own `job->state` field would have shown at that point + - _Requirements: 1.2 (revised), 1.3 (revised)_ + - _Boundary: sim_reconcile_unaccounted_jobs_ + +- [x] 4.2 Update/replace the test(s) that pinned the deferred-until-confirmation behavior as correct + - Rename/invert `test_captured_pend_job_excluded_from_resinuse_until_run` to `test_captured_job_included_in_resinuse_same_round_as_capture`: assert `job->state == JOB_STAT_RUN` immediately after capture, and `resInUse` reflects the charge after the SAME round's `sched_account_jobs()` call, not a later one + - Add a companion assertion that a later, separately-recorded `JOB_STATE_CHANGE`-to-RUN event replayed for the same (already-RUN) job is a safe idempotent no-op -- no double-charge, no crash, no regressed terminal-state guard + - Confirm RED first (temporarily revert the `job->state = JOB_STAT_RUN` line and rebuild) before confirming GREEN with it restored + - Observable: the updated/new tests fail without the `job->state = JOB_STAT_RUN` line and pass with it; full scheduler suite shows no new regressions + - _Requirements: 1.2 (revised), 1.3 (revised), 1.4 (new)_ + - _Boundary: sim_reconcile_unaccounted_jobs, sched_account_jobs, core_events.c (verification only, no code change expected there)_ + - _Depends: 4.1_ + ## Implementation Notes - Task 1.2: the pre-existing `test_recordstart_symlink_escape_replies_error` failure (bug-1560, `test_admin_request.c`) is unrelated to this spec and appears in every task's full-suite run — do not treat it as a regression. diff --git a/.wolf/anatomy-index.json b/.wolf/anatomy-index.json index 0054b0e1..7bb7dd9f 100644 --- a/.wolf/anatomy-index.json +++ b/.wolf/anatomy-index.json @@ -1,13 +1,13 @@ { "version": 1, "meta": { - "lastScanned": "2026-09-08T11:07:37.227Z", - "fileCount": 520, + "lastScanned": "2026-09-11T11:32:34.355Z", + "fileCount": 531, "hits": 0, "misses": 0, - "renderedHash": "a2cea6ea904bba6d2560cb159eb016cd7dedbe3c914fd04a2936080d17b6d4a4", - "storeUpdatedAt": "2026-09-08T11:07:37.235Z", - "rootHash": "42bc1856659f4da8" + "renderedHash": "33fd3fb89102d63e9b4431d49b33d7db59e3b7a1038c68f9c3f47eb849642bd2", + "storeUpdatedAt": "2026-09-11T11:32:34.363Z", + "rootHash": "92eb48ac76f08b14" }, "files": { ".agent-shell/transcripts/2026-08-11-11-39-59.md": { @@ -3232,53 +3232,48 @@ }, ".kiro/specs/scheduler-sim-eager-reconciliation/design.md": { "description": "Design Document", - "tokens": 5580, - "hash": "aeef295f5a43e930", - "size": 22411, - "mtimeMs": 1788766157571.1833, - "updatedAt": "2026-09-08T03:09:38.917Z", - "source": "scan", - "importance": 0.1036 + "tokens": 6493, + "hash": "e1712bb6eb14a47b", + "size": 26074, + "mtimeMs": 1789122761693.9182, + "updatedAt": "2026-09-11T10:32:41.734Z", + "source": "hook" }, ".kiro/specs/scheduler-sim-eager-reconciliation/requirements.md": { "description": "Requirements Document", - "tokens": 2312, - "hash": "8299f9e42c6289ac", - "size": 9247, - "mtimeMs": 1787410336086.588, - "updatedAt": "2026-09-08T03:09:38.917Z", - "source": "scan", - "importance": 0.1036 + "tokens": 2957, + "hash": "d4e3125d696b25d8", + "size": 11826, + "mtimeMs": 1789122511639.8337, + "updatedAt": "2026-09-11T10:28:31.934Z", + "source": "hook" }, ".kiro/specs/scheduler-sim-eager-reconciliation/research.md": { "description": "Research & Design Decisions", - "tokens": 4657, - "hash": "f8516de9056ed5ce", - "size": 18701, - "mtimeMs": 1787799324441.019, - "updatedAt": "2026-09-08T03:09:38.917Z", - "source": "scan", - "importance": 0.1036 + "tokens": 5166, + "hash": "898efe2454a970a7", + "size": 20742, + "mtimeMs": 1789122782506.744, + "updatedAt": "2026-09-11T10:33:02.552Z", + "source": "hook" }, ".kiro/specs/scheduler-sim-eager-reconciliation/spec.json": { "description": "", "tokens": 135, - "hash": "43d3b56694f90a0d", + "hash": "377d12c2909a6790", "size": 472, - "mtimeMs": 1788428267490.361, - "updatedAt": "2026-09-08T03:09:38.917Z", - "source": "scan", - "importance": 0.1036 + "mtimeMs": 1789122958918.679, + "updatedAt": "2026-09-11T10:35:58.957Z", + "source": "hook" }, ".kiro/specs/scheduler-sim-eager-reconciliation/tasks.md": { "description": "Implementation Plan", - "tokens": 1748, - "hash": "97c985b251ea97f9", - "size": 6998, - "mtimeMs": 1788428261573.9675, - "updatedAt": "2026-09-08T03:09:38.917Z", - "source": "scan", - "importance": 0.1036 + "tokens": 2306, + "hash": "6310d2ad5ea30170", + "size": 9229, + "mtimeMs": 1789126354308.741, + "updatedAt": "2026-09-11T11:32:34.355Z", + "source": "hook" }, ".kiro/specs/scheduler-sim-reconcile-terminal-guard/design.md": { "description": "Design Document", @@ -9846,11 +9841,11 @@ }, "src/scheduler/src/resource.c": { "description": "include ", - "tokens": 8748, - "hash": "7cebbf2308d7cecd", - "size": 32808, - "mtimeMs": 1788865587298.907, - "updatedAt": "2026-09-08T11:06:27.345Z", + "tokens": 9227, + "hash": "ca6ef787567c9f0d", + "size": 34606, + "mtimeMs": 1789114397450.9985, + "updatedAt": "2026-09-11T08:13:17.493Z", "source": "hook" }, "src/scheduler/include/scheduler.h": { @@ -9873,11 +9868,11 @@ }, "src/scheduler/testsuite/test_resource_accounting.c": { "description": "Declares pulled", - "tokens": 13505, - "hash": "398d76345c4bcfc7", - "size": 50643, - "mtimeMs": 1788865657178.1558, - "updatedAt": "2026-09-08T11:07:37.227Z", + "tokens": 14464, + "hash": "dfff7b8f3627d03d", + "size": 54238, + "mtimeMs": 1789113953031.7354, + "updatedAt": "2026-09-11T08:05:53.084Z", "source": "hook" }, "src/scheduler/testsuite/test_seqjob_allocator.c": { @@ -9897,6 +9892,105 @@ "mtimeMs": 1788863804898.0203, "updatedAt": "2026-09-08T10:36:44.938Z", "source": "hook" + }, + "src/scheduler/testsuite/test_repl_resource_table.c": { + "description": "Declares char", + "tokens": 21755, + "hash": "dd600ba102f485ac", + "size": 81581, + "mtimeMs": 1789112203009.6707, + "updatedAt": "2026-09-11T07:36:43.058Z", + "source": "hook" + }, + "src/scheduler/src/repl_info.c": { + "description": "include ", + "tokens": 16226, + "hash": "93051e2f01c3c047", + "size": 60847, + "mtimeMs": 1789119601249.3374, + "updatedAt": "2026-09-11T09:40:01.301Z", + "source": "hook" + }, + "src/scheduler/src/repl_internal.h": { + "description": "if !defined(_REPL_INTERNAL_H_)", + "tokens": 7304, + "hash": "24afb3229d6ca8bf", + "size": 27387, + "mtimeMs": 1789118155563.0457, + "updatedAt": "2026-09-11T09:15:55.610Z", + "source": "hook" + }, + "src/scheduler/src/repl_break.c": { + "description": "include ", + "tokens": 9921, + "hash": "9310483fabc9331d", + "size": 37203, + "mtimeMs": 1789118179109.396, + "updatedAt": "2026-09-11T09:16:19.153Z", + "source": "hook" + }, + "src/scheduler/testsuite/test_repl_breakpoints_inspection.c": { + "description": "Declares and", + "tokens": 23533, + "hash": "4be7ad8f51665b9c", + "size": 88246, + "mtimeMs": 1789118449404.6335, + "updatedAt": "2026-09-11T09:20:49.445Z", + "source": "hook" + }, + "src/scheduler/testsuite/test_repl_help.c": { + "description": "Declares error", + "tokens": 6047, + "hash": "fd7bce4b295b3f60", + "size": 22673, + "mtimeMs": 1789118187832.9336, + "updatedAt": "2026-09-11T09:16:27.880Z", + "source": "hook" + }, + "src/scheduler/src/sim.c": { + "description": "include ", + "tokens": 7432, + "hash": "8faca8e26f0c65c3", + "size": 27870, + "mtimeMs": 1789118673321.5144, + "updatedAt": "2026-09-11T09:24:33.371Z", + "source": "hook" + }, + "src/scheduler/include/sim.h": { + "description": "if !defined(_SIM_H_)", + "tokens": 4213, + "hash": "2832b81c98fbb84a", + "size": 15797, + "mtimeMs": 1789118690779.5962, + "updatedAt": "2026-09-11T09:24:50.824Z", + "source": "hook" + }, + "src/scheduler/src/sim_loop.c": { + "description": "include ", + "tokens": 17336, + "hash": "480088b526e953bf", + "size": 65007, + "mtimeMs": 1789125652889.2712, + "updatedAt": "2026-09-11T11:20:52.930Z", + "source": "hook" + }, + "src/scheduler/testsuite/test_sim_reconcile_unaccounted.c": { + "description": "Declares documents", + "tokens": 26015, + "hash": "da7d252221e88c98", + "size": 97554, + "mtimeMs": 1789126034777.6755, + "updatedAt": "2026-09-11T11:27:14.815Z", + "source": "hook" + }, + "src/scheduler/testsuite/test_sim_contention_parity.c": { + "description": "Declares calls", + "tokens": 19164, + "hash": "d502f4092e147d70", + "size": 71865, + "mtimeMs": 1789125200160.1223, + "updatedAt": "2026-09-11T11:13:20.196Z", + "source": "hook" } } } \ No newline at end of file diff --git a/.wolf/anatomy.md b/.wolf/anatomy.md index b1d426d3..80114d34 100644 --- a/.wolf/anatomy.md +++ b/.wolf/anatomy.md @@ -1,7 +1,7 @@ # anatomy.md -> Auto-maintained by OpenWolf. Last scanned: 2026-09-08T11:07:37.227Z -> Files: 520 tracked | Anatomy hits: 0 | Misses: 0 +> Auto-maintained by OpenWolf. Last scanned: 2026-09-11T11:32:34.355Z +> Files: 531 tracked | Anatomy hits: 0 | Misses: 0 ## ./ @@ -519,11 +519,11 @@ ## .kiro/specs/scheduler-sim-eager-reconciliation/ -- `design.md` — Design Document (~5580 tok) -- `requirements.md` — Requirements Document (~2312 tok) -- `research.md` — Research & Design Decisions (~4657 tok) +- `design.md` — Design Document (~6493 tok) +- `requirements.md` — Requirements Document (~2957 tok) +- `research.md` — Research & Design Decisions (~5166 tok) - `spec.json` (~135 tok) -- `tasks.md` — Implementation Plan (~1748 tok) +- `tasks.md` — Implementation Plan (~2306 tok) ## .kiro/specs/scheduler-sim-reconcile-terminal-guard/ @@ -847,11 +847,17 @@ ## src/scheduler/include/ - `scheduler.h` — if !defined(_SCHEDULER_H_) (~4716 tok) +- `sim.h` — if !defined(_SIM_H_) (~4213 tok) ## src/scheduler/src/ - `plugin.c` — include (~4554 tok) -- `resource.c` — include (~8748 tok) +- `repl_break.c` — include (~9921 tok) +- `repl_info.c` — include (~16226 tok) +- `repl_internal.h` — if !defined(_REPL_INTERNAL_H_) (~7304 tok) +- `resource.c` — include (~9227 tok) +- `sim_loop.c` — include (~17336 tok) +- `sim.c` — include (~7432 tok) ## src/scheduler/src/plugins/ @@ -862,5 +868,10 @@ - `test_core_events.c` — Declares and (~18252 tok) - `test_parjob_combinatorial.c` — Declares of (~12038 tok) -- `test_resource_accounting.c` — Declares pulled (~13505 tok) +- `test_repl_breakpoints_inspection.c` — Declares and (~23533 tok) +- `test_repl_help.c` — Declares error (~6047 tok) +- `test_repl_resource_table.c` — Declares char (~21755 tok) +- `test_resource_accounting.c` — Declares pulled (~14464 tok) - `test_seqjob_allocator.c` — Declares char (~3701 tok) +- `test_sim_contention_parity.c` — Declares calls (~19164 tok) +- `test_sim_reconcile_unaccounted.c` — Declares documents (~26015 tok) diff --git a/.wolf/buglog.json b/.wolf/buglog.json index fb1e7a8e..ee32f3f8 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -21666,6 +21666,199 @@ ], "occurrences": 1, "last_seen": "2026-09-10T06:57:00Z" + }, + { + "id": "bug-1799", + "timestamp": "2026-09-11T07:32:00Z", + "error_message": "User-reported (xce_scheduler -i -s , live cluster-captured replay /tmp/scheduler-replay-20260911-064117.replay, worker pc13v, 22 real slots): `info worker pc13v` at session=98 shows `resource slots: value=22.00 used=22.00` while the printed `assigned job:` list for that same worker sums to 26 slots (jobs [5]-[9] each *4 = 20, plus six single-slot jobs from a second batch = 6); the very next session (99, no new DISPATCH/PEND events, same assigned-jobs list) shows `used=26.00` -- exceeding the worker's own `value=22.00` capacity, an admission-invariant violation (max slots exceeded). Session 98's own event stream shows two back-to-back `WORKERS_PUSH`+`JOB_PUSH` (njobs=20) events for the same worker/jobids, consistent with a live scheduler-connection resync/reconnect being faithfully replayed (matching bug-1734/bug-1739's documented resync scenario), plus `sched_add_job failed ... rc=-17, merging state into existing tracked job instead of discarding` warnings for nearly every job in the resync batch, and one `resync claims RUN for jobid ...[7]/[8] with no allocation data -- refusing to merge a phantom RUN` (bug-1739's guard firing as designed).", + "file": "src/scheduler/src/scheduler.c, src/scheduler/src/core_session.c", + "root_cause": "Interaction between two independently-reasonable pieces of code that were never reconciled with each other. (1) bug-1751 (2026-09-01, worker-rejoin-resync task 5) deliberately added an `IS_WORKER_RECONCILING(w->status)` skip to `core_reset_workers()` (core_session.c:136-142, the sim/normal-mode-shared per-round resInUse-reset helper called from `xce_core_begin_session()`): a worker mid-resync has its `w->resInUse` zeroing SKIPPED that round, on purpose, so its in-flight usage isn't wiped while reconciliation is still in progress. (2) `sched_account_jobs()` (scheduler.c:1434-1461) is the ONLY thing that re-derives `resInUse`/`sharedInUse` each round -- it walks every job in `core->jobs` and, for each one with `JOB_IS_RUN(job->state) && job->accounts != NULL`, calls `sched_charge_account()` (an unconditional ADD via `sched_update_res_accounts(..., charge=1)`, resource.c:608-633) -- with NO equivalent RECONCILING check and NO reset of its own. Its own documented contract (core.h:311-322) is explicit that callers 'must call this strictly after the session's resval reset' -- a precondition bug-1751's reset-skip silently violates for exactly the RECONCILING window it targets. Consequence: for every round a worker stays RECONCILING (entered via a WORKERS_PUSH re-announce during a scheduler reconnect/resync, exactly what session 98's two WORKERS_PUSH events reflect), every currently-RUN job's slot charge on that worker gets RE-ADDED on top of the NOT-reset resInUse from the prior round, instead of the intended zero-then-resum. This produces exactly the observed shape: `used=` (accumulating additively) trails one round behind the true `assigned jobs` sum at first (22 vs 26 at session 98, since only some of the newly-RUN jobs' charges had been double-counted-in by that round's pass), then overtakes real capacity outright (26 at session 99). This is NOT the same mechanism as bug-1739 (JOBS_PUSH resync-merge phantom-RUN guard, which governs whether `job->allocs`/`state` get attached at all) nor bug-1624 (already-fixed dispatch-to-RUN synchronization gap) nor bug-1727 (already-fixed `info worker` display double-count of `value=`) -- all four live in the same resource-accounting neighborhood but are structurally distinct bugs. Root cause identified via static trace only (codegraph_explore + direct read of core_session.c/scheduler.c/core.h); not yet reproduced with an isolated minimal repro or instrumented trace confirming the exact round-by-round resInUse progression.", + "fix": "NOT YET FIXED -- investigation only, per user request (this session's live replay-driven Q&A, not yet escalated to an implementation task). Likely fix direction: give `sched_account_jobs()` the SAME `IS_WORKER_RECONCILING` awareness `core_reset_workers()`/`sched_reset_worker_()` already have (bug-1751) -- either (a) skip charging jobs whose host is currently RECONCILING (mirroring the reset-skip so the pair stays consistent: neither reset nor resum touches a RECONCILING worker's resInUse until it clears), or (b) have `core_reset_workers()` NOT skip the reset for a RECONCILING worker but instead have `sched_account_jobs()` re-derive from job state alone regardless of RECONCILING (i.e. resInUse is always a clean per-round sum, and RECONCILING-safety is enforced elsewhere, e.g. admission control refusing new dispatches to a RECONCILING host, which per bug-1751/bug-1745 already exists as a separate candidate-filter check). Needs the worker-rejoin-resync spec's own design intent re-read (.kiro/specs/worker-rejoin-resync/) before picking (a) vs (b), plus a live repro (re-run this exact replay file with an instrumented sched_account_jobs() logging resInUse before/after each round) to confirm the round-by-round mechanism before implementing, and full TDD (scheduler-driver-equivalence steering requires normal-mode/sim-mode parity for any fix here).", + "tags": [ + "scheduler", + "sim", + "replay", + "resource-accounting", + "resInUse", + "sched_account_jobs", + "worker-rejoin-resync", + "IS_WORKER_RECONCILING", + "over-commit", + "slots", + "max-capacity-exceeded", + "core_reset_workers", + "bug-1751-interaction", + "not-fixed", + "user-reported", + "found-live" + ], + "related_bugs": [ + "bug-1751", + "bug-1739", + "bug-1734", + "bug-1624", + "bug-1727", + "bug-1613" + ], + "occurrences": 1, + "last_seen": "2026-09-11T06:43:54Z" + }, + { + "id": "bug-1800", + "timestamp": "2026-09-11T07:55:00Z", + "error_message": "User request (live replay session, worker pc13v): `info worker`'s CONSUMABLE value=/used= line for a bytes-valued resource (mem/swp/tmp) printed raw byte counts with no unit scaling -- e.g. `resource mem: value=20684476416.00 used=87960930222080.00` -- unlike every OTHER bytes-valued field in the same listing (maxmem/maxswap/maxtmp already render `value=33182072832.000000 (30.90 GB)`, and mem/tmp/swp themselves render human-scaled via the Load Metrics path when NOT flagged consumable). bug-1727's own header comment had explicitly scoped this out: 'no bytes-valued CONSUMABLE resource is exercised anywhere in this codebase's tests today.'", + "file": "src/scheduler/src/repl_info.c", + "root_cause": "repl_info_worker_one()'s and repl_info_resource_one()'s CONSUMABLE Value_u_fvalue branches both hardcoded `%.2f`/`%.2f` for value=/used=, deliberately bypassing repl_format_numeric_resource_value()'s existing units_mode-aware/repl_is_bytes_valued_resource()-gated formatting that every other numeric field in this listing already goes through -- correct for a non-bytes-valued consumable (slots, licenses) but wrong for mem/tmp/swp, which are bytes-valued CONSUMABLE resources in live practice even though no test exercised that combination before this session.", + "fix": "FIXED (2026-09-11). Added a new shared static helper repl_format_consumable_value_and_used() (repl_info.c, right after repl_format_numeric_resource_value()) that routes value=/used= through the existing repl_format_numeric_resource_value() formatter (units_mode-aware, plus the swp MB->bytes special case) when repl_is_bytes_valued_resource(rname) or rname==\"swp\" is true, and keeps the exact pre-existing \"%.2f\"/\"%.2f\" formatting otherwise (slots/licenses unaffected). Wired into both repl_info_worker_one()'s per-host CONSUMABLE case and repl_info_resource_one()'s GLOBAL/SHARED CONSUMABLE case, which share the fix via this one helper so they can't disagree. Updated the stale doc comments at repl_info.c's Resource Value Formatter section header and both call sites (previously asserted this pair was 'NOT units_mode-aware' and that no bytes-valued CONSUMABLE resource existed in tests -- no longer true). TDD: added 4 new tests to test_repl_resource_table.c (test_info_worker_units_mode_both_renders_raw_and_human_for_consumable_bytes_resource, test_info_worker_units_mode_raw_renders_bare_values_for_consumable_bytes_resource, test_info_worker_units_mode_converts_swp_megabytes_for_consumable_resource, test_info_resource_units_mode_both_renders_raw_and_human_for_consumable_bytes_resource) covering both/raw modes, the swp MB special case, and the info-resource twin. RED confirmed first (4 new failures, 76 pre-existing tests still green, `xce-build.sh scheduler asan xce.build --incremental`); GREEN confirmed after the fix (77/77 Ok, zero regressions). All existing pinned CONSUMABLE value=/used= tests (bug-1601/1613/1727, using \"slots\"/\"licenses\"/\"gpu_licenses\") were verified to use non-bytes-valued resource names before the fix, so they were unaffected by construction, not just by accident.", + "tags": [ + "scheduler", + "repl", + "info-worker", + "info-resource", + "resource-accounting", + "units-mode", + "bytes-human-format", + "mem", + "swp", + "tmp", + "consumable", + "display-feature", + "user-requested", + "fixed", + "tdd" + ], + "related_bugs": [ + "bug-1727", + "bug-1601", + "bug-1613" + ], + "occurrences": 1, + "last_seen": "2026-09-11T07:55:00Z" + }, + { + "id": "bug-1801", + "timestamp": "2026-09-11T08:35:00Z", + "error_message": "Follow-up fix for bug-1799 (user asked to 'fix bug-1799', this session): the resInUse over-commit caused by core_reset_workers()'s RECONCILING/UNAVAIL reset-skip (bug-1751) having no matching skip in sched_account_jobs()'s per-round re-derive pass.", + "file": "src/scheduler/src/resource.c", + "root_cause": "See bug-1799's own entry for the full mechanism. Confirmed exactly as diagnosed: sched_update_account() (resource.c, the function both sched_charge_account() and sched_account_jobs()'s per-round resum ultimately call) had no IS_WORKER_UNAVAIL/IS_WORKER_RECONCILING check in its per-host charge loop, so a job on a worker mid-resync got its charge re-added every round on top of the not-reset total.", + "fix": "FIXED (2026-09-11). Added `if (IS_WORKER_UNAVAIL(w->status) || IS_WORKER_RECONCILING(w->status)) { continue; }` to sched_update_account()'s per-host loop (resource.c), right after resolving the worker `w`, before the sched_update_res_accounts() charge call -- the exact same combined status check core_reset_workers()/sched_reset_worker_() already use for the reset half (bug-1751), restoring the 'resum only what reset also touched' symmetry. Scoped to the per-host branch only: core->sharedInUse's own reset (core_reset_shared_resval()) is unconditional with no per-worker gating, so the shared-resource charge is untouched. This is a single-point fix (sched_update_account() backs both sched_charge_account()/sched_uncharge_account()), so it also applies (harmlessly/defensively -- a RECONCILING/UNAVAIL worker should never be a fresh DISPATCH target per the existing bug-1745/1751 candidate-filter exclusion) to the dispatch-time charge path via sched_commit_job_alloc(). TDD: added test_charge_account_skips_reconciling_or_unavail_worker_host_charge (test_resource_accounting.c) -- charges a job account against a worker flagged RECONCILING then UNAVAIL (per-host resInUse must stay 0 both times, shared resInUse still charges normally), then clears the status bit and re-charges (must resume normally on the very next call, proving this is a per-call skip, not a permanent block). RED confirmed by temporarily reverting the skip and rebuilding (`xce-build.sh scheduler asan xce.build --incremental`): exactly this one new test failed (`Expected 0 Was 2.5`), 76 other tests green. GREEN confirmed after restoring the fix: 77/77 Ok, zero regressions.", + "tags": [ + "scheduler", + "resource-accounting", + "resInUse", + "sched_account_jobs", + "sched_update_account", + "worker-rejoin-resync", + "IS_WORKER_RECONCILING", + "IS_WORKER_UNAVAIL", + "over-commit", + "slots", + "bug-1751-interaction", + "fixed", + "tdd", + "user-requested" + ], + "related_bugs": [ + "bug-1799", + "bug-1751", + "bug-1739", + "bug-1734", + "bug-1624", + "bug-1727" + ], + "occurrences": 1, + "last_seen": "2026-09-11T08:35:00Z" + }, + { + "id": "bug-1802", + "timestamp": "2026-09-11T08:35:00Z", + "error_message": "User request (same live replay session as bug-1799/1800): 'how can we step to the next session which is not dummy(something occurs)?' -- the REPL's `break ` vocabulary required setting all ten per-event breakpoint kinds individually to catch 'the next session with ANY event at all,' with no single convenience kind for that common workflow.", + "file": "src/scheduler/src/repl_break.c, src/scheduler/src/repl_internal.h", + "root_cause": "Not a bug -- a missing convenience feature. BP_KIND_TABLE (repl_break.c) only offered the ten specific per-event kinds (CONFIG_PUSH/JOB_PUSH/.../JOB_RUSAGE) plus BEGIN_ROUND/DECISION; repl_breakpoint_matches_event() compared bp->kind for exact equality against repl_bp_kind_for_core_event(ev->type), with no wildcard option.", + "fix": "FIXED (2026-09-11). Added a new XCE_REPL_BP_ANY enum member (repl_internal.h) and a `{\"ANY\", XCE_REPL_BP_ANY}` BP_KIND_TABLE entry (listed first for discoverability). repl_breakpoint_matches_event() (repl_break.c) now special-cases bp->kind == XCE_REPL_BP_ANY to match any event whose type maps to a real per-event kind (repl_bp_kind_for_core_event() returning nonzero) instead of requiring exact kind equality -- still subject to the breakpoint's own filters (though a bare `break ANY`, the intended usage, has none). Deliberately does NOT match BEGIN_ROUND (fires on every session including dummy ones -- matching it would defeat the whole point) or DECISION (a separate drained-decision hook). `break ANY` + `continue` now stops the REPL at the very next session that decodes at least one real event, transparently skipping any number of dummy (virtual-clock-only) sessions in between, in one command instead of ten. TDD: added test_any_breakpoint_skips_dummy_sessions_to_first_real_event (test_repl_breakpoints_inspection.c, full replay-file + REPL-script Group D fixture) -- two dummy sessions (rounds 1000/2000, zero events each) followed by one JOB_STATE_CHANGE event, `break ANY` + `continue` must land exactly at event #1 having processed both dummy rounds as 'idle' (round-rollup evidence), and must never print BEGIN_ROUND. Also extended the two existing BP_KIND_TABLE token-enumeration tests (test_repl_help.c's help-text test, test_repl_breakpoints_inspection.c's unrecognized-event-type-error test) to include \"ANY\" so both stay genuinely exhaustive against the real table. RED confirmed (the new test failed on the FIRST attempt for an unrelated reason -- the hit message names the breakpoint's own user-typed token, \"ANY\", not the underlying event's type name \"JOB_STATE_CHANGE\", which the test's first draft wrongly assumed; corrected the expected string, not the implementation, which was already correct). GREEN confirmed: 77/77 Ok, zero regressions, alongside bug-1801's fix in the same build.", + "tags": [ + "scheduler", + "repl", + "sim", + "replay", + "breakpoints", + "break", + "ANY", + "dummy-session", + "cli-ux", + "feature", + "user-requested", + "fixed", + "tdd" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-09-11T08:35:00Z" + }, + { + "id": "bug-1803", + "timestamp": "2026-09-11T09:35:00Z", + "error_message": "User feedback (live replay session, immediately after bug-1802's `break ANY` shipped): 'It seems that break any command will introduce too many break in single session' -- live transcript showed `break ANY` + repeated `step` firing 'hit breakpoint 1 @ event #N: any' on EVERY event (JOB_UPDATE, JOB_STATE_CHANGE, ...) inside an already-busy session (session 40), forcing many single-event `step`s just to get back OUT of that session -- the opposite of the intended 'skip past dummy sessions in one hop' workflow. User explicitly requested removing `break ANY` and instead having `info summary` display the next non-dummy session id so `step N` can jump there directly.", + "file": "src/scheduler/src/repl_break.c, src/scheduler/src/repl_internal.h, src/scheduler/src/sim.c, src/scheduler/src/repl_info.c", + "root_cause": "Design mismatch, not a code defect: a breakpoint is inherently a persistent, repeatedly-firing mechanism (correct behavior for its stated purpose -- it fires on every matching event until deleted), but the user's actual need ('skip past a run of dummy sessions to the next one worth looking at') is a ONE-TIME navigation query, not a recurring pause condition. Modeling it as a breakpoint kind (bug-1802) meant it kept firing on every subsequent matching event for the rest of the session, well past the point it had already served its purpose, with no natural way to say 'just this once.'", + "fix": "REVERTED `break ANY` entirely (bug-1802's XCE_REPL_BP_ANY enum member, BP_KIND_TABLE entry, repl_breakpoint_matches_event() special case, and its 3 tests) back to the exact pre-bug-1802 code, confirmed via `grep -rn XCE_REPL_BP_ANY` returning nothing but the historical doc-comment note left in repl_internal.h. REPLACED with a one-time informational display: `info summary` now also prints 'next non-dummy session: N -- run `step M` to reach it' (or 'none remaining'), computed by a new sim_prescan_next_nonempty_session() (sim.c, exported via sim.h) that peeks ahead in sim->replay_file from the current read position (same ftell()/fseek()-around-a-shape-only-scan technique sim_prescan_session_count() already establishes for the startup session-count banner) for the next ReplaySession boundary preceded by at least one real event, then rewinds -- a pure read-only query, never a recurring pause. Returns a RELATIVE 'sessions ahead' count (not an absolute session number) specifically to avoid needing xce_sched_repl_state_t's full definition inside sim.c (an incomplete type there by the project's own Simulation Isolation boundary -- caught by a real compile error on the first attempt, `invalid use of incomplete typedef`, when the function tried to read sim->repl_state->session_count directly); repl_info.c (which has the full repl_internal.h definition) adds the relative count to rs->session_count for the absolute display number. TDD: added test_info_summary_reports_next_nondummy_session_when_present and test_info_summary_reports_none_remaining_when_all_dummy (test_repl_breakpoints_inspection.c). RED confirmed by temporarily hardcoding the summary's prescan result to a sentinel non-0/non-1 value and rebuilding: exactly these 2 new tests failed, 75 others green. GREEN confirmed after restoring: 77/77 Ok, zero regressions. The existing test_info_summary_reports_counts (a bare in-memory fixture with no `.replay` file, sim->replay_file==NULL) is correctly unaffected -- the new line is silently omitted whenever the prescan can't run, mirroring sched_sim_driver_start()'s own 'purely informational, silently skipped on failure' precedent for its sibling prescan function.", + "tags": [ + "scheduler", + "repl", + "sim", + "replay", + "break-ANY", + "reverted", + "info-summary", + "next-non-dummy-session", + "sim_prescan_next_nonempty_session", + "cli-ux", + "user-requested", + "fixed", + "tdd", + "design-mismatch" + ], + "related_bugs": [ + "bug-1802" + ], + "occurrences": 1, + "last_seen": "2026-09-11T09:35:00Z" + }, + { + "id": "bug-1804", + "timestamp": "2026-09-11T09:55:00Z", + "error_message": "User design assertion (same live replay session as bug-1799/1800/1803): 'according to the design, when a job has been scheduled by scheduler, we can say job is running in scheduler angle' -- directly challenging this session's own earlier explanation that a job showing DISPATCH-then-still-PEND for multiple sessions was 'by design, not a bug' (deferred-until-confirmation semantics from the scheduler-sim-eager-reconciliation spec). User asked for confirmation and, once confirmed as a genuine design fork (not a bug in the then-current implementation), explicitly directed adopting the OTHER interpretation: revise the spec to match normal-mode's own synchronous dispatch-commit semantics (bug-1624).", + "file": ".kiro/specs/scheduler-sim-eager-reconciliation/{requirements,design,research,tasks,spec.json}, src/scheduler/src/sim_loop.c", + "root_cause": "Not a bug in the sense of an implementation defect -- a genuine, previously-researched design fork between two defensible readings of 'job->state during replay': (a) reconstruct what the ORIGINAL live scheduler process's own job->state field actually held (RUN, synchronously, the instant sched_commit_decision() decided to dispatch -- bug-1624), vs (b) track the SEPARATELY-recorded JOB_STATE_CHANGE/worker-confirmation event's own timing as the operative signal (the scheduler-sim-eager-reconciliation spec's originally-shipped, explicitly-researched choice -- its research.md's own 'Does bug-1624's normal-mode fix already close this gap?' entry correctly noted bug-1624 is a live in-memory write, not a recordable event, so a recording's own JOB_STATE_CHANGE event genuinely CAN land rounds after its DISPATCH decision even post-bug-1624 -- a true fact, but the spec then chose interpretation (b) on top of that fact without it being forced). Live operator use surfaced interpretation (b)'s practical cost directly: a job showing PEND in `info jobs`/`info worker` for multiple sessions after its own `[sim] DISPATCH` line had already printed, reading as a scheduler bug even though it was working exactly as the shipped spec specified.", + "fix": "FIXED (2026-09-11, user-directed). Adopted interpretation (a): `sim_reconcile_unaccounted_jobs()` (sim_loop.c)'s FOUND branch now sets `job->state = JOB_STAT_RUN;` synchronously, in the same operation as the allocation capture (`job->allocs`/`job->accounts`), mirroring `scheduler.c`'s `sched_commit_decision()` (bug-1624) exactly. `sched_account_jobs()`'s own unchanged `JOB_IS_RUN` gate therefore charges the job's resInUse starting the SAME round as capture, not a later confirmation round. A later, separately-recorded `JOB_STATE_CHANGE`/`JOB_UPDATE`-to-RUN event, when it replays, is now a safe idempotent re-affirmation (RUN->RUN, `core_events.c`'s handler reassigns `job->state` unconditionally with no charging side effect) rather than the sole state-transition trigger. Revised the full `scheduler-sim-eager-reconciliation` spec in place (requirements.md Requirement 1's ACs 1.2-1.5 rewritten, Boundary Context's out-of-scope bullet corrected; design.md's Overview/Non-Goals/Boundary Commitments/Architecture flowchart/System Flows sequence diagram/Requirements Traceability/Service Interface doc/Testing Strategy all updated with dated 'Correction (2026-09-11)' notes, not silently rewritten; research.md's own 'Does bug-1624 already close this gap' entry got a superseding revision note explaining the (a) vs (b) trade-off explicitly, since the original factual finding was correct and remains true -- only the DECISION built on top of it changed; tasks.md got a new task 4/4.1/4.2 documenting the revision, both marked done; spec.json's updated_at bumped). TDD: renamed/inverted test_captured_pend_job_excluded_from_resinuse_until_run to test_captured_job_included_in_resinuse_same_round_as_capture (test_sim_reconcile_unaccounted.c) -- asserts job->state==RUN and resInUse charged immediately after capture (no separate job->state write needed), plus a new Requirement-1.4 companion check that a redundant RUN->RUN confirmation event (driven through the REAL XCE_CORE_EVENT_JOB_STATE_CHANGE handler) doesn't touch the allocation and doesn't change what gets charged after a proper core_reset_workers()-then-sched_account_jobs() round boundary (the test's own first draft skipped that reset before its second sched_account_jobs() call, an artificial double-charge that can never happen in real per-round production flow -- caught by RED/GREEN, fixed in the test, not the production code). Also rewrote test_sim_contention_parity.c's entire second test (test_contention_capacity_check_correct_across_multi_round_dispatch_confirm_gap) -- a 300+ line fixture/assertion block built around the OLD 'deferred until confirmation, partial-then-full usage across rounds 1/4/5' narrative -- to the NEW 'both dispatched jobs saturate capacity immediately at round 1; the later confirmation events are safe no-ops' narrative, since the old assertions (capacity free at round 1, only-A-charged at round 4) are now factually backwards. Audited every other file calling/depending on sim_reconcile_unaccounted_jobs() (test_snapshot_save_load.c, test_sim_snapshot_reconcile_gap.c, test_repl_resource_table.c, test_sim_round_boundary_reconciliation.c, test_sim_e2e_original_motivation.c, test_sim_verify.c) and confirmed each is unaffected: their fixtures either put the DISPATCH and JOB_STATE_CHANGE in the SAME round (a no-op change under the new synchronous-capture semantics, since the job was already RUN via the explicit event before reconciliation ever ran) or never rely on job-state/resInUse timing at all (pure --sim-verify decision-content comparisons, or jobs that are never captured by this mechanism in the first place). RED confirmed (temporarily reverted the `job->state = JOB_STAT_RUN;` line, rebuilt: exactly the 2 updated tests failed for the expected reasons, 75 others green). GREEN confirmed after restoring: full scheduler suite 77/77 Ok, zero regressions.", + "tags": [ + "scheduler", + "sim", + "replay", + "sim_reconcile_unaccounted_jobs", + "sched_commit_decision", + "job-state", + "job-lifecycle", + "resource-accounting", + "resInUse", + "design-fork", + "scheduler-sim-eager-reconciliation", + "spec-revision", + "user-directed", + "bug-1624", + "driver-equivalence", + "fixed", + "tdd" + ], + "related_bugs": [ + "bug-1613", + "bug-1611", + "bug-1624", + "bug-1799", + "bug-1801" + ], + "occurrences": 1, + "last_seen": "2026-09-11T09:55:00Z" } ] } diff --git a/.wolf/cerebrum.md b/.wolf/cerebrum.md index b6104919..165fcb6e 100644 --- a/.wolf/cerebrum.md +++ b/.wolf/cerebrum.md @@ -231,3 +231,7 @@ - **(2026-08-26, xce-test-rc-file task 3.1) `xce_test` uses `libedit`'s `readline()`/`isatty(STDIN_FILENO)`-gated `g_interactive`, so a genuine interactive-mode integration test needs a REAL pty, not a piped stdin** — confirmed working via a small embedded Python `pty.fork()` driver (write commands to the master fd with a short settle delay after each, drain with `select()`, detect child exit via non-blocking `waitpid`, `os.waitstatus_to_exitcode()` for the real exit code). Same technique task 2.2's own implementer used for manual verification; this task turned it into a reusable scripted harness. The prompt itself (`fill_prompt`, top-level context) renders as literal `"xce>"` — grepping for that substring in the pty transcript is a good "this really was a TTY session, not a piped fallback" sanity check, distinct from the actual assertion under test. - **A `.kiro/specs/*/tasks.md` checkbox count (`- [ ]` vs `- [x]`) is NOT reliable at face value when it includes section-HEADER checkboxes ("- [ ] N. Some phase name") — a spec whose every N.M subtask is `[x]` but whose parent `- [ ] N.` header was never flipped reads as "in progress" when it is actually 100% done.** (2026-09-03, cross-spec audit) Recurred a second time after the first fix (`5c82ad0b`, 2026-08-24, 5 specs): `scheduler-sim-eager-reconciliation` looked like "6/9, stalled since Aug 22" by a naive checkbox count, but all 3 "unchecked" items were section headers whose children (1.1/1.2/2.1/3.1/3.2/3.3) were already all `[x]`, with a `task 3.3 full regression confirmation` commit proving genuine completion on 2026-08-27. `worker-rejoin-resync` was the same drift the other direction (28/28 real tasks incl. headers all `[x]`, but `spec.json.phase` itself never advanced past `tasks-generated`). **Before reporting any spec as "stalled" or "N/M done," check headers and children SEPARATELY (`grep -E '^\s*-\s*\[ \]\s*[0-9]+\.'` for unchecked headers vs `^\s*-\s*\[ \]\s*[0-9]+\.[0-9]'` for unchecked subtasks) and cross-check `spec.json.phase`/`updated_at` against the tasks.md file's own git log — a low, stale-looking checkbox ratio and a phase field that hasn't moved are both metadata that can independently lag the real state, not just one signal.** - **(2026-09-07, bug-1789) When a per-job "how much of resource X does this job want" value needs to reach a NEW consumer (e.g. a limit-evaluation function) that only sees a *subset* of the data (`ctx->alloc_req.shared`, filled only with genuinely-`XCE_RESF_GLOBAL`-flagged resources), do NOT "fix" it by writing that value into the shared/global request vector too, even if an existing sibling code path two lines above already branches on the same condition and looks like it's establishing the pattern to mirror.** `sched_fill_limit_requests()` (resource.c) needed a GLOBAL limit's underlying `"slots"` resource want, which seqjob/parjob's `prepare_allocation` hooks always staged into `ctx->alloc_req.host` (never `.shared`, since `"slots"` is deliberately per-host-only and excluded from `calc_wanted_global_resvals()`). The natural-looking fix — branch that one unbranched store by `ctx->worker == NULL` to mirror the `calc_wanted_*_resvals()` if/else two lines above it — compiled, passed 2 new targeted regression tests, and then broke 6 *other* previously-green tests (seqjob/parjob dispatch going from DISPATCH to PEND): `ctx->alloc_req.shared` is DUAL-PURPOSE — also read by `sched_check_shared_satisfy()` against `sched->core.sharedInst`, the genuinely-global capacity pool, which has no entry for `"slots"` at all, so injecting a spurious "wants slots" entry there made EVERY job's general resource-satisfy check spuriously fail, not just limit-scoped ones. **General lesson: before writing a new value into a shared/multi-purpose vector to fix ONE consumer, grep every OTHER reader of that same vector and confirm none of them will now see a value they don't know how to interpret — prefer teaching the ONE actually-broken consumer to read the value from its true source (here: `ctx->nslots`, already computed correctly earlier in the same call) over widening what a shared vector carries.** Caught only by re-running the full test suite after the first fix, not by reasoning about the change in isolation — a good reminder that "just mirror the adjacent branch" is not a substitute for checking a shared data structure's OTHER readers. **Follow-up refinement (same bug, user-prompted):** the eventual read-side fix initially special-cased by resource NAME (`strcmp(vres->res->name, "slots")`) — correctly called out as unreliable, since it silently fails to generalize to any future per-host resource not literally spelled "slots". Replaced with the resource's own `XCE_RESF_GLOBAL` flag (the same flag `calc_wanted_global_resvals()`/`calc_wanted_host_resvals()` already dispatch on) to pick `alloc_req->shared` vs `.host` — before hand-rolling a name/string-based special case for "which bucket does this belong in," check whether the codebase already has a flag/enum driving the identical classification elsewhere and reuse it. +- **(2026-09-11, bug-1799, found via live replay Q&A; FIXED as bug-1801 same session) When a per-round "reset then re-derive from scratch" accounting invariant (`core_reset_workers()` zeroes `w->resInUse` every round; `sched_account_jobs()` then re-sums it from every currently-RUN job's `accounts`) gets a conditional EXCLUSION added to only the reset half, the re-derive half must get the identical exclusion too, or the invariant silently breaks for exactly the excluded case.** bug-1751 (2026-09-01) deliberately added `IS_WORKER_RECONCILING(w->status)` to `core_reset_workers()`'s skip-list (`core_session.c:136-142`) so a resyncing worker's in-flight usage isn't wiped mid-reconciliation — correct in isolation. But `sched_account_jobs()` (`scheduler.c:1434`) has no matching check: it still walks every `JOB_IS_RUN` job and unconditionally ADDS its charge via `sched_charge_account()`, with no reset of its own, every round — violating its own documented contract (`core.h:317-318`, "must call this strictly after the session's resval reset"). Net effect: for every round a worker stays RECONCILING (entered via a WORKERS_PUSH resync/reconnect), every RUN job's charge gets re-added ON TOP of the not-reset total instead of a clean resum — `resInUse` grows unboundedly and can exceed the resource's real capacity (observed live: `slots: value=22.00 used=26.00`). **General technique: whenever a "wipe, then fully re-derive" pair of functions exists, and one of them grows a new conditional skip, immediately check whether the OTHER half of the pair has (or needs) the identical skip — a reset-then-resum invariant is only as safe as its weakest half, and grepping for the new condition's flag name in the sibling function is the fast way to check.** Not yet fixed — see bug-1799/`.wolf/buglog.json` for the two candidate fix directions (mirror the RECONCILING check into `sched_account_jobs()`, vs. drop the reset-skip and enforce RECONCILING-safety purely via admission control) and the worker-rejoin-resync spec's own design intent, which needs re-reading before picking one. +- **(2026-09-11, bug-1802/bug-1803) A user asking "how do I skip past N uninteresting things to the next interesting one" wants a ONE-TIME navigation query, not a recurring pause condition — do not model it as a breakpoint (or any other persistent/repeatedly-firing mechanism), even when a breakpoint mechanism already exists and technically CAN express the match condition.** Shipped `break ANY` (matches any event, so `continue` skips dummy/empty sessions) for exactly this ask; it worked for the dummy-sessions case but ALSO kept firing on every matching event once inside a busy, non-dummy session — the user immediately reported it "introduces too many breaks in a single session," since a breakpoint's whole nature is to keep firing until deleted, wrong for a need that's satisfied the first time. Replaced with a one-shot informational display instead (`info summary` now shows "next non-dummy session: N -- run `step M` to reach it", from a read-only forward-scan-and-rewind prescan of the replay file) — no persistent state, no repeated firing, just an answer to "where should I jump to" that the existing `step N` primitive already handles once known. **General lesson: when a feature request has the shape "let me get from A to the next B, skipping everything in between," prefer a query/display that tells the operator where B is (so they use an existing jump/skip primitive) over a new watch/trigger/breakpoint-style mechanism that fires repeatedly — the latter only matches the ask for the FIRST hit and actively works against it afterward.** Also worth remembering: a `sim.c`-owned prescan function must return RELATIVE counts, not absolute ones requiring `xce_sched_repl_state_t`'s full definition — that type is only fully defined in `repl_internal.h` (an incomplete/opaque type in `sim.c` by this project's own Simulation Isolation boundary), caught immediately by a real compile error (`invalid use of incomplete typedef`) on the first attempt to dereference `sim->repl_state->session_count` from within `sim.c`. +- **(2026-09-11, bug-1804) When two different code paths can each independently decide "this job is now running" (a live scheduler's own dispatch decision vs. a separately-recorded worker-confirmation event), and a design has to pick ONE as authoritative for `job->state`, the "right" answer is a genuine design fork, not a factual question with one correct resolution — surface it to the user rather than assuming either reading.** `scheduler-sim-eager-reconciliation`'s own research.md had ALREADY correctly researched this exact question ("does bug-1624 already close this gap?") and correctly concluded bug-1624's live in-memory `job->state = RUN` write is not itself a recordable event, so a recording's separately-recorded `JOB_STATE_CHANGE` genuinely CAN land rounds later — a true fact. But then it built a design on TOP of that fact (defer `job->state` to the confirmation event's own timing) without that choice being forced by the fact itself; the equally-valid alternative (reconstruct what the live scheduler's OWN state field would have shown, i.e. RUN synchronously at dispatch, matching bug-1624 exactly) was never surfaced as a choice. Live use exposed the shipped choice's cost directly (a job showing PEND for several sessions after its own DISPATCH already printed, reading as a bug) — the user then explicitly picked the other interpretation. **When you find a factually-correct research conclusion with a DESIGN DECISION built on top of it that could have gone the other way, don't just implement whichever the existing spec already chose — ask which one the user actually wants, especially when the user's own stated mental model points the other direction.** Also: when revising an `implementation-complete` Kiro spec after a design-decision reversal, update requirements.md/design.md/research.md/tasks.md IN PLACE with dated "Revision"/"Correction" notes (this codebase's own established pattern, e.g. design.md's pre-existing "Correction (2026-09-07, bug-1759/bug-1779)" block) rather than leaving stale prose that no longer matches the code — a future reader trusting the unrevised research.md conclusion as still-current design intent is exactly how this class of drift compounds. +- **(2026-09-11, bug-1804) A test that calls a "re-derive from scratch" function (`sched_account_jobs()`) a SECOND time to prove a later event didn't change anything must reset the state it re-derives FIRST (`core_reset_workers()`), exactly mirroring the real per-round `xce_core_begin_session()`-then-`sched_account_jobs()` sequence — calling it twice back-to-back with no reset in between is a test-methodology artifact (an artificial double-charge, e.g. `resInUse` going 4.0→8.0) that can never happen in real production flow, not a real bug.** Caught immediately by RED/GREEN: the first draft's "redundant confirmation event must not double-charge" assertion failed with `Expected 4 Was 8` purely because the test itself skipped the reset step production always performs — fixed by calling `core_reset_workers()` (white-box-includable from `core_session.c`, already `#include`d in this test file) before the second `sched_account_jobs()` call, not by touching any production code. General pattern: before treating a `sched_account_jobs()`-adjacent test failure as evidence of a real double-charge bug, check whether the test itself reproduces the real reset-then-resum round-boundary sequence, or just calls the resum half twice. diff --git a/.wolf/memory.md b/.wolf/memory.md index 47aa092d..72968de4 100644 --- a/.wolf/memory.md +++ b/.wolf/memory.md @@ -4473,3 +4473,77 @@ | Time | Action | File(s) | Outcome | ~Tokens | |------|--------|---------|---------|--------| + +## Session: 2026-09-11 00:16 + +| Time | Action | File(s) | Outcome | ~Tokens | +|------|--------|---------|---------|--------| +| 00:36 | Edited src/scheduler/testsuite/test_repl_resource_table.c | modified reported() | ~4288 | +| 00:44 | Edited src/scheduler/src/repl_info.c | expanded (+6 lines) | ~249 | +| 00:44 | Edited src/scheduler/src/repl_info.c | added 1 condition(s) | ~543 | +| 00:44 | Edited src/scheduler/src/repl_info.c | expanded (+7 lines) | ~414 | +| 00:44 | Edited src/scheduler/src/repl_info.c | expanded (+13 lines) | ~325 | +| 01:04 | Edited src/scheduler/src/resource.c | added 1 condition(s) | ~696 | +| 01:05 | Edited src/scheduler/testsuite/test_resource_accounting.c | modified test_charge_account_skips_reconciling_or_unavail_worker_host_charge() | ~1031 | +| 01:05 | Edited src/scheduler/testsuite/test_resource_accounting.c | 2→3 lines | ~59 | +| 01:06 | Edited src/scheduler/src/resource.c | 6→3 lines | ~57 | +| 01:08 | Edited src/scheduler/src/repl_internal.h | modified enum() | ~482 | +| 01:08 | Edited src/scheduler/src/repl_break.c | 28→33 lines | ~426 | +| 01:08 | Edited src/scheduler/src/repl_break.c | added 2 condition(s) | ~273 | +| 01:10 | Edited src/scheduler/testsuite/test_repl_breakpoints_inspection.c | modified test_any_breakpoint_skips_dummy_sessions_to_first_real_event() | ~710 | +| 01:10 | Edited src/scheduler/testsuite/test_repl_breakpoints_inspection.c | 1→2 lines | ~38 | +| 01:11 | Edited src/scheduler/testsuite/test_repl_help.c | 6→6 lines | ~80 | +| 01:11 | Edited src/scheduler/testsuite/test_repl_breakpoints_inspection.c | 6→6 lines | ~82 | +| 01:13 | Edited src/scheduler/src/resource.c | added 1 condition(s) | ~68 | +| 01:24 | Edited src/scheduler/testsuite/test_repl_breakpoints_inspection.c | 7→8 lines | ~131 | +| 01:31 | Edited src/scheduler/testsuite/test_repl_breakpoints_inspection.c | expanded (+8 lines) | ~337 | +| 02:15 | Edited src/scheduler/src/repl_internal.h | modified enum() | ~486 | +| 02:16 | Edited src/scheduler/src/repl_break.c | 14→9 lines | ~101 | +| 02:16 | Edited src/scheduler/src/repl_break.c | modified repl_breakpoint_matches_event() | ~118 | +| 02:16 | Edited src/scheduler/testsuite/test_repl_help.c | 6→6 lines | ~75 | +| 02:16 | Edited src/scheduler/testsuite/test_repl_breakpoints_inspection.c | 6→6 lines | ~76 | +| 02:16 | Edited src/scheduler/testsuite/test_repl_breakpoints_inspection.c | removed 66 lines | ~19 | +| 02:17 | Edited src/scheduler/testsuite/test_repl_breakpoints_inspection.c | 2→1 lines | ~18 | +| 02:17 | Edited src/scheduler/src/sim.c | added 9 condition(s) | ~1233 | +| 02:17 | Edited src/scheduler/src/sim.c | 5→6 lines | ~31 | +| 02:18 | Edited src/scheduler/include/sim.h | modified for() | ~234 | +| 02:18 | Edited src/scheduler/src/repl_info.c | added 2 condition(s) | ~458 | +| 02:20 | Edited src/scheduler/testsuite/test_repl_breakpoints_inspection.c | modified requested() | ~986 | +| 02:20 | Edited src/scheduler/testsuite/test_repl_breakpoints_inspection.c | 1→3 lines | ~52 | +| 02:24 | Edited src/scheduler/src/sim.c | modified only() | ~783 | +| 02:24 | Edited src/scheduler/include/sim.h | modified only() | ~300 | +| 02:25 | Edited src/scheduler/src/repl_info.c | modified if() | ~182 | +| 02:33 | Edited src/scheduler/src/repl_info.c | modified if() | ~43 | +| 02:40 | Edited src/scheduler/src/repl_info.c | modified if() | ~40 | +| 03:27 | Edited .kiro/specs/scheduler-sim-eager-reconciliation/requirements.md | 4→5 lines | ~569 | +| 03:28 | Edited .kiro/specs/scheduler-sim-eager-reconciliation/requirements.md | modified Revision() | ~920 | +| 03:29 | Edited .kiro/specs/scheduler-sim-eager-reconciliation/design.md | modified Correction() | ~698 | +| 03:30 | Edited .kiro/specs/scheduler-sim-eager-reconciliation/design.md | 4→4 lines | ~247 | +| 03:31 | Edited .kiro/specs/scheduler-sim-eager-reconciliation/design.md | 18→18 lines | ~255 | +| 03:31 | Edited .kiro/specs/scheduler-sim-eager-reconciliation/design.md | 20→19 lines | ~452 | +| 03:32 | Edited .kiro/specs/scheduler-sim-eager-reconciliation/design.md | 4→5 lines | ~222 | +| 03:32 | Edited .kiro/specs/scheduler-sim-eager-reconciliation/design.md | modified still() | ~269 | +| 03:32 | Edited .kiro/specs/scheduler-sim-eager-reconciliation/design.md | modified New() | ~491 | +| 03:33 | Edited .kiro/specs/scheduler-sim-eager-reconciliation/research.md | 1→6 lines | ~659 | +| 03:35 | Edited .kiro/specs/scheduler-sim-eager-reconciliation/tasks.md | modified Revision() | ~602 | +| 03:35 | Edited .kiro/specs/scheduler-sim-eager-reconciliation/tasks.md | modified Revision() | ~68 | +| 03:35 | Edited .kiro/specs/scheduler-sim-eager-reconciliation/tasks.md | inline fix | ~27 | +| 03:35 | Edited .kiro/specs/scheduler-sim-eager-reconciliation/spec.json | inline fix | ~12 | +| 03:37 | Edited src/scheduler/src/sim_loop.c | modified left() | ~875 | +| 03:38 | Edited src/scheduler/src/sim_loop.c | modified is() | ~516 | +| 03:40 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | modified 12() | ~1510 | +| 03:40 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | inline fix | ~20 | +| 04:12 | Edited src/scheduler/testsuite/test_sim_contention_parity.c | modified below() | ~1320 | +| 04:12 | Edited src/scheduler/testsuite/test_sim_contention_parity.c | modified 1() | ~595 | +| 04:13 | Edited src/scheduler/testsuite/test_sim_contention_parity.c | modified TEST_ASSERT_EQUAL_FLOAT_MESSAGE() | ~665 | +| 04:15 | Edited src/scheduler/src/sim_loop.c | inline fix | ~20 | +| 04:20 | Edited src/scheduler/src/sim_loop.c | inline fix | ~11 | +| 04:27 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | expanded (+11 lines) | ~386 | +| 04:32 | Edited .kiro/specs/scheduler-sim-eager-reconciliation/tasks.md | modified Revision() | ~68 | +| 04:32 | Edited .kiro/specs/scheduler-sim-eager-reconciliation/tasks.md | inline fix | ~27 | +| 04:22 | Session end: 63 writes across 18 files (test_repl_resource_table.c, repl_info.c, resource.c, test_resource_accounting.c, repl_internal.h) | 7 reads | ~28951 tok | + +## Session: 2026-09-12 04:22 + +| Time | Action | File(s) | Outcome | ~Tokens | +|------|--------|---------|---------|--------| diff --git a/issues/simulator-1.md b/issues/simulator-1.md index b5080b51..41eb7d7c 100644 --- a/issues/simulator-1.md +++ b/issues/simulator-1.md @@ -1,62 +1,242 @@ +Following is the scheduler runs in simulation mode: -Simluator REPL output -``` -rshen@pc13v ~/p/x/w/s/src (scheduler-framework)> ./build.dbg/scheduler/xce_scheduler -i -s /tmp/scheduler-replay-20260907-032446.replay -[2026-09-07 11:25:55,060] [32922] [ WARN] [xce_sched_core_load_snapshot] xce_sched_core_load_snapshot failed to resolve job picker plugin (rc=-2); picker rebuild will be skipped for this load -[sim] replay file contains 33 sessions -[sim] loaded: 2 jobs, snapshot @ ts=1788751486917208975 (2026-09-07T03:24:46Z) +rshen@pc13v ~/p/x/w/s/src (scheduler-framework)> ./build.dbg/scheduler/xce_scheduler -i -s /tmp/scheduler-replay-20260911-064117.replay +[2026-09-11 14:49:54,696] [801857] [ WARN] [xce_sched_core_load_snapshot] xce_sched_core_load_snapshot failed to resolve job picker plugin (rc=-2); picker rebuild will be skipped for this load +[sim] replay file contains 420 sessions +[sim] loaded: 0 jobs, snapshot @ ts=1789108877564564549 (2026-09-11T06:41:17Z) [sim] (jobs above reflect this snapshot only -- step/continue to replay later events) [sim] ready. type 'help' for commands. (sim) info summary -[sim] summary: 2 jobs (0 pending, 1 running), 1 workers (1 online), 4 queues, 33 resources -(sim) info jobs -[sim] jobs: - 0-14-1788751409-0 DONE xce - 0-14-1788751435-0 RUN xce -[sim] info jobs: 2 job(s) -(sim) info worker pc13v +[sim] summary: 0 jobs (0 pending, 0 running), 1 workers (1 online), 4 queues, 33 resources +(sim) step +(sim session=1 t=2026-09-11T06:41:17Z) step +(sim session=2 t=2026-09-11T06:41:18Z) step +(sim session=3 t=2026-09-11T06:41:19Z) step +(sim session=4 t=2026-09-11T06:41:20Z) step +(sim session=5 t=2026-09-11T06:41:21Z) step +(sim session=6 t=2026-09-11T06:41:22Z) step +(sim session=7 t=2026-09-11T06:41:23Z) step +[sim] t=1789108884567768744 (2026-09-11T06:41:24Z) session=8 +[sim] event: RESVALS_DELTA n=11 + +- how can we step to the next session which is not dummy(something occurs)? or how can I known what's the next non dummy session? for above I just want to goto session 8 directly + +for session 40: + +(sim session=39 t=2026-09-11T06:41:55Z) step +[sim] t=1789108916579881546 (2026-09-11T06:41:56Z) session=40 +[sim] event: JOB_PUSH njobs=10 jobids=[0-15-1789108916-0[0-9]] +[sim] DISPATCH jobid=0-15-1789108916-0[0] +[sim] DISPATCH jobid=0-15-1789108916-0[1] +[sim] DISPATCH jobid=0-15-1789108916-0[2] +[sim] DISPATCH jobid=0-15-1789108916-0[3] +[sim] DISPATCH jobid=0-15-1789108916-0[4] +[sim] PEND jobid=0-15-1789108916-0[5] +[sim] PEND jobid=0-15-1789108916-0[6] +[sim] PEND jobid=0-15-1789108916-0[7] +[sim] PEND jobid=0-15-1789108916-0[8] +[sim] PEND jobid=0-15-1789108916-0[9] +(sim session=40 t=2026-09-11T06:41:56Z) info worker pc13v [sim] worker pc13v: status=online [sim] -- Consumable/Capacity -- -[sim] resource slots: value=22.00 used=2.00 -[sim] resource mem: value=25749303296.00 used=0.00 -[sim] resource swp: value=16384.00 used=0.00 -[sim] resource tmp: value=541460271104.00 used=0.00 +[sim] resource slots: value=22.00 used=0.00 +[sim] resource mem: value=20684476416.00 used=0.00 +[sim] resource swp: value=13132.17 used=0.00 +[sim] resource tmp: value=576636968960.00 used=0.00 [sim] -- Load Metrics -- -[sim] resource r1m: value=0.16 -[sim] resource r15m: value=1.27 -[sim] resource r15s: value=0.16 -[sim] resource ut: value=1.19 -[sim] resource pg: value=129.07 +[sim] resource r1m: value=2.30 +[sim] resource r15m: value=4.67 +[sim] resource r15s: value=2.30 +[sim] resource ut: value=1.16 +[sim] resource pg: value=263.47 [sim] resource ls: value=1 -[sim] resource it: value=57.83 +[sim] resource it: value=3135.20 [sim] resource io: value=0 [sim] resource cpuf: value=1 [sim] resource maxcpus: value=22 [sim] resource maxmem: value=33182072832.000000 (30.90 GB) -[sim] resource maxswap: value=10000.000000 (9.77 KB) -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^I think maxswap is not 10 KB -[sim] resource maxtmp: value=2147483648.000000 (2.00 GB) +[sim] resource maxswap: value=17179865088.000000 (16.00 GB) +[sim] resource maxtmp: value=1005921779712.000000 (936.84 GB) [sim] resource ndisks: value=1 [sim] -- Static Hardware Attributes -- [sim] resource type: value=x86_64-linux-gnu [sim] resource model: value=x86_64-linux-gnu -[sim] assigned jobs: 0-14-1788751435-0 -(sim) -``` +[sim] assigned jobs: 0-15-1789108916-0[0-4]*4 -xce_test output -``` -xce/client>workerlist -HostName Status HostType HostModel CPUfactor MaxCPU MaxMem MaxSwap MaxTmp NDisks -pc13v OK x86_64-linux-gnu x86_64-linux-gnu 1.000000 22 30.90 GB 1.00 GB 2.00 GB 1 -``` +- The used resources value of slots is still 0.00? as we have dispatched 5 jobs? -Issues: -- resource maxswap value is not correct +Can we improve resource usage value display? -Improvement: -- It will be better to show number of slots allocated in the worker node, so following will be better? -``` -[sim] assigned jobs: 0-14-1788751435-0*2 - ^ indicate 2 slots -``` +(sim session=48 t=2026-09-11T06:42:07Z) info worker pc13v +[sim] worker pc13v: status=online +[sim] -- Consumable/Capacity -- +[sim] resource slots: value=22.00 used=20.00 +[sim] resource mem: value=20684476416.00 used=87960930222080.00 <<>> +[sim] resource swp: value=13132.17 used=0.00 <> +[sim] resource tmp: value=576636968960.00 used=0.00 <> +[sim] -- Load Metrics -- +[sim] resource r1m: value=2.30 +[sim] resource r15m: value=4.67 +[sim] resource r15s: value=2.30 +[sim] resource ut: value=1.16 +[sim] resource pg: value=263.47 +[sim] resource ls: value=1 +[sim] resource it: value=3135.20 +[sim] resource io: value=0 +[sim] resource cpuf: value=1 +[sim] resource maxcpus: value=22 +[sim] resource maxmem: value=33182072832.000000 (30.90 GB) +[sim] resource maxswap: value=17179865088.000000 (16.00 GB) +[sim] resource maxtmp: value=1005921779712.000000 (936.84 GB) +[sim] resource ndisks: value=1 +[sim] -- Static Hardware Attributes -- +[sim] resource type: value=x86_64-linux-gnu +[sim] resource model: value=x86_64-linux-gnu +[sim] assigned jobs: 0-15-1789108916-0[0-4]*4 + +Something wrong in session 98, the count value of slots used is inconsistent with assigned jobs in pc13v? + +[sim] t=1789109023245433276 (2026-09-11T06:43:43Z) session=98 +[sim] event: JOB_UPDATE jobid=0-15-1789108916-0[5] +[sim] event: JOB_UPDATE jobid=0-15-1789108916-0[6] +[sim] event: WORKERS_PUSH nworkers=1 names=[pc13v] +[sim] event: JOB_PUSH njobs=20 jobids=[0-15-1789108916-0[0-9],0-15-1789108940-0[0-9]] +[sim] event: WORKERS_PUSH nworkers=1 names=[pc13v] +[sim] event: JOB_PUSH njobs=20 jobids=[0-15-1789108916-0[0-9],0-15-1789108940-0[0-9]] +[sim] event: JOB_UPDATE jobid=0-15-1789108916-0[9] +[sim] event: JOB_UPDATE jobid=0-15-1789108940-0[0] +[sim] event: JOB_RUSAGE jobid=0-15-1789108940-0[0] +[sim] event: JOB_UPDATE jobid=0-15-1789108940-0[1] +[sim] event: JOB_RUSAGE jobid=0-15-1789108940-0[1] +[sim] event: JOB_STATE_CHANGE jobid=0-15-1789108916-0[5] state=RUN +[sim] event: JOB_STATE_CHANGE jobid=0-15-1789108916-0[6] state=RUN +[sim] event: JOB_STATE_CHANGE jobid=0-15-1789108916-0[7] state=RUN +[sim] event: JOB_STATE_CHANGE jobid=0-15-1789108916-0[8] state=RUN +[sim] event: JOB_STATE_CHANGE jobid=0-15-1789108916-0[9] state=RUN +[2026-09-11 06:43:43,245] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[0] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,245] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[1] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[2] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[3] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[4] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[5] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[6] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[7] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: resync claims RUN for jobid 0-15-1789108916-0[7] with no allocation data -- refusing to merge a phantom RUN (existing state 0x1 kept) +[2026-09-11 06:43:43,246] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[8] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[9] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[0] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[1] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[2] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[3] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[4] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[5] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[6] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[7] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[8] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,247] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[9] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[0] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[1] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[2] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[3] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[4] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[5] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[6] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[7] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: resync claims RUN for jobid 0-15-1789108916-0[7] with no allocation data -- refusing to merge a phantom RUN (existing state 0x1 kept) +[2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[8] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,248] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: resync claims RUN for jobid 0-15-1789108916-0[8] with no allocation data -- refusing to merge a phantom RUN (existing state 0x1 kept) +[2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108916-0[9] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[0] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[1] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[2] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[3] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[4] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[5] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[6] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[7] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[8] (rc=-17), merging state into existing tracked job instead of discarding +[2026-09-11 06:43:43,249] [801857] [ WARN] [ xce_core_push_events] xce_core_push_events JOBS_PUSH: sched_add_job failed for jobid 0-15-1789108940-0[9] (rc=-17), merging state into existing tracked job instead of discarding +[sim] DISPATCH jobid=0-15-1789108940-0[2] +[sim] DISPATCH jobid=0-15-1789108940-0[3] +[sim] DISPATCH jobid=0-15-1789108940-0[4] +[sim] DISPATCH jobid=0-15-1789108940-0[5] +[sim] PEND jobid=0-15-1789108940-0[6] +[sim] PEND jobid=0-15-1789108940-0[7] +[sim] PEND jobid=0-15-1789108940-0[8] +[sim] PEND jobid=0-15-1789108940-0[9] +(sim session=98 t=2026-09-11T06:43:43Z) info worker pc13v +[sim] worker pc13v: status=online +[sim] -- Consumable/Capacity -- +[sim] resource slots: value=22.00 used=22.00 +[sim] resource mem: value=20555202560.00 used=35184372088832.00 +[sim] resource swp: value=13132.59 used=0.00 +[sim] resource tmp: value=576442761216.00 used=0.00 +[sim] -- Load Metrics -- +[sim] resource r1m: value=0.84 +[sim] resource r15m: value=4.24 +[sim] resource r15s: value=0.84 +[sim] resource ut: value=3.52 +[sim] resource pg: value=1752 +[sim] resource ls: value=1 +[sim] resource it: value=3136.95 +[sim] resource io: value=0 +[sim] resource cpuf: value=1 +[sim] resource maxcpus: value=22 +[sim] resource maxmem: value=33182072832.000000 (30.90 GB) +[sim] resource maxswap: value=17179865088.000000 (16.00 GB) +[sim] resource maxtmp: value=1005921779712.000000 (936.84 GB) +[sim] resource ndisks: value=1 +[sim] -- Static Hardware Attributes -- +[sim] resource type: value=x86_64-linux-gnu +[sim] resource model: value=x86_64-linux-gnu +[sim] assigned job: jobid=0-15-1789108916-0[5]*4 +[sim] assigned job: jobid=0-15-1789108916-0[6]*4 +[sim] assigned job: jobid=0-15-1789108916-0[7]*4 +[sim] assigned job: jobid=0-15-1789108916-0[8]*4 +[sim] assigned job: jobid=0-15-1789108916-0[9]*4 +[sim] assigned job: jobid=0-15-1789108940-0[0] +[sim] assigned job: jobid=0-15-1789108940-0[1] +[sim] assigned job: jobid=0-15-1789108940-0[2] +[sim] assigned job: jobid=0-15-1789108940-0[3] +[sim] assigned job: jobid=0-15-1789108940-0[4] +[sim] assigned job: jobid=0-15-1789108940-0[5] +(sim session=98 t=2026-09-11T06:43:54Z) + +- it seems the scheduler has broken max slots constraint? Investigate why? check next session, the used slots get to 26 + +(sim session=99 t=2026-09-11T06:43:54Z) info worker pc13v +[sim] worker pc13v: status=online +[sim] -- Consumable/Capacity -- +[sim] resource slots: value=22.00 used=26.00 +[sim] resource mem: value=20555202560.00 used=35184372088832.00 +[sim] resource swp: value=13132.59 used=0.00 +[sim] resource tmp: value=576442761216.00 used=0.00 +[sim] -- Load Metrics -- +[sim] resource r1m: value=0.84 +[sim] resource r15m: value=4.24 +[sim] resource r15s: value=0.84 +[sim] resource ut: value=3.52 +[sim] resource pg: value=1752 +[sim] resource ls: value=1 +[sim] resource it: value=3136.95 +[sim] resource io: value=0 +[sim] resource cpuf: value=1 +[sim] resource maxcpus: value=22 +[sim] resource maxmem: value=33182072832.000000 (30.90 GB) +[sim] resource maxswap: value=17179865088.000000 (16.00 GB) +[sim] resource maxtmp: value=1005921779712.000000 (936.84 GB) +[sim] resource ndisks: value=1 +[sim] -- Static Hardware Attributes -- +[sim] resource type: value=x86_64-linux-gnu +[sim] resource model: value=x86_64-linux-gnu +[sim] assigned job: jobid=0-15-1789108916-0[5]*4 +[sim] assigned job: jobid=0-15-1789108916-0[6]*4 +[sim] assigned job: jobid=0-15-1789108916-0[7]*4 +[sim] assigned job: jobid=0-15-1789108916-0[8]*4 +[sim] assigned job: jobid=0-15-1789108916-0[9]*4 +[sim] assigned job: jobid=0-15-1789108940-0[0] +[sim] assigned job: jobid=0-15-1789108940-0[1] +[sim] assigned job: jobid=0-15-1789108940-0[2] +[sim] assigned job: jobid=0-15-1789108940-0[3] +[sim] assigned job: jobid=0-15-1789108940-0[4] +[sim] assigned job: jobid=0-15-1789108940-0[5] diff --git a/src/scheduler/include/sim.h b/src/scheduler/include/sim.h index be81c8a7..77f21603 100644 --- a/src/scheduler/include/sim.h +++ b/src/scheduler/include/sim.h @@ -253,4 +253,23 @@ extern int sched_sim_driver_start(xce_sched_sim_driver_t* sim, extern int sched_sim_driver_run(xce_sched_sim_driver_t* sim, const sched_sim_args_t* args); +/* + * sim_prescan_next_nonempty_session - user-requested (2026-09-11): looks + * ahead in sim->replay_file (from its CURRENT read position) for the next + * ReplaySession boundary preceded by at least one real event -- i.e. the + * next "non-dummy" session -- without consuming/advancing the stream. + * `*out_sessions_ahead` is the number of session boundaries from right now + * to reach it (exactly the count `step N` needs), NOT an absolute session + * number -- this function only sees the raw replay stream, not + * xce_sched_repl_state_t's session_count (an opaque type here, fully + * defined only in repl_internal.h). repl_info.c's `info summary` adds this + * to rs->session_count itself for display, and reports the result so an + * operator can `step N` straight there instead of single-stepping through + * a run of dummy (virtual-clock-only) sessions first. See sim.c's own doc + * comment on this function for the full contract, return values, and + * scope decisions. + */ +extern int sim_prescan_next_nonempty_session(xce_sched_sim_driver_t* sim, + uint64_t* out_sessions_ahead); + #endif diff --git a/src/scheduler/src/repl_info.c b/src/scheduler/src/repl_info.c index 9772572f..f6f7114f 100644 --- a/src/scheduler/src/repl_info.c +++ b/src/scheduler/src/repl_info.c @@ -292,14 +292,20 @@ static void repl_info_queue(xce_sched_sim_driver_t* sim, const char* arg) * show -- `info resource`'s GLOBAL/SHARED-only scope (bug-1593) may * genuinely have none of a given category for a given replay. * - * The Consumable/Capacity value=/used= pair deliberately keeps its - * existing unconditional "%.2f" formatting (bug-1601/1613/1727's own - * pinned shapes, test_repl_breakpoints_inspection.c) and is NOT - * units_mode-aware -- Req 6.5's whole-number rule is explicitly scoped to - * non-consumable values only (test_repl_resource_table.c's own file header - * comment), and no bytes-valued CONSUMABLE resource is exercised anywhere - * in this codebase's tests today. Only a Load Metrics (non-consumable - * numeric) value goes through repl_format_numeric_resource_value() below. + * The Consumable/Capacity value=/used= pair keeps its existing + * unconditional "%.2f" formatting for a NON-bytes-valued resource (slots, + * licenses, ...) -- bug-1601/1613/1727's own pinned shapes, + * test_repl_breakpoints_inspection.c -- since Req 6.5's whole-number rule + * is explicitly scoped to non-consumable values only + * (test_repl_resource_table.c's own file header comment). A bytes-valued + * CONSUMABLE resource (mem/tmp/swp -- a live worker's own memory/swap/tmp + * capacity, all flagged CONSUMABLE in practice) DOES go through + * repl_format_numeric_resource_value() for both value= and used=, same as + * every other bytes-valued field in this listing (maxmem/maxswap/maxtmp, + * and mem/tmp/swp themselves when not flagged consumable via the Load + * Metrics path below) -- see + * repl_format_consumable_value_and_used()'s own doc comment for the exact + * split and the user-reported gap this closes (2026-09-11). */ typedef enum { XCE_REPL_RES_SECTION_NONE = 0, @@ -414,6 +420,43 @@ static void repl_format_numeric_resource_value(xce_repl_units_mode_t mode, } } +/* + * repl_format_consumable_value_and_used - formats a CONSUMABLE numeric + * resource's value=/used= pair for repl_info_worker_one()'s per-host case + * and repl_info_resource_one()'s GLOBAL/SHARED case. A byte/megabyte-valued + * resource (repl_is_bytes_valued_resource(), or `swp`'s documented MB + * special case -- the SAME classification repl_format_numeric_resource_value() + * already applies to the Load Metrics section) renders BOTH `avail` and + * `used` through that same formatter, respecting `mode` -- so + * `used=87960930222080.00` (a live-reported bug, 2026-09-11) instead + * becomes e.g. `used=87960930222080.000000 (80.00 TB)` under the default + * "both" mode, matching how maxmem/maxswap/maxtmp already render alongside + * it. Every other consumable resource (slots, licenses, ...) keeps the + * existing pinned "%.2f"/"%.2f" shape unchanged -- bug-1601/1613/1727's own + * tests assert this exact format and must not regress. + */ +static void repl_format_consumable_value_and_used(xce_repl_units_mode_t mode, + const char* rname, + double avail, double used, + char* value_buf, + size_t value_bufsz, + char* used_buf, + size_t used_bufsz) +{ + bool is_swp = (rname != NULL) && (strcmp(rname, "swp") == 0); + + if (repl_is_bytes_valued_resource(rname) || is_swp) { + repl_format_numeric_resource_value(mode, rname, avail, value_buf, + value_bufsz); + repl_format_numeric_resource_value(mode, rname, used, used_buf, + used_bufsz); + return; + } + + snprintf(value_buf, value_bufsz, "%.2f", avail); + snprintf(used_buf, used_bufsz, "%.2f", used); +} + /* repl_current_units_mode - sim->repl_state may legitimately be NULL (a few * white-box tests dispatch `info worker`/`info resource` without ever * calling repl_state_ensure()); default to the same XCE_REPL_UNITS_BOTH @@ -656,18 +699,25 @@ static void repl_info_worker_one(xce_sched_sim_driver_t* sim, * user's exact reported numbers (value=44.00 * used=22.00 on a 22-slot worker at full * occupancy: 22 (never-decremented total) + 22 - * (used) = 44). Print avail_v as-is; NOT - * units_mode-aware (this section's own doc comment - * above explains why). */ + * (used) = 44). Print avail_v as-is. A + * byte-valued resource (mem/tmp/swp) additionally + * renders both fields units_mode-aware, via + * repl_format_consumable_value_and_used() -- see + * that function's own doc comment. */ double avail = avail_v->u.fvalue; double used = ((used_v != NULL) && (used_v->which == Value_u_fvalue)) ? used_v->u.fvalue : 0.0; + char value_str[80]; + char used_str[80]; + repl_format_consumable_value_and_used( + units_mode, rname, avail, used, value_str, + sizeof(value_str), used_str, sizeof(used_str)); fprintf(stdout, - "[sim] resource %s: value=%.2f used=%.2f\n", - rname, avail, used); + "[sim] resource %s: value=%s used=%s\n", + rname, value_str, used_str); } else { char valbuf[80]; @@ -988,9 +1038,22 @@ static void repl_info_resource_one(xce_sched_sim_driver_t* sim, * never decremented as jobs consume it (confirmed against * resource.c's unsatisfy_() admission formula, which reads * it as `total` and compares against `inuse` separately). - * Print it as-is instead of adding `used` on top. */ - fprintf(stdout, "[sim] value=%.2f used=%.2f\n", - v->u.fvalue, used); + * Print it as-is instead of adding `used` on top. A + * byte-valued resource additionally renders both fields + * units_mode-aware, via the same + * repl_format_consumable_value_and_used() helper + * repl_info_worker_one() uses. */ + { + char value_str[80]; + char used_str[80]; + + repl_format_consumable_value_and_used( + repl_current_units_mode(sim), r->name, v->u.fvalue, + used, value_str, sizeof(value_str), used_str, + sizeof(used_str)); + fprintf(stdout, "[sim] value=%s used=%s\n", value_str, + used_str); + } } else { /* Requirement 6.1/6.2/6.5: same Load Metrics rendering * repl_info_worker_one() uses -- units_mode-aware for a @@ -1176,6 +1239,38 @@ static void repl_info_summary(xce_sched_sim_driver_t* sim) "(%d online), %d queues, %d resources\n", n_jobs, n_pending, n_running, n_workers, n_online, n_queues, n_resources); + + /* + * User-requested (2026-09-11, replacing a reverted `break ANY` + * breakpoint kind -- see repl_internal.h's doc comment on + * xce_repl_bp_kind_t): orient the operator toward the next session + * actually worth looking at, instead of leaving them to single-step + * through a run of dummy (virtual-clock-only, zero-event) sessions one + * at a time. sim_prescan_next_nonempty_session() (sim.c) is a pure + * read-only peek -- silently omitted (not an error) whenever it can't + * answer (no replay file, e.g. a bare in-memory test fixture with no + * `.replay` backing; or a genuine I/O error), matching + * sched_sim_driver_start()'s own "purely informational, silently + * skipped on failure" precedent for its sibling prescan. + */ + { + uint64_t sessions_ahead = 0; + int rc2 = sim_prescan_next_nonempty_session(sim, &sessions_ahead); + + if (rc2 == 0) { + xce_sched_repl_state_t* rs = sim->repl_state; + uint64_t current = (rs != NULL) ? rs->session_count : 0; + + fprintf(stdout, + "[sim] next non-dummy session: %llu -- run 'step %llu' " + "to reach it\n", + (unsigned long long)(current + sessions_ahead), + (unsigned long long)sessions_ahead); + } else if (rc2 == 1) { + fprintf(stdout, + "[sim] next non-dummy session: none remaining\n"); + } + } } /* diff --git a/src/scheduler/src/repl_internal.h b/src/scheduler/src/repl_internal.h index 40d484d4..b04e0d88 100644 --- a/src/scheduler/src/repl_internal.h +++ b/src/scheduler/src/repl_internal.h @@ -77,6 +77,17 @@ typedef enum { * xce_core_event_t members -- BEGIN_ROUND is the ReplaySession boundary * itself, DECISION is a drained xce_decision_t -- each gets its own * breakpoint-kind value and its own matching path (repl_break.c). + * + * bug-1802 (2026-09-11): an XCE_REPL_BP_ANY convenience kind ("stop at the + * next session with any event, skipping dummy ones") was added then + * reverted the same session -- live use showed it fires on EVERY matching + * event inside an already-busy session too, forcing many single-event + * `step`s to get back OUT of that session instead of the single hop the + * feature was meant to provide. Replaced with `info summary` reporting the + * next non-dummy session number up front (repl_info.c's + * repl_info_summary(), sim.c's sim_prescan_next_nonempty_session()) so an + * operator can `step N` directly there instead -- no breakpoint needed at + * all. See buglog.json bug-1802/bug-1803 for the full history. */ typedef enum { XCE_REPL_BP_CONFIG_PUSH = 1, diff --git a/src/scheduler/src/resource.c b/src/scheduler/src/resource.c index cbb60525..b360f2ae 100644 --- a/src/scheduler/src/resource.c +++ b/src/scheduler/src/resource.c @@ -605,6 +605,31 @@ int sched_ctx_uncommit_host(xce_sched_ctx_t* ctx, int idx, return xvec_idval(uncomit, committed, XVEC_SUB_FLOAT); } +/* + * bug-1799 fix (2026-09-11): `w->resInUse` is a per-round "reset, then + * fully re-derive from every currently-RUN job" quantity -- core_session.c's + * core_reset_workers() (the reset half) deliberately SKIPS zeroing a + * worker's resInUse while it is IS_WORKER_UNAVAIL/IS_WORKER_RECONCILING + * (bug-1751, worker-rejoin-resync task 5), so its last known-good in-flight + * usage survives until the worker's real state is confirmed. sched_account_ + * jobs() (scheduler.c) is the re-derive half -- called once per round via + * this function (sched_charge_account()) -- and had NO matching skip: it + * unconditionally re-added every RUN job's charge on top of whatever the + * (possibly not-reset) resInUse already held, so a worker that stayed + * RECONCILING/UNAVAIL across several rounds had its resInUse grow by one + * extra full charge PER ROUND instead of being cleanly re-summed -- + * eventually exceeding the resource's real capacity (live-observed: a + * 22-slot worker showing used=26.00). Skipping the per-host charge for + * exactly the same two status bits core_reset_workers() already skips the + * reset for restores the "resum only what reset also touched" symmetry: + * while a worker is UNAVAIL/RECONCILING, its resInUse is simply left + * FROZEN at its last value (neither reset nor re-summed) until the status + * bit clears, at which point normal reset+resum resumes and self-corrects + * on the very next round. Scoped to the per-HOST loop only -- core-> + * sharedInUse's own reset (core_reset_shared_resval(), core_session.c) is + * unconditional (no per-worker gating exists for it), so the shared- + * resource charge two lines below is intentionally left untouched. + */ static int sched_update_account(xce_sched_core_t* core, xce_job_account_t* account, int charge) { @@ -626,6 +651,10 @@ static int sched_update_account(xce_sched_core_t* core, continue; } + if (IS_WORKER_UNAVAIL(w->status) || IS_WORKER_RECONCILING(w->status)) { + continue; + } + sched_update_res_accounts(core, &(w->resInUse), 0, haccount->naccount, haccount->accounts, charge); } diff --git a/src/scheduler/src/sim.c b/src/scheduler/src/sim.c index 9ef56d53..3c8d05d4 100644 --- a/src/scheduler/src/sim.c +++ b/src/scheduler/src/sim.c @@ -1,4 +1,5 @@ #include +#include #include #include #include @@ -408,6 +409,124 @@ static int sim_prescan_session_count(xce_sched_sim_driver_t* sim, return 0; } +/* + * sim_prescan_next_nonempty_session - user-requested (2026-09-11, replacing + * a reverted `break ANY` breakpoint kind -- see repl_internal.h's own doc + * comment on xce_repl_bp_kind_t for why): looks ahead in sim->replay_file + * from the CURRENT read position (exactly what sim_loop.c's main loop will + * read next) for the next ReplaySession boundary that has at least one + * ReplayEvent preceding it in the stream -- a "non-dummy" session, as + * opposed to one that only advances the virtual clock with nothing else + * happening (e.g. a run of sessions that just tick the clock forward with + * no JOB_PUSH/WORKERS_PUSH/etc. in between, a common shape in a real + * captured replay). repl_info.c's repl_info_summary() reports the result so + * an operator can `step N` straight there instead of single-stepping + * through every dummy session first. + * + * Same ftell()/fseek()-around-a-shape-only-scan technique + * sim_prescan_session_count() above establishes -- see that function's own + * doc comment for why this is safe (capn_init_fp() is always unpacked, one + * plain fread() per message, no internal read-ahead buffer). Rewinds the + * file back to the exact starting position before returning in every case + * (found, not found, or error) -- this is a read-only peek, never an + * advance, so the real event loop's own subsequent read is unaffected. + * + * `*out_sessions_ahead` is populated with the number of ReplaySession + * boundaries from RIGHT NOW to (and including) the next non-dummy one -- + * i.e. exactly the count `step N` needs -- only when a non-dummy session is + * actually found before EOF. Deliberately relative, not absolute: this + * function only sees sim->replay_file's raw message stream, not + * sim->repl_state (an opaque, repl_internal.h-only-complete type sim.c + * has no business depending on -- "Simulation isolation" boundary, same + * rationale repl_break.c/repl_info.c's own file headers document for the + * reverse direction). The caller (repl_info.c, which DOES have + * repl_state's full definition) adds this to rs->session_count itself to + * get the absolute "session=N" number for display. + * + * Returns: 0 if a non-dummy session was found (*out_sessions_ahead set, + * always >= 1); 1 if every remaining session before EOF is dummy + * (*out_sessions_ahead untouched); a negative errno on an I/O error + * (ftell/fseek failure). + * + * Scope decisions: (1) like sim_prescan_session_count(), only sees + * sim->replay_file's own stream -- a run under --sim-alt-events substitutes + * a different event stream at run time (sim_loop.c) that this scan does + * not see. (2) if called while paused MID-session (e.g. an event + * breakpoint fired partway through a busy session's own event batch), this + * scan cannot see events already consumed before the current file + * position, so it answers "how many MORE session boundaries from right + * now," not "is the in-progress session itself non-dummy" -- correct and + * exactly what `step N` needs for the overwhelmingly common case (paused + * cleanly at a session boundary between `step`/`continue` calls), a narrow + * gap only for the rarer mid-session-breakpoint case. + */ +int sim_prescan_next_nonempty_session(xce_sched_sim_driver_t* sim, + uint64_t* out_sessions_ahead) +{ + FILE* f; + long resume_at; + uint64_t boundaries_seen = 0; + bool saw_event = false; + + if ((sim == NULL) || (sim->replay_file == NULL) + || (out_sessions_ahead == NULL)) { + return -EINVAL; + } + + f = sim->replay_file; + resume_at = ftell(f); + if (resume_at < 0) { + return -errno; + } + + for (;;) { + struct capn c; + capn_ptr root; + + if (capn_init_fp(&c, f, 0) != 0) { + break; /* end of file (or malformed trailing data) */ + } + + if (msg_root_has_shape(&c, &root, ReplaySession_word_count, + ReplaySession_pointer_count)) { + boundaries_seen++; + capn_free(&c); + + if (saw_event) { + if (fseek(f, resume_at, SEEK_SET) != 0) { + return -errno; + } + *out_sessions_ahead = boundaries_seen; + return 0; + } + continue; + } + + if (msg_root_has_shape(&c, &root, ReplayEvent_word_count, + ReplayEvent_pointer_count)) { + saw_event = true; + capn_free(&c); + continue; + } + + if (msg_root_has_shape(&c, &root, SnapshotEntry_word_count, + SnapshotEntry_pointer_count)) { + capn_free(&c); /* mid-replay checkpoint -- not a real event */ + continue; + } + + /* Unrecognized shape -- end of the message stream. */ + capn_free(&c); + break; + } + + if (fseek(f, resume_at, SEEK_SET) != 0) { + return -errno; + } + + return 1; +} + int sched_sim_driver_start(xce_sched_sim_driver_t* sim, const sched_sim_args_t* args) { diff --git a/src/scheduler/src/sim_loop.c b/src/scheduler/src/sim_loop.c index 7f47837a..13874d76 100644 --- a/src/scheduler/src/sim_loop.c +++ b/src/scheduler/src/sim_loop.c @@ -594,15 +594,26 @@ static void* sim_picker_thread_fn(void* arg) * double-application guard. Any job in `sim->core.jobs` with `job->accounts * == NULL` is eligible to be searched for a matching recorded DISPATCH * decision this round, regardless of whether it is PEND or RUN -- a job's - * recorded DISPATCH decision and its RUN-state confirmation can legitimately - * land many rounds apart in the original recording (the dominant real-world - * case), and this function is called unconditionally every round, so a - * not-yet-running job is simply re-checked round after round until its - * originating decision is found (or the replay ends). Actual resource - * charging remains fully deferred to sched_account_jobs()'s own unchanged - * `JOB_IS_RUN` gate (scheduler.c) -- capturing a job's allocation here while - * it is not yet RUN is inert until that gate later includes it; this - * function performs no charging itself. Requirement 3.3's "report only the + * recorded DISPATCH decision and its separately-recorded RUN-state + * confirmation event can legitimately land many rounds apart in the + * original recording, and this function is called unconditionally every + * round, so a not-yet-captured job is simply re-checked round after round + * until its originating decision is found (or the replay ends). + * + * job->state transition (REVISED 2026-09-11, scheduler-sim-eager- + * reconciliation Requirement 1.2/1.3, user-directed): a successful FOUND + * match ALSO sets job->state = JOB_STAT_RUN synchronously, in the same + * operation as the allocation capture -- mirroring scheduler.c's + * sched_commit_decision() (bug-1624) exactly, which treats a job as + * running from the scheduler's own perspective the instant it decides to + * dispatch, independent of any later, separately-recorded worker- + * confirmation signal. sched_account_jobs()'s own unchanged `JOB_IS_RUN` + * gate (scheduler.c) therefore includes this job's charge starting THIS + * round, not a later one when its confirmation event happens to replay. + * (Previously this function left job->state untouched here, deferring the + * RUN transition entirely to a later JOB_STATE_CHANGE/JOB_UPDATE replay -- + * see requirements.md's Requirement 1 2026-09-11 revision for the full + * rationale trail.) Requirement 3.3's "report only the * fresh one" guarantee needs no separate mechanism here: a job that * independently receives a fresh live decision in this same round already * has job->accounts set by sched_commit_decision() (scheduler.c, unchanged) @@ -774,6 +785,32 @@ static void sim_reconcile_unaccounted_jobs(xce_sched_sim_driver_t* sim, entry->body.dispatch->alloc = NULL; job->accounts = make_job_account(job->allocs); + /* User-directed revision (2026-09-11, scheduler-sim-eager- + * reconciliation requirements.md Requirement 1.2/1.3): set + * job->state = JOB_STAT_RUN synchronously, in the SAME + * operation as the allocation capture above -- mirroring + * scheduler.c's sched_commit_decision() (bug-1624) exactly, + * which treats a job as running from the scheduler's own + * perspective the instant it decides to dispatch, independent + * of whatever later, separately-recorded signal confirms the + * worker actually started it. Previously this function left + * job->state untouched here, deferring the RUN transition + * entirely to a later-replayed JOB_STATE_CHANGE/JOB_UPDATE + * event -- live use showed this as a confusing multi-session + * "DISPATCH already printed, but info jobs still shows PEND" + * gap that does not match what the original live scheduler + * process's own job->state field actually held at this exact + * point. sched_account_jobs()'s own unchanged JOB_IS_RUN gate + * (scheduler.c) now includes this job's charge starting THIS + * round, not a later one -- no change needed there. A later + * replay of the job's own separately-recorded confirmation + * event still applies via core_events.c's normal + * XCE_CORE_EVENT_JOB_STATE_CHANGE path; that handler's + * unconditional `job->state = new_state;` reassignment + * (RUN -> RUN) is a verified-safe no-op, not a double-charge + * or a terminal-guard violation. */ + job->state = JOB_STAT_RUN; + /* bug-1759/bug-1779: mirrors sched_allocator()'s own ordering * (scheduler.c) as closely as this reconstruction allows -- * remove the job from the picker's candidate set now that it is diff --git a/src/scheduler/testsuite/test_repl_breakpoints_inspection.c b/src/scheduler/testsuite/test_repl_breakpoints_inspection.c index 327305f5..4e4e758f 100644 --- a/src/scheduler/testsuite/test_repl_breakpoints_inspection.c +++ b/src/scheduler/testsuite/test_repl_breakpoints_inspection.c @@ -1902,6 +1902,99 @@ static void test_event_breakpoint_pauses_repl_at_matching_event(void) printf("test_event_breakpoint_pauses_repl_at_matching_event: PASS\n"); } +/* + * test_info_summary_reports_next_nondummy_session_when_present - + * user-requested (2026-09-11, replacing a reverted `break ANY` breakpoint + * kind -- see repl_internal.h's own doc comment on xce_repl_bp_kind_t): + * `info summary`, called at the very start (before any `step`/`continue`, + * i.e. session_count==0), must report the next session that decodes at + * least one real event, skipping two fully-empty dummy sessions (rounds + * 1000 and 2000), plus the exact `step` count to reach it. + */ +static void +test_info_summary_reports_next_nondummy_session_when_present(void) +{ + char replay_path[PATH_MAX]; + char aux_dir[PATH_MAX]; + char script_path[PATH_MAX]; + char capture_path[PATH_MAX]; + FILE* f; + char* captured; + + f = write_header_and_snapshot(replay_path, sizeof(replay_path), + "_nextnd"); + TEST_ASSERT_NOT_NULL(f); + + append_replay_session(f, 1000ULL); /* session 1: dummy */ + append_replay_session(f, 2000ULL); /* session 2: dummy */ + append_jobstatechange_event(f, 2500ULL, 444ULL); /* session 3's event */ + append_replay_session(f, 3000ULL); /* session 3: real */ + append_end_of_stream_marker(f); + fclose(f); + + make_tmp_dir(aux_dir, sizeof(aux_dir), "_nextndaux"); + snprintf(script_path, sizeof(script_path), "%s/script.txt", aux_dir); + snprintf(capture_path, sizeof(capture_path), "%s/capture.out", aux_dir); + write_text_file(script_path, "info summary\nquit\n"); + + captured = run_group_d_and_capture(replay_path, script_path, capture_path); + + TEST_ASSERT_NOT_NULL_MESSAGE( + strstr(captured, "next non-dummy session: 3 -- run 'step 3' to " + "reach it"), + "info summary must report session 3 (skipping the two dummy " + "sessions 1 and 2) and the exact step count (3) to reach it from " + "the very start (session_count==0)"); + + free(captured); + + printf("test_info_summary_reports_next_nondummy_session_when_present: " + "PASS\n"); +} + +/* + * test_info_summary_reports_none_remaining_when_all_dummy - the file has + * exactly two sessions, BOTH dummy, and nothing after them -- `info + * summary` must say so plainly rather than reporting a bogus session + * number or silently omitting the line. + */ +static void +test_info_summary_reports_none_remaining_when_all_dummy(void) +{ + char replay_path[PATH_MAX]; + char aux_dir[PATH_MAX]; + char script_path[PATH_MAX]; + char capture_path[PATH_MAX]; + FILE* f; + char* captured; + + f = write_header_and_snapshot(replay_path, sizeof(replay_path), + "_nonend"); + TEST_ASSERT_NOT_NULL(f); + + append_replay_session(f, 1000ULL); /* session 1: dummy */ + append_replay_session(f, 2000ULL); /* session 2: dummy */ + append_end_of_stream_marker(f); + fclose(f); + + make_tmp_dir(aux_dir, sizeof(aux_dir), "_nonendaux"); + snprintf(script_path, sizeof(script_path), "%s/script.txt", aux_dir); + snprintf(capture_path, sizeof(capture_path), "%s/capture.out", aux_dir); + write_text_file(script_path, "info summary\nquit\n"); + + captured = run_group_d_and_capture(replay_path, script_path, capture_path); + + TEST_ASSERT_NOT_NULL_MESSAGE( + strstr(captured, "next non-dummy session: none remaining"), + "info summary must plainly report that no non-dummy session " + "remains, not a bogus number or silence"); + + free(captured); + + printf("test_info_summary_reports_none_remaining_when_all_dummy: " + "PASS\n"); +} + static void test_decision_breakpoint_pauses_after_decision_output(void) { char replay_path[PATH_MAX]; @@ -2149,6 +2242,8 @@ int main(void) RUN_TEST(test_info_resource_all_reports_when_none_are_global_shared); RUN_TEST(test_info_job_accepts_quoted_jobid_argument); RUN_TEST(test_info_summary_reports_counts); + RUN_TEST(test_info_summary_reports_next_nondummy_session_when_present); + RUN_TEST(test_info_summary_reports_none_remaining_when_all_dummy); RUN_TEST(test_info_session_reports_position_after_synthetic_events); RUN_TEST(test_event_breakpoint_pauses_repl_at_matching_event); diff --git a/src/scheduler/testsuite/test_repl_resource_table.c b/src/scheduler/testsuite/test_repl_resource_table.c index a9438453..ba0c2902 100644 --- a/src/scheduler/testsuite/test_repl_resource_table.c +++ b/src/scheduler/testsuite/test_repl_resource_table.c @@ -1714,6 +1714,404 @@ test_info_resource_units_mode_both_renders_raw_and_human_for_bytes_resource( "bytes_resource: PASS\n"); } +/* + * User-reported (2026-09-11, live replay session): a CONSUMABLE byte-valued + * per-host resource (mem/tmp/swp) rendered its value=/used= pair as a raw + * `%.2f` byte count (e.g. "used=87960930222080.00"), unlike every other + * bytes-valued field in this same listing (maxmem/maxswap/maxtmp, and mem/ + * tmp/swp themselves when NOT flagged consumable, via the Load Metrics + * path) which already render human-scaled. repl_info_worker_one()'s own + * pre-fix header comment explicitly scoped this out ("no bytes-valued + * CONSUMABLE resource is exercised anywhere in this codebase's tests + * today") -- this test (and its "raw"/swp/info-resource siblings below) + * closes that gap. A non-bytes-valued consumable (slots/licenses) must keep + * its existing pinned "%.2f"/"%.2f" shape unchanged -- see + * test_info_worker_reports_consumable_value_and_used (bug-1727, + * test_repl_breakpoints_inspection.c), left untouched by this fix. + */ +static void +test_info_worker_units_mode_both_renders_raw_and_human_for_consumable_bytes_resource( + void) +{ + xce_sched_sim_driver_t sim; + xce_core_event_t events[1]; + xce_resource_t wire_resource; + xce_resource_t* r; + xce_worker_t wire_worker; + xce_worker_t* worker_ptrs[1]; + xce_val_t val; + xce_name_val_t nv; + xce_name_val_t* nv_ptrs[1]; + xce_val_t used_val; + xce_sched_worker_t* sw; + char* captured; + char expected_value[64]; + char expected_used[64]; + char expected_line[160]; + double avail; + double used; + int rc; + + avail = 8589934592.0; /* 8 GB */ + used = 4294967296.0; /* 4 GB */ + + memset(&sim, 0, sizeof(sim)); + rc = xce_sched_core_init(&(sim.core)); + TEST_ASSERT_EQUAL_INT(0, rc); + + memset(&wire_resource, 0, sizeof(wire_resource)); + wire_resource.name = "mem"; + wire_resource.valtype = XCE_VAL_NUMERIC; + wire_resource.flags = XCE_RESF_CONSUMABLE; + r = makeResource(sim.core.resmgr, &wire_resource); + TEST_ASSERT_NOT_NULL(r); + + memset(&val, 0, sizeof(val)); + val.which = Value_u_fvalue; + val.u.fvalue = avail; + memset(&nv, 0, sizeof(nv)); + nv.name = "mem"; + nv.val = &val; + nv_ptrs[0] = &nv; + + memset(&wire_worker, 0, sizeof(wire_worker)); + wire_worker.name = "hostA"; + wire_worker.status = 0; + wire_worker.nresvals_to_sched = 1; + wire_worker.resvals_to_sched = nv_ptrs; + worker_ptrs[0] = &wire_worker; + + memset(events, 0, sizeof(events)); + events[0].type = XCE_CORE_EVENT_WORKERS_PUSH; + events[0].u.workers_push.nworkers = 1; + events[0].u.workers_push.workers = worker_ptrs; + + rc = xce_core_push_events(&(sim.core), events, 1); + TEST_ASSERT_EQUAL_INT(0, rc); + + sw = sched_get_worker(&(sim.core), "hostA"); + TEST_ASSERT_NOT_NULL(sw); + + memset(&used_val, 0, sizeof(used_val)); + used_val.which = Value_u_fvalue; + used_val.u.fvalue = used; + zarray_put(&(sw->resInUse), r->idx, dup_val(&used_val)); + + sim.repl_state = repl_state_ensure(&sim); + TEST_ASSERT_NOT_NULL(sim.repl_state); + sim.repl_state->units_mode = XCE_REPL_UNITS_BOTH; + + captured = capture_dispatch(&sim, "info worker hostA"); + + repl_format_bytes_human(avail, expected_value, sizeof(expected_value)); + repl_format_bytes_human(used, expected_used, sizeof(expected_used)); + snprintf(expected_line, sizeof(expected_line), "value=%f (%s) used=%f (%s)", + avail, expected_value, used, expected_used); + + TEST_ASSERT_NOT_NULL_MESSAGE( + strstr(captured, expected_line), + "a CONSUMABLE byte-valued resource (mem/tmp/swp) must render both " + "value= and used= human-scaled under units_mode=both, the same " + "as every other bytes-valued field in this listing"); + + free(captured); + + printf( + "test_info_worker_units_mode_both_renders_raw_and_human_for_" + "consumable_bytes_resource: PASS\n"); +} + +/* + * AC 6.2 'raw' counterpart for a CONSUMABLE byte-valued resource: under + * units_mode=raw, value=/used= must show ONLY the bare wire numbers, no + * "GB"/human suffix -- mirrors + * test_info_worker_units_mode_raw_renders_bare_wire_value_without_mutating_stored_value's + * contract, applied to the consumable value=/used= pair instead of a + * single value= line. + */ +static void +test_info_worker_units_mode_raw_renders_bare_values_for_consumable_bytes_resource( + void) +{ + xce_sched_sim_driver_t sim; + xce_core_event_t events[1]; + xce_resource_t wire_resource; + xce_resource_t* r; + xce_worker_t wire_worker; + xce_worker_t* worker_ptrs[1]; + xce_val_t val; + xce_name_val_t nv; + xce_name_val_t* nv_ptrs[1]; + xce_val_t used_val; + xce_sched_worker_t* sw; + char* captured; + char expected_line[160]; + double avail; + double used; + int rc; + + avail = 8589934592.0; + used = 4294967296.0; + + memset(&sim, 0, sizeof(sim)); + rc = xce_sched_core_init(&(sim.core)); + TEST_ASSERT_EQUAL_INT(0, rc); + + memset(&wire_resource, 0, sizeof(wire_resource)); + wire_resource.name = "mem"; + wire_resource.valtype = XCE_VAL_NUMERIC; + wire_resource.flags = XCE_RESF_CONSUMABLE; + r = makeResource(sim.core.resmgr, &wire_resource); + TEST_ASSERT_NOT_NULL(r); + + memset(&val, 0, sizeof(val)); + val.which = Value_u_fvalue; + val.u.fvalue = avail; + memset(&nv, 0, sizeof(nv)); + nv.name = "mem"; + nv.val = &val; + nv_ptrs[0] = &nv; + + memset(&wire_worker, 0, sizeof(wire_worker)); + wire_worker.name = "hostA"; + wire_worker.status = 0; + wire_worker.nresvals_to_sched = 1; + wire_worker.resvals_to_sched = nv_ptrs; + worker_ptrs[0] = &wire_worker; + + memset(events, 0, sizeof(events)); + events[0].type = XCE_CORE_EVENT_WORKERS_PUSH; + events[0].u.workers_push.nworkers = 1; + events[0].u.workers_push.workers = worker_ptrs; + + rc = xce_core_push_events(&(sim.core), events, 1); + TEST_ASSERT_EQUAL_INT(0, rc); + + sw = sched_get_worker(&(sim.core), "hostA"); + TEST_ASSERT_NOT_NULL(sw); + + memset(&used_val, 0, sizeof(used_val)); + used_val.which = Value_u_fvalue; + used_val.u.fvalue = used; + zarray_put(&(sw->resInUse), r->idx, dup_val(&used_val)); + + sim.repl_state = repl_state_ensure(&sim); + TEST_ASSERT_NOT_NULL(sim.repl_state); + sim.repl_state->units_mode = XCE_REPL_UNITS_RAW; + + captured = capture_dispatch(&sim, "info worker hostA"); + + snprintf(expected_line, sizeof(expected_line), "value=%f used=%f", avail, + used); + + TEST_ASSERT_NOT_NULL_MESSAGE( + strstr(captured, expected_line), + "AC 6.2 'raw': a CONSUMABLE byte-valued resource must render bare " + "wire values for both value= and used= under units_mode=raw"); + TEST_ASSERT_NULL_MESSAGE( + strstr(captured, "GB"), + "AC 6.2 'raw': must show no human-readable unit suffix"); + + free(captured); + + printf( + "test_info_worker_units_mode_raw_renders_bare_values_for_" + "consumable_bytes_resource: PASS\n"); +} + +/* + * AC 6.1's swp-is-megabytes special case, applied to the CONSUMABLE + * value=/used= pair: both fields must be converted MB->bytes before + * human-scaling, mirroring + * test_info_worker_units_mode_converts_swp_megabytes_before_human_scaling's + * contract for the Load Metrics (non-consumable) path. + */ +static void +test_info_worker_units_mode_converts_swp_megabytes_for_consumable_resource( + void) +{ + xce_sched_sim_driver_t sim; + xce_core_event_t events[1]; + xce_resource_t wire_resource; + xce_resource_t* r; + xce_worker_t wire_worker; + xce_worker_t* worker_ptrs[1]; + xce_val_t val; + xce_name_val_t nv; + xce_name_val_t* nv_ptrs[1]; + xce_val_t used_val; + xce_sched_worker_t* sw; + char* captured; + char expected_value_human[64]; + char expected_used_human[64]; + double avail_mb; + double used_mb; + int rc; + + avail_mb = 16384.0; /* 16384 MB -> 16 GB */ + used_mb = 4096.0; /* 4096 MB -> 4 GB */ + + memset(&sim, 0, sizeof(sim)); + rc = xce_sched_core_init(&(sim.core)); + TEST_ASSERT_EQUAL_INT(0, rc); + + memset(&wire_resource, 0, sizeof(wire_resource)); + wire_resource.name = "swp"; + wire_resource.valtype = XCE_VAL_NUMERIC; + wire_resource.flags = XCE_RESF_CONSUMABLE; + r = makeResource(sim.core.resmgr, &wire_resource); + TEST_ASSERT_NOT_NULL(r); + + memset(&val, 0, sizeof(val)); + val.which = Value_u_fvalue; + val.u.fvalue = avail_mb; + memset(&nv, 0, sizeof(nv)); + nv.name = "swp"; + nv.val = &val; + nv_ptrs[0] = &nv; + + memset(&wire_worker, 0, sizeof(wire_worker)); + wire_worker.name = "hostA"; + wire_worker.status = 0; + wire_worker.nresvals_to_sched = 1; + wire_worker.resvals_to_sched = nv_ptrs; + worker_ptrs[0] = &wire_worker; + + memset(events, 0, sizeof(events)); + events[0].type = XCE_CORE_EVENT_WORKERS_PUSH; + events[0].u.workers_push.nworkers = 1; + events[0].u.workers_push.workers = worker_ptrs; + + rc = xce_core_push_events(&(sim.core), events, 1); + TEST_ASSERT_EQUAL_INT(0, rc); + + sw = sched_get_worker(&(sim.core), "hostA"); + TEST_ASSERT_NOT_NULL(sw); + + memset(&used_val, 0, sizeof(used_val)); + used_val.which = Value_u_fvalue; + used_val.u.fvalue = used_mb; + zarray_put(&(sw->resInUse), r->idx, dup_val(&used_val)); + + sim.repl_state = repl_state_ensure(&sim); + TEST_ASSERT_NOT_NULL(sim.repl_state); + sim.repl_state->units_mode = XCE_REPL_UNITS_HUMAN; + + captured = capture_dispatch(&sim, "info worker hostA"); + + repl_format_bytes_human(avail_mb * 1024.0 * 1024.0, expected_value_human, + sizeof(expected_value_human)); + repl_format_bytes_human(used_mb * 1024.0 * 1024.0, expected_used_human, + sizeof(expected_used_human)); + + TEST_ASSERT_NOT_NULL_MESSAGE( + strstr(captured, expected_value_human), + "swp's value= must be converted MB->bytes before human-scaling, " + "same as the Load Metrics path"); + TEST_ASSERT_NOT_NULL_MESSAGE( + strstr(captured, expected_used_human), + "swp's used= must be converted MB->bytes before human-scaling too"); + + free(captured); + + printf( + "test_info_worker_units_mode_converts_swp_megabytes_for_" + "consumable_resource: PASS\n"); +} + +/* + * info resource's counterpart: a GLOBAL/SHARED CONSUMABLE byte-valued + * resource must render its value=/used= pair human-scaled too, mirroring + * test_info_resource_units_mode_both_renders_raw_and_human_for_bytes_resource + * (the non-consumable case) and this file's own info-worker tests above. + */ +static void +test_info_resource_units_mode_both_renders_raw_and_human_for_consumable_bytes_resource( + void) +{ + xce_sched_sim_driver_t sim; + xce_core_event_t events[1]; + xce_resource_t wire_mem; + xce_resource_t* r; + xce_val_t val_mem; + xce_resval_t rv_mem; + xce_resval_t* rv_ptrs[1]; + xce_val_t* stored; + xce_val_t used_val; + char* captured; + char expected_value[64]; + char expected_used[64]; + char expected_line[160]; + double avail; + double used; + int rc; + + avail = 8589934592.0; /* 8 GB */ + used = 2147483648.0; /* 2 GB */ + + memset(&sim, 0, sizeof(sim)); + rc = xce_sched_core_init(&(sim.core)); + TEST_ASSERT_EQUAL_INT(0, rc); + + memset(&wire_mem, 0, sizeof(wire_mem)); + wire_mem.name = "mem"; + wire_mem.valtype = XCE_VAL_NUMERIC; + wire_mem.flags = XCE_RESF_SHARED | XCE_RESF_CONSUMABLE; + r = makeResource(sim.core.resmgr, &wire_mem); + TEST_ASSERT_NOT_NULL(r); + + memset(&val_mem, 0, sizeof(val_mem)); + val_mem.which = Value_u_fvalue; + val_mem.u.fvalue = avail; + memset(&rv_mem, 0, sizeof(rv_mem)); + rv_mem.name = "mem"; + rv_mem.val = &val_mem; + rv_ptrs[0] = &rv_mem; + + memset(events, 0, sizeof(events)); + events[0].type = XCE_CORE_EVENT_RESVALS_FULL; + events[0].u.resvals_full.n = 1; + events[0].u.resvals_full.resvals = rv_ptrs; + + rc = xce_core_push_events(&(sim.core), events, 1); + TEST_ASSERT_EQUAL_INT(0, rc); + + memset(&used_val, 0, sizeof(used_val)); + used_val.which = Value_u_fvalue; + used_val.u.fvalue = used; + zarray_put(&(sim.core.sharedInUse), r->idx, dup_val(&used_val)); + + sim.repl_state = repl_state_ensure(&sim); + TEST_ASSERT_NOT_NULL(sim.repl_state); + sim.repl_state->units_mode = XCE_REPL_UNITS_BOTH; + + captured = capture_dispatch(&sim, "info resource mem"); + + repl_format_bytes_human(avail, expected_value, sizeof(expected_value)); + repl_format_bytes_human(used, expected_used, sizeof(expected_used)); + snprintf(expected_line, sizeof(expected_line), "value=%f (%s) used=%f (%s)", + avail, expected_value, used, expected_used); + + TEST_ASSERT_NOT_NULL_MESSAGE( + strstr(captured, expected_line), + "a GLOBAL/SHARED CONSUMABLE byte-valued resource must render both " + "value= and used= human-scaled under units_mode=both"); + + stored = (xce_val_t*)zarray_get(&(sim.core.sharedInst), r->idx); + TEST_ASSERT_NOT_NULL(stored); + TEST_ASSERT_EQUAL_INT(Value_u_fvalue, stored->which); + TEST_ASSERT_EQUAL_FLOAT_MESSAGE( + (float)avail, (float)stored->u.fvalue, + "AC 6.4: displaying a resource under any units_mode must never " + "mutate the underlying stored/wire value"); + + free(captured); + + printf( + "test_info_resource_units_mode_both_renders_raw_and_human_for_" + "consumable_bytes_resource: PASS\n"); +} + int main(void) { UNITY_BEGIN(); @@ -1724,6 +2122,9 @@ int main(void) RUN_TEST(test_info_worker_units_mode_human_renders_scaled_value_without_mutating_stored_value); RUN_TEST(test_info_worker_units_mode_both_renders_raw_and_human_without_mutating_stored_value); RUN_TEST(test_info_worker_units_mode_converts_swp_megabytes_before_human_scaling); + RUN_TEST(test_info_worker_units_mode_both_renders_raw_and_human_for_consumable_bytes_resource); + RUN_TEST(test_info_worker_units_mode_raw_renders_bare_values_for_consumable_bytes_resource); + RUN_TEST(test_info_worker_units_mode_converts_swp_megabytes_for_consumable_resource); RUN_TEST(test_info_worker_assigned_job_listing_compresses_contiguous_array_indices); RUN_TEST(test_info_worker_assigned_job_listing_appends_slot_count_when_uniform); RUN_TEST(test_info_worker_assigned_job_listing_excludes_terminal_job_despite_stale_allocs); @@ -1731,6 +2132,7 @@ int main(void) RUN_TEST(test_info_resource_groups_resources_into_labeled_table_sections_in_order); RUN_TEST(test_info_resource_load_metric_whole_number_omits_trailing_zeros); RUN_TEST(test_info_resource_units_mode_both_renders_raw_and_human_for_bytes_resource); + RUN_TEST(test_info_resource_units_mode_both_renders_raw_and_human_for_consumable_bytes_resource); return UNITY_END(); } diff --git a/src/scheduler/testsuite/test_resource_accounting.c b/src/scheduler/testsuite/test_resource_accounting.c index edae4385..4ca749cc 100644 --- a/src/scheduler/testsuite/test_resource_accounting.c +++ b/src/scheduler/testsuite/test_resource_accounting.c @@ -369,6 +369,86 @@ static void test_commit_job_alloc_identical_across_core_containers(void) TEST_ASSERT_EQUAL_FLOAT(shared_sched[0], shared_sim[0]); } +/* + * test_charge_account_skips_reconciling_or_unavail_worker_host_charge - + * bug-1799 fix: sched_charge_account()'s per-host resInUse charge must be + * skipped for a worker currently IS_WORKER_RECONCILING or IS_WORKER_UNAVAIL. + * core_reset_workers() (core_session.c, bug-1751) already skips RESETTING + * such a worker's resInUse every round; sched_account_jobs()'s own + * per-round re-derive pass reaches resInUse only through this same + * sched_charge_account()->sched_update_account() path, so without a + * matching skip here it re-added a full charge ON TOP of the not-reset + * total every round the worker stayed RECONCILING/UNAVAIL -- resInUse grew + * unboundedly and could exceed the resource's real capacity (live-observed: + * a 22-slot worker showing used=26.00). core->sharedInUse has no + * per-worker gating at all (its own reset, core_reset_shared_resval(), is + * unconditional) and must keep charging normally regardless of any one + * host's status. + */ +static void +test_charge_account_skips_reconciling_or_unavail_worker_host_charge(void) +{ + xce_scheduler_t sched; + xce_job_account_t* acct; + xce_sched_worker_t* w; + float shared_val[1]; + float host_val[1]; + int shared_idx[1] = { SHARED_RESIDX }; + int host_idx[1] = { HOST_RESIDX }; + int rc; + + memset(&sched, 0, sizeof(sched)); + setup_core_with_worker(&(sched.core), WORKER_NAME); + + w = sched_get_worker(&(sched.core), WORKER_NAME); + TEST_ASSERT_NOT_NULL(w); + + acct = make_test_account(WORKER_NAME, SHARED_RESIDX, SHARED_USED, + SHARED_RESERVED, HOST_RESIDX, HOST_USED, + HOST_RESERVED); + + /* --- RECONCILING: shared charges normally, per-host charge skipped --- */ + w->status = XCE_WORKER_RECONCILING; + rc = sched_charge_account(&(sched.core), acct); + TEST_ASSERT_EQUAL_INT(0, rc); + + get_resvals(&(sched.core.sharedInUse), 1, shared_idx, shared_val); + TEST_ASSERT_EQUAL_FLOAT_MESSAGE( + EXPECTED_SHARED_CHARGE, shared_val[0], + "shared resources have no per-worker gating and must still charge " + "while the host is RECONCILING"); + + get_resvals(&(w->resInUse), 1, host_idx, host_val); + TEST_ASSERT_EQUAL_FLOAT_MESSAGE( + 0.0f, host_val[0], + "bug-1799: a RECONCILING worker's per-host resInUse must NOT be " + "charged -- core_reset_workers() already skips resetting it, so " + "charging on top would grow it every round it stays RECONCILING"); + + /* --- UNAVAIL: same per-host skip --- */ + w->status = XCE_WORKER_UNAVAIL; + rc = sched_charge_account(&(sched.core), acct); + TEST_ASSERT_EQUAL_INT(0, rc); + + get_resvals(&(w->resInUse), 1, host_idx, host_val); + TEST_ASSERT_EQUAL_FLOAT_MESSAGE( + 0.0f, host_val[0], + "bug-1799: an UNAVAIL worker's per-host resInUse must NOT be " + "charged either -- the same reset-skip condition covers both bits"); + + /* --- status clears: normal charging resumes on the very next call --- */ + w->status = 0; + rc = sched_charge_account(&(sched.core), acct); + TEST_ASSERT_EQUAL_INT(0, rc); + + get_resvals(&(w->resInUse), 1, host_idx, host_val); + TEST_ASSERT_EQUAL_FLOAT_MESSAGE( + EXPECTED_HOST_CHARGE, host_val[0], + "once the worker's status bit clears, charging must resume " + "normally on the very next call -- this is a per-call skip, not a " + "permanent block"); +} + /* * resource-accounting.md's reservation-vs-actual-usage reconciliation for * a REAL (gauge-style) resource, e.g. "mem" -- sched_reconcile_host_ @@ -1142,6 +1222,7 @@ int main(void) UNITY_BEGIN(); RUN_TEST(test_charge_account_identical_across_core_containers); RUN_TEST(test_commit_job_alloc_identical_across_core_containers); + RUN_TEST(test_charge_account_skips_reconciling_or_unavail_worker_host_charge); RUN_TEST( test_reconcile_host_resource_usage_below_reservation_keeps_reservation); RUN_TEST( diff --git a/src/scheduler/testsuite/test_sim_contention_parity.c b/src/scheduler/testsuite/test_sim_contention_parity.c index 725c511c..aea20e2f 100644 --- a/src/scheduler/testsuite/test_sim_contention_parity.c +++ b/src/scheduler/testsuite/test_sim_contention_parity.c @@ -1380,49 +1380,70 @@ test_contention_parity_matches_real_normal_mode_recording_round_by_round( } /* ====================================================================== - * scheduler-sim-eager-reconciliation task 3.1 (Req 1.1-1.4): a resource- - * contention fixture where a job's recorded DISPATCH decision and its - * JOB_STATE_CHANGE-to-RUN confirmation are separated by SEVERAL rounds -- - * not the same-round shape the test above already covers (there, + * scheduler-sim-eager-reconciliation task 3.1 (Req 1.1-1.4), REVISED + * 2026-09-11 (user-directed, requirements.md Requirement 1's revision): + * a resource-contention fixture where a job's recorded DISPATCH decision + * and its JOB_STATE_CHANGE-to-RUN confirmation are separated by SEVERAL + * rounds -- not the same-round shape the test above already covers (there, * JOB_STATE_CHANGE lands in the SAME round window as the matching DISPATCH * decision, which is exactly the shape the OLD, pre-task-1.1 trigger gate * (`!JOB_IS_RUN(job->state)` in sim_reconcile_unaccounted_jobs(), removed - * by task 1.1) already handled correctly). This is the genuinely NEW - * scenario the OLD gate could never reconcile: jobs A and C are PEND at - * the exact round their own DISPATCH decision is recorded (round 1), and - * only reach RUN several rounds later (rounds 4 and 5 respectively) via a - * bare JOB_STATE_CHANGE event with no DISPATCH decision of its own -- by - * which point round 1's own decoded_session (the only place their - * DISPATCH ever existed) is long freed. Under the OLD gate, job->accounts - * would stay NULL forever for both jobs, and sched_account_jobs()'s own - * JOB_IS_RUN + accounts!=NULL charging gate would therefore never charge - * them -- resInUse would read a PERMANENTLY-ZERO 0.0 for the rest of the - * replay, wrongly making the worker's full WORKER_CAPACITY look available - * even once both jobs are genuinely running. See this task's own Status - * Report for the live revert-rebuild confirmation that this exact fixture - * reproduces that failure against a reverted tree. + * by task 1.1) already handled correctly). Jobs A and C are PEND at the + * exact round their own DISPATCH decision is recorded (round 1), and only + * reach a SEPARATELY-RECORDED RUN confirmation several rounds later + * (rounds 4 and 5 respectively) via a bare JOB_STATE_CHANGE event with no + * DISPATCH decision of its own -- by which point round 1's own + * decoded_session (the only place their DISPATCH ever existed) is long + * freed. + * + * Pre-task-1.1: job->accounts stayed NULL forever for both jobs (the OLD + * `!JOB_IS_RUN` gate never re-searched a PEND job past its own round), so + * resInUse read a PERMANENTLY-ZERO 0.0 for the rest of the replay. + * + * Task 1.1 through 2026-09-07: `sim_reconcile_unaccounted_jobs()` captured + * job->allocs/job->accounts at round 1 regardless of state, but LEFT + * job->state untouched -- charging stayed deferred to sched_account_jobs()'s + * unchanged JOB_IS_RUN gate until the SEPARATELY-recorded JOB_STATE_CHANGE + * event replayed at round 4/5, several rounds after capture. + * + * 2026-09-11 revision (current, this fixture): capture now ALSO sets + * job->state = JOB_STAT_RUN synchronously (mirroring scheduler.c's + * sched_commit_decision(), bug-1624) -- so BOTH jobs A and C are + * genuinely RUN and charged starting round 1 itself, the SAME round their + * DISPATCH decision is captured, not deferred to rounds 4/5. The later, + * separately-recorded JOB_STATE_CHANGE-to-RUN events at rounds 4/5 are + * still replayed (this fixture keeps them, unchanged, to prove they + * remain safe/idempotent) but no longer change resInUse at all -- they + * re-affirm an already-RUN job. This test's core value survives + * unchanged: prove capacity accounting is CORRECT across a realistic + * multi-round dispatch-to-confirmation gap, not permanently wrong for the + * rest of the replay (the pre-task-1.1 bug) -- but "correct" now means + * "already fully accounted from round 1", not "deferred until + * confirmation." * * Fixture shape (reuses this file's own established constants verbatim -- * WORKER_CAPACITY=10.0, JOBA_WANT=8.0, JOBB_WANT=5.0, JOBC_WANT=2.0, * JOBD_WANT=1.0 -- never re-typed): * Round 1 (begin_round=2000): A/B/C pushed PEND; A(8.0) and C(2.0) - * dispatched (A+C=10.0, exactly saturating capacity), B genuinely - * denied. NEITHER A NOR C's JOB_STATE_CHANGE appears anywhere near - * this round. + * dispatched (A+C=10.0, exactly saturating capacity) -- BOTH + * genuinely RUN and charged from this round onward (2026-09-11 + * revision), B genuinely denied. NEITHER A NOR C's JOB_STATE_CHANGE + * appears anywhere near this round. * Round 2, 3 (begin_round=3000/4000): pure filler -- B is genuinely * re-offered and re-denied each round (builtin_fcfs's own documented * "no-op, retried next round" contract), widening the gap before A/C's - * eventual confirmations below to genuinely "several rounds", and - * proving the deferred-zero state from round 1 persists correctly - * across MULTIPLE idle rounds, not just one. - * Round 4 (begin_round=5000): job A's JOB_STATE_CHANGE to RUN -- 3 - * rounds after its own round-1 DISPATCH. Job C is still PEND -- only - * PARTIAL usage (8.0 of 10.0) must be reflected from this round on. - * Round 5 (begin_round=6000): job C's JOB_STATE_CHANGE to RUN -- 4 - * rounds after ITS OWN round-1 DISPATCH (a deliberately DIFFERENT - * separation than job A's 3 rounds, proving this isn't a single-job - * coincidence). Both jobs now genuinely RUN -- FULL usage (10.0 of - * 10.0) must be reflected from this round on. + * eventual (now redundant) confirmations below to genuinely "several + * rounds", and proving the round-1 charge persists correctly across + * MULTIPLE idle rounds, not just one. + * Round 4 (begin_round=5000): job A's SEPARATELY-recorded JOB_STATE_ + * CHANGE to RUN replays -- 3 rounds after its own round-1 DISPATCH. + * A safe no-op re-affirmation (already RUN since round 1); resInUse + * stays at the SAME full 10.0 it already reached at round 1. + * Round 5 (begin_round=6000): job C's SEPARATELY-recorded JOB_STATE_ + * CHANGE to RUN replays -- 4 rounds after ITS OWN round-1 DISPATCH (a + * deliberately DIFFERENT separation than job A's 3 rounds, proving + * this isn't a single-job coincidence). Also a safe no-op; resInUse + * is unchanged, still 10.0. * * The single test function below drives this SAME fixture shape three * times, truncated after round 1, round 4, and round 5 respectively (via @@ -1597,41 +1618,43 @@ test_contention_capacity_check_correct_across_multi_round_dispatch_confirm_gap( g_capturing = 0; g_round_idx = 0; - /* ---- Checkpoint 1: right after round 1 -- A and C are dispatched but - * still PEND (neither has reached RUN yet). Capacity must look FULLY - * free: a hypothetical job wanting the entire WORKER_CAPACITY must be - * granted, since nothing is genuinely running yet (Requirement 1.2, at - * the capacity-check OUTCOME level, not just a raw resInUse read). ---- */ + /* ---- Checkpoint 1 (REVISED 2026-09-11): right after round 1 -- A and + * C are captured AND already genuinely RUN (synchronous capture-time + * transition, mirroring sched_commit_decision()/bug-1624). Capacity + * must already look FULLY SATURATED: a hypothetical job wanting the + * entire WORKER_CAPACITY must be DENIED, since both A and C are + * already counted from this round onward (Requirement 1.2/1.3, at the + * capacity-check OUTCOME level, not just a raw resInUse read). Under + * the PRE-2026-09-11 design this same check was granted (1) -- the + * inversion itself is the proof the revision landed. ---- */ run_multiround_contention_fixture(&sim1, 1, "_mr1"); rc = check_host_satisfy_want(&sim1, WORKER_CAPACITY); TEST_ASSERT_EQUAL_INT_MESSAGE( - 1, rc, - "round 1: A and C are dispatched-but-still-PEND -- capacity must " - "look FULLY free (a job wanting the entire WORKER_CAPACITY must be " - "granted), since neither job has genuinely reached RUN yet " - "(Requirement 1.2)"); + 0, rc, + "round 1 (2026-09-11 revision): A and C are captured AND already " + "genuinely RUN, synchronously with capture -- capacity must " + "already look FULLY SATURATED (a job wanting the entire " + "WORKER_CAPACITY must be DENIED), not deferred until a later " + "confirmation round (Requirement 1.2/1.3)"); - /* ---- Checkpoint 2: right after round 4 -- job A confirmed RUN, 3 - * rounds after its own round-1 DISPATCH; job C still PEND. Capacity - * must reflect PARTIAL usage: exactly JOBC_WANT (2.0) of headroom - * remains -- a job wanting exactly that must be granted, one wanting - * more (JOBB_WANT, 5.0) must be denied. This is the strongest single - * proof that combined accounting is correct, not just "some job got - * counted": an implementation that dropped job A's contribution (the - * OLD same-round-only bug) would wrongly grant JOBB_WANT here too. ---- */ + /* ---- Checkpoint 2 (REVISED 2026-09-11): right after round 4 -- job + * A's separately-recorded confirmation event has now ALSO replayed, + * but changes nothing: A and C were both already RUN since round 1. + * Capacity must STILL be fully saturated -- even JOBC_WANT (2.0), + * previously grantable under the deferred design, must now be denied, + * since C itself has been counted since round 1 too. ---- */ run_multiround_contention_fixture(&sim4, 4, "_mr4"); rc = check_host_satisfy_want(&sim4, JOBC_WANT); TEST_ASSERT_EQUAL_INT_MESSAGE( - 1, rc, - "round 4: only job A (8.0) is genuinely RUN -- exactly JOBC_WANT " - "(2.0) of headroom must remain and must be granted"); + 0, rc, + "round 4 (2026-09-11 revision): both A and C have been genuinely " + "RUN and fully charged since round 1 -- even JOBC_WANT (2.0) must " + "now be denied, since there is no remaining headroom"); rc = check_host_satisfy_want(&sim4, JOBB_WANT); TEST_ASSERT_EQUAL_INT_MESSAGE( 0, rc, - "round 4: JOBB_WANT (5.0) exceeds the 2.0 remaining headroom -- " - "must be denied. A capacity check that wrongly dropped job A's " - "already-RUN 8.0 usage (the OLD same-round-only bug) would wrongly " - "grant this"); + "round 4: JOBB_WANT (5.0) must also be denied -- capacity has been " + "fully saturated by A+C since round 1"); /* ---- Checkpoint 3: the full 5-round replay -- per-round resInUse * snapshots (Requirement 1.1-1.4 across EVERY round, not just @@ -1661,37 +1684,41 @@ test_contention_capacity_check_correct_across_multi_round_dispatch_confirm_gap( "starting snapshot -- resInUse must read 0.0 (task 4.6's own " "load-time call, not a replayed round)"); TEST_ASSERT_EQUAL_FLOAT_MESSAGE( - 0.0f, g_round_resinuse[1], - "round 1: A/C dispatched but still PEND -- resInUse must read 0.0, " - "not fabricated or prematurely charged (Requirement 1.2)"); + WORKER_CAPACITY, g_round_resinuse[1], + "round 1 (2026-09-11 revision): A(8.0) and C(2.0) are BOTH " + "captured and synchronously transitioned to RUN this round -- " + "resInUse must already reflect the FULL combined 10.0, not " + "deferred to a later confirmation round (Requirement 1.2/1.3)"); TEST_ASSERT_EQUAL_FLOAT_MESSAGE( - 0.0f, g_round_resinuse[2], - "round 2: idle filler round -- the deferred-zero state from round " - "1 must persist, not drift, across an EXTRA round"); + WORKER_CAPACITY, g_round_resinuse[2], + "round 2: idle filler round -- the round-1 charge must persist " + "unchanged across an EXTRA round, not drift or reset"); TEST_ASSERT_EQUAL_FLOAT_MESSAGE( - 0.0f, g_round_resinuse[3], - "round 3: a SECOND idle filler round -- still 0.0, widening the " - "gap before A's confirmation below to genuinely several rounds"); + WORKER_CAPACITY, g_round_resinuse[3], + "round 3: a SECOND idle filler round -- still the full 10.0, " + "widening the gap before A's now-redundant confirmation below to " + "genuinely several rounds"); TEST_ASSERT_EQUAL_FLOAT_MESSAGE( - JOBA_WANT, g_round_resinuse[4], - "round 4: job A's JOB_STATE_CHANGE to RUN just replayed, 3 rounds " - "after A's own round-1 DISPATCH -- resInUse must now reflect A's " - "8.0 usage (Requirement 1.3), and ONLY A's -- job C is still PEND"); + WORKER_CAPACITY, g_round_resinuse[4], + "round 4: job A's separately-recorded JOB_STATE_CHANGE-to-RUN " + "event replays, 3 rounds after A's own round-1 DISPATCH -- a safe " + "no-op re-affirmation (A has been RUN and charged since round 1); " + "resInUse must remain exactly 10.0, not drop, not double-charge"); TEST_ASSERT_EQUAL_FLOAT_MESSAGE( WORKER_CAPACITY, g_round_resinuse[5], - "round 5: job C's JOB_STATE_CHANGE to RUN just replayed, 4 rounds " - "after C's own round-1 DISPATCH (a DIFFERENT separation than A's 3 " - "rounds, proving this isn't a single-job coincidence) -- resInUse " - "must now reflect the FULL combined 10.0 usage of both A and C " - "together"); + "round 5: job C's separately-recorded JOB_STATE_CHANGE-to-RUN " + "event replays, 4 rounds after C's own round-1 DISPATCH (a " + "DIFFERENT separation than A's 3 rounds, proving this isn't a " + "single-job coincidence) -- also a safe no-op; resInUse must " + "remain exactly 10.0, not double-charge to 20.0"); rc = check_host_satisfy_want(&sim5, JOBD_WANT); TEST_ASSERT_EQUAL_INT_MESSAGE( 0, rc, - "round 5: capacity is fully saturated by A+C (10.0 of 10.0) -- " - "even the smallest positive want (JOBD_WANT) must be denied. Under " - "the OLD same-round-only reconciliation gate, neither A nor C " - "would EVER have been captured (their DISPATCH and RUN-" + "round 5: capacity is fully saturated by A+C (10.0 of 10.0, since " + "round 1) -- even the smallest positive want (JOBD_WANT) must be " + "denied. Under the PRE-task-1.1 reconciliation gate, neither A nor " + "C would EVER have been captured (their DISPATCH and RUN-" "confirmation rounds never coincide), so resInUse would read a " "PERMANENTLY-ZERO 0.0 here and this check would WRONGLY grant " "admission"); diff --git a/src/scheduler/testsuite/test_sim_reconcile_unaccounted.c b/src/scheduler/testsuite/test_sim_reconcile_unaccounted.c index bd1e2f35..2e4ce618 100644 --- a/src/scheduler/testsuite/test_sim_reconcile_unaccounted.c +++ b/src/scheduler/testsuite/test_sim_reconcile_unaccounted.c @@ -1036,46 +1036,43 @@ static void test_reconcile_null_args_are_a_safe_noop(void) } /* ---------------------------------------------------------------------- */ -/* Test 12 (scheduler-sim-eager-reconciliation task 2.1, Req 1.2, 1.3, */ -/* 1.4): resource-in-use accounting must defer to the job's REAL running */ -/* state, not its capture time. task 1.1 widened sim_reconcile_unaccounted_ */ -/* jobs()'s trigger so a job's allocation is captured (job->allocs/ */ -/* job->accounts set) as soon as its recorded DISPATCH decision is found, */ -/* regardless of job->state -- this test proves the OTHER half of that */ -/* design's correctness: sched_account_jobs() (scheduler.c, UNCHANGED, */ -/* real body pulled in above) still gates charging on JOB_IS_RUN(state), */ -/* so a captured-but-still-PEND job contributes nothing to w->resInUse */ -/* (Req 1.2) until it genuinely transitions to RUN, at which point the */ -/* SAME captured allocation (never re-captured, never re-typed) is charged */ -/* (Req 1.3). Req 1.4 ("matches normal-mode's equivalent-point reporting") */ -/* is satisfied by construction: this test drives the exact same */ -/* sched_account_jobs() normal mode's own driver.c calls at the identical */ -/* round-boundary position (design.md's Architecture Integration note) -- */ -/* no separate normal-mode comparison fixture is needed to prove a shared */ -/* function behaves consistently with itself. */ +/* Test 12 (scheduler-sim-eager-reconciliation task 4.1/4.2, Req 1.2, 1.3, */ +/* 1.4, REVISED 2026-09-11, user-directed): a captured job's resource */ +/* consumption must be reflected in resInUse starting the SAME round its */ +/* dispatch decision is captured, not deferred to a later round's */ +/* separately-recorded RUN-state confirmation event. sim_reconcile_ */ +/* unaccounted_jobs() now sets job->state = JOB_STAT_RUN synchronously, */ +/* in the same operation as the allocation capture -- mirroring */ +/* scheduler.c's sched_commit_decision() (bug-1624) exactly. This */ +/* INVERTS the pre-2026-09-11 version of this test (formerly */ +/* test_captured_pend_job_excluded_from_resinuse_until_run), which */ +/* asserted the OPPOSITE: that a captured-but-PEND job stayed PEND and */ +/* contributed zero to resInUse until a SEPARATE, later job->state = RUN */ +/* write. See requirements.md's Requirement 1 2026-09-11 revision and */ +/* research.md's corresponding entry for the full rationale trail. */ /* ---------------------------------------------------------------------- */ static void -test_captured_pend_job_excluded_from_resinuse_until_run(void) +test_captured_job_included_in_resinuse_same_round_as_capture(void) { xce_sched_sim_driver_t sim; xce_sched_job_t* job; xce_sched_worker_t* w; xce_job_alloc_t* jalloc; replay_session_t* decoded; + xce_core_event_t ev; int residx; float got; int rc; + int push_rc; fixture_init(&sim); residx = register_slots_resource(&(sim.core)); w = add_worker(&(sim.core), "worker1", residx, 10.0f); - /* PEND-state job, no known allocation yet -- exactly task 1.1's own - * widened trigger shape: its recorded DISPATCH decision has been found - * this round, but its RUN-state confirmation has not been replayed - * yet. */ + /* PEND-state job, no known allocation yet -- its recorded DISPATCH + * decision is about to be found this round. */ job = add_job(&(sim.core), 7030ULL, JOB_STAT_PEND, NULL); TEST_ASSERT_NULL(job->accounts); TEST_ASSERT_NULL(job->allocs); @@ -1088,49 +1085,21 @@ test_captured_pend_job_excluded_from_resinuse_until_run(void) TEST_ASSERT_EQUAL_PTR_MESSAGE( jalloc, job->allocs, - "sanity: the PEND job's recorded DISPATCH decision must be captured " - "(task 1.1's own widened gate) before this test's own checkpoints " - "mean anything"); + "the job's recorded DISPATCH decision must be captured before this " + "test's own checkpoints mean anything"); TEST_ASSERT_NOT_NULL_MESSAGE( - job->accounts, "sanity: job->accounts must be built from the " - "captured allocation even though the job is still " - "PEND"); + job->accounts, + "job->accounts must be built from the captured allocation"); TEST_ASSERT_TRUE_MESSAGE( - JOB_STAT_PEND == job->state, - "sanity: the job must still genuinely be PEND at this checkpoint -- " - "otherwise the next assertion (resInUse == 0) would be trivially " - "true for the wrong reason"); + JOB_STAT_RUN == job->state, + "Requirement 1.2 (2026-09-11 revision): capturing a job's dispatch " + "decision must ALSO transition it to RUN synchronously, mirroring " + "scheduler.c's sched_commit_decision() (bug-1624) -- no separate " + "job->state write should be needed here"); - /* Checkpoint 1 (Requirement 1.2): recompute resInUse from scratch via - * the REAL sched_account_jobs() -- a captured-but-PEND job must - * contribute nothing. */ - rc = sched_account_jobs(&(sim.core)); - TEST_ASSERT_EQUAL_INT(0, rc); - - got = -1.0f; - rc = get_resvals(&(w->resInUse), 1, &residx, &got); - TEST_ASSERT_EQUAL_INT(1, rc); - TEST_ASSERT_EQUAL_FLOAT_MESSAGE( - 0.0f, got, - "Requirement 1.2: a job whose allocation was captured early (while " - "still PEND) must contribute ZERO to its worker's resInUse until it " - "genuinely transitions to RUN -- sched_account_jobs()'s own " - "JOB_IS_RUN gate (scheduler.c, unchanged by this feature) is the " - "sole mechanism deferring this, and must still be effective now " - "that capture happens earlier than before"); - - /* Transition the SAME job to RUN -- direct field write, same white-box - * technique test_sim_contention_parity.c's own capacity_aware_ - * allocator_thread_fn() uses to model a worker-confirmed state change - * without driving the full plugin_job_state_change() machinery (out of - * this file's own boundary -- scheduler.c is not white-box included). */ - job->state = JOB_STAT_RUN; - - /* Checkpoint 2 (Requirement 1.3): recompute resInUse again -- the SAME - * captured allocation (jalloc, never re-captured -- sim_reconcile_ - * unaccounted_jobs() is not called a second time here, proving this - * checkpoint's nonzero value comes from the EARLIER capture, not a - * fresh one) must now be charged. */ + /* Checkpoint 1 (Requirement 1.3, 2026-09-11 revision): recompute + * resInUse via the REAL sched_account_jobs() -- the SAME round as + * capture must ALREADY reflect the charge, not a later one. */ rc = sched_account_jobs(&(sim.core)); TEST_ASSERT_EQUAL_INT(0, rc); @@ -1139,14 +1108,65 @@ test_captured_pend_job_excluded_from_resinuse_until_run(void) TEST_ASSERT_EQUAL_INT(1, rc); TEST_ASSERT_EQUAL_FLOAT_MESSAGE( 4.0f, got, - "Requirement 1.3: once the captured-early job genuinely transitions " - "to RUN, its worker's resInUse must reflect its allocation from " - "that point forward -- using the EXACT allocation amount (4.0) " - "captured earlier by sim_reconcile_unaccounted_jobs(), not a " - "freshly re-captured or re-typed value"); + "Requirement 1.3 (2026-09-11 revision): a captured job's resInUse " + "charge must appear the SAME round its dispatch decision is " + "captured -- sched_account_jobs()'s unchanged JOB_IS_RUN gate now " + "sees the job as RUN immediately, since capture itself sets " + "job->state = JOB_STAT_RUN synchronously"); + + /* Checkpoint 2 (Requirement 1.4, new 2026-09-11): a LATER, separately- + * recorded JOB_STATE_CHANGE-to-RUN event for the SAME job (the + * historical worker-confirmation signal, still present in a real + * recording and still replayed) must be a safe, idempotent + * re-affirmation -- no double-charge, no crash, allocs/accounts + * untouched (since !JOB_IS_RUN(new_state) is false for RUN, the + * handler's cleanup branch is skipped). Drives the REAL + * XCE_CORE_EVENT_JOB_STATE_CHANGE handler (core_events.c), same + * technique test_reconcile_then_release_via_job_state_change_is_asan_ + * clean() above already establishes. */ + memset(&ev, 0, sizeof(ev)); + ev.type = XCE_CORE_EVENT_JOB_STATE_CHANGE; + ev.u.job_state_change.jobid = 7030ULL; + ev.u.job_state_change.old_state = JOB_STAT_RUN; + ev.u.job_state_change.new_state = JOB_STAT_RUN; + + push_rc = xce_core_push_events(&(sim.core), &ev, 1); + TEST_ASSERT_EQUAL_INT(0, push_rc); + + TEST_ASSERT_EQUAL_PTR_MESSAGE( + jalloc, job->allocs, + "Requirement 1.4: a redundant RUN->RUN confirmation event must not " + "touch the already-captured allocation"); + TEST_ASSERT_NOT_NULL_MESSAGE( + job->accounts, + "Requirement 1.4: a redundant RUN->RUN confirmation event must not " + "free the already-captured accounts"); + + /* core_reset_workers() (core_session.c, white-box #include'd above) + * mirrors the REAL per-round sequence sched_sim_driver_run() always + * uses (xce_core_begin_session() zeroes resInUse -> sched_account_jobs() + * re-sums it from scratch) -- production NEVER calls sched_account_jobs() + * a second time without this reset in between, so this call is what + * makes the following recompute a faithful round-4-boundary proof + * instead of an artificial double-charge test-methodology artifact. */ + core_reset_workers(&(sim.core)); + + rc = sched_account_jobs(&(sim.core)); + TEST_ASSERT_EQUAL_INT(0, rc); + + got = -1.0f; + rc = get_resvals(&(w->resInUse), 1, &residx, &got); + TEST_ASSERT_EQUAL_INT(1, rc); + TEST_ASSERT_EQUAL_FLOAT_MESSAGE( + 4.0f, got, + "Requirement 1.4: after a proper reset-then-resum round boundary, " + "the redundant RUN->RUN confirmation event must NOT have changed " + "what gets charged -- still exactly 4.0, the same single " + "allocation, not fabricated or duplicated"); printf( - "test_captured_pend_job_excluded_from_resinuse_until_run: PASS\n"); + "test_captured_job_included_in_resinuse_same_round_as_capture: " + "PASS\n"); } /* ---------------------------------------------------------------------- */ @@ -2121,7 +2141,7 @@ int main(void) RUN_TEST( test_reconcile_reports_reconstructed_decision_through_existing_channel); RUN_TEST(test_reconcile_null_args_are_a_safe_noop); - RUN_TEST(test_captured_pend_job_excluded_from_resinuse_until_run); + RUN_TEST(test_captured_job_included_in_resinuse_same_round_as_capture); RUN_TEST( test_reconcile_still_skips_already_captured_job_regardless_of_state); RUN_TEST(test_reconcile_no_match_leaves_pend_job_uncaptured_and_retries);