From 94115b0024398f67a1e23339ef2298b63aa64f53 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Mon, 21 Sep 2026 09:50:27 +0000 Subject: [PATCH] fix(scheduler): sim mode force-dispatched recorded decisions past live capacity (bug-1836, bug-1838) User-reported: replaying a captured session dispatched 44 single-slot jobs onto a 22-slot host. Root cause traced through two layers: - bug-1836: sim_reconcile_unaccounted_jobs() (the "Recorded-Decision Reconciliation" backstop) force-applied any recorded DISPATCH decision it found, with zero live capacity check. Added sim_reconciled_alloc_fits_capacity(), reusing the same sched_check_host_satisfy()/sched_check_shared_satisfy() admission checks the live picker/allocator path already uses. - bug-1838: even with that gate, reconciliation still preempted every freshly-PEND job before the live picker/allocator got a turn each round -- unnecessary now that scheduler-sim-live-allocation gave sim mode the same real, capacity-checked picker/allocator threads normal mode uses. Narrowed reconciliation's trigger back to JOB_IS_RUN-only (undoing a 2026-09-11 widening whose own justification no longer holds), so a freshly-pushed job now always flows through the same live, verified decision path as normal mode. Reconciliation now only ever fires for a job already RUN from before the replay's own observation window -- the one case structurally invisible to the live picker. Also logged (bug-1837, not fixed here, needs its own design decision): sim_reconcile_unaccounted_jobs() calls a picker's on_dispatch() hook from the driver thread instead of the allocator thread plugin.h documents it belonging to -- currently inert since the only real picker plugin happens to lock internally, but not contract-safe for a future lock-free one. Fixing bug-1838 required re-examining every test exercising the old PEND-inclusive trigger; several needed their fixtures corrected to register a real worker (harmless before any capacity check existed), and others needed their actual intent re-derived and their assertions inverted or restated now that a PEND job is never eligible for reconciliation. Dated correction notes added in place to the scheduler-sim-eager-reconciliation spec, matching its own existing 2026-09-11 correction-note precedent. Verified: full scheduler suite 85/85, independently re-confirmed by running the four most affected binaries directly. Co-Authored-By: Claude Sonnet 5 --- .../design.md | 2 + .../requirements.md | 4 +- .wolf/anatomy-index.json | 112 +++-- .wolf/anatomy.md | 22 +- .wolf/buglog.json | 95 ++++ .wolf/cerebrum.md | 4 + .wolf/memory.md | 67 +++ src/scheduler/include/core.h | 21 + src/scheduler/include/scheduler.h | 19 +- src/scheduler/src/sim_loop.c | 442 ++++++++++++++++-- src/scheduler/testsuite/test_admin_request.c | 49 ++ .../testsuite/test_driver_decision_cb.c | 49 ++ .../testsuite/test_driver_session_ordering.c | 49 ++ .../testsuite/test_driver_translate.c | 49 ++ .../test_maxtasks_wiring_sim_driver_parity.c | 49 ++ .../testsuite/test_record_lifecycle.c | 49 ++ .../test_repl_breakpoints_inspection.c | 49 ++ .../testsuite/test_repl_equivalence.c | 49 ++ .../testsuite/test_repl_event_echo.c | 49 ++ .../testsuite/test_repl_execution_control.c | 49 ++ src/scheduler/testsuite/test_repl_help.c | 49 ++ .../testsuite/test_repl_limit_table.c | 49 ++ .../testsuite/test_repl_prompt_context.c | 49 ++ .../testsuite/test_repl_resource_table.c | 49 ++ .../testsuite/test_repl_round_rollup.c | 49 ++ .../testsuite/test_repl_snapshot_params.c | 49 ++ .../testsuite/test_repl_step_verbosity.c | 49 ++ .../testsuite/test_resval_ownership.c | 49 ++ src/scheduler/testsuite/test_sim_alt_events.c | 49 ++ .../testsuite/test_sim_contention_parity.c | 143 +++--- .../test_sim_e2e_original_motivation.c | 1 + .../testsuite/test_sim_equivalence.c | 49 ++ src/scheduler/testsuite/test_sim_event_loop.c | 49 ++ .../testsuite/test_sim_live_allocation.c | 281 +++++++++++ .../testsuite/test_sim_print_decision_gate.c | 49 ++ .../test_sim_reconcile_unaccounted.c | 383 ++++++++++++--- .../testsuite/test_sim_record_output.c | 49 ++ .../testsuite/test_sim_resource_contention.c | 1 + .../test_sim_round_boundary_reconciliation.c | 1 + .../testsuite/test_sim_session_ordering.c | 49 ++ .../test_sim_snapshot_reconcile_gap.c | 49 ++ src/scheduler/testsuite/test_sim_startup.c | 49 ++ src/scheduler/testsuite/test_sim_verify.c | 229 +++++---- .../testsuite/test_snapshot_save_load.c | 1 + 44 files changed, 2769 insertions(+), 333 deletions(-) diff --git a/.kiro/specs/scheduler-sim-eager-reconciliation/design.md b/.kiro/specs/scheduler-sim-eager-reconciliation/design.md index 27d65b63..0496baca 100644 --- a/.kiro/specs/scheduler-sim-eager-reconciliation/design.md +++ b/.kiro/specs/scheduler-sim-eager-reconciliation/design.md @@ -23,6 +23,8 @@ **Correction (2026-09-07, bug-1759/bug-1779):** the "live picker" Non-Goal above was never meant to license `sim_reconcile_unaccounted_jobs()` leaving a reconciled job as a live picker candidate — it means "do not give the live picker real resource-matching allocation logic" (still out of scope, still `scheduler-allocation`'s territory). Whether a reconciled job stays in the picker's own `pendJobs` bookkeeping is a `scheduler-sim-decision-parity` concern (see that spec's design.md, "Integration with `--sim-verify`", which already documented "the picker never re-decides an already-resolved job" as the correct assumption). The implementation had drifted from that documented assumption: `sim_reconcile_unaccounted_jobs()` never called `picker_iface->on_dispatch()`, so a reconciled job stayed a live candidate and was live-observed being re-offered and re-PENDed every subsequent round (bug-1759: 22/50 already-dispatched jobs in one batch; bug-1779: 5/10 array elements). Fixed by adding that missing `on_dispatch()` call, mirroring `sched_allocator()`'s own live-dispatch ordering (`scheduler.c`) exactly — a correction to already-agreed-upon behavior, not a new capability, so it did not need to reopen this spec's own requirements/tasks. `test_sim_verify_no_false_mismatch_across_round_separated_dispatch` (`test_sim_verify.c`) was updated to match: rounds 2-3 of its fixture are now empty `ReplaySession`s (no more picker-echoed PEND for an already-reconciled job) rather than baking in the pre-fix stale-PEND behavior as "expected." +**Correction (2026-09-21, bug-1838, superseding the 2026-09-11 revision above):** the 2026-09-11 widening (capture "regardless of the job's current state," including PEND) turned out to be the root cause of a live-observed over-dispatch: a 22-slot host took 44 concurrent single-slot jobs during replay (bug-1836), because a PEND job's recorded DISPATCH decision was force-applied by reconciliation with no capacity check against current resource state. bug-1836 first closed the immediate hole by adding a capacity gate (`sim_reconciled_alloc_fits_capacity()`) to reconciliation itself — but investigation into why a PEND job could ever reach this force-apply path at all found that the underlying justification for the 2026-09-11 widening no longer holds: the since-completed `scheduler-sim-live-allocation` spec gave simulation mode a genuine, capacity-checked live picker/allocator thread pair (`sched_job_picker()`/`sched_allocator()`, `sched_election.c`) that both drivers now spawn identically — "one core, two consumers" (`.kiro/steering/scheduler-driver-equivalence.md`) is now actually true for dispatch, not just aspirational. A freshly-PEND job now correctly reaches that SAME live capacity-checked path simulation mode itself runs, exactly like normal mode; reconciliation force-applying a stale recorded decision for it is redundant at best and a second, competing source of truth at worst. The trigger is therefore narrowed back to RUN-only (`JOB_IS_RUN(job->state)`, mutually excluding DONE/EXIT for free under this codebase's 4-state job model): reconciliation's remaining, genuinely necessary scope is a job already confirmed RUN (via a separately-recorded `JOB_STATE_CHANGE`/`JOB_UPDATE` event) whose allocation data was never captured — structurally invisible to the live picker, since `next_job()` never offers a RUN job. This does NOT reopen the 2026-09-11 revision's OTHER change (synchronous RUN-transition-at-capture for the jobs reconciliation still does handle) — that remains correct and unchanged; only the ELIGIBILITY condition for entering capture at all is revised. See `sim_reconcile_unaccounted_jobs()`'s own updated header comment (`sim_loop.c`) for the complete before/after rationale, and Requirement 1's own 2026-09-21 correction (requirements.md) for the corresponding Acceptance Criteria update. + ## Boundary Commitments ### This Spec Owns diff --git a/.kiro/specs/scheduler-sim-eager-reconciliation/requirements.md b/.kiro/specs/scheduler-sim-eager-reconciliation/requirements.md index 39712067..72874b93 100644 --- a/.kiro/specs/scheduler-sim-eager-reconciliation/requirements.md +++ b/.kiro/specs/scheduler-sim-eager-reconciliation/requirements.md @@ -22,8 +22,10 @@ This is a follow-up to the `scheduler-sim-decision-parity` spec (implementation- **Revision (2026-09-11, user-directed, live replay session):** the original version of Acceptance Criteria 2/3 below deferred a captured job's transition to a running state until its separately-recorded `JOB_STATE_CHANGE` confirmation event replayed, potentially many rounds after its dispatch decision was captured -- reproducing a live-observed operator confusion (`info jobs`/`info worker` showing a job as still PEND for multiple sessions after `[sim] DISPATCH` had already printed for it). Investigation confirmed normal-mode execution does NOT work this way: `sched_commit_decision()` (`scheduler.c`, bug-1624) sets `job->state = JOB_STAT_RUN` synchronously, in the SAME operation that captures the dispatch allocation -- from the live scheduler's own perspective, a job IS running the instant the scheduler decides to dispatch it, independent of whatever later, separately-recorded signal confirms the worker actually started it. The Acceptance Criteria below are revised so simulation-mode reconciliation reproduces this same scheduler's-eye-view semantics instead of the worker-confirmation-timeline semantics the original version specified. This is a deliberate, explicit decision to prioritize matching live-mode's own internal state model over preserving the separately-recorded confirmation event's timing as a distinct, independently meaningful signal -- see `research.md`'s corresponding entry for the full rationale trail and the design point that made this ambiguous in the first place. +**Correction (2026-09-21, bug-1838, superseding Acceptance Criterion 1 below):** "regardless of the job's current state at that point" (originally meaning PEND was explicitly an eligible starting state) turned out to be the root cause of a live-observed over-dispatch (bug-1836: 44 single-slot jobs force-applied onto a 22-slot host, since a PEND job's recorded decision was captured with no live capacity check). Investigation found the premise that justified including PEND no longer holds: the since-completed `scheduler-sim-live-allocation` spec gave simulation mode its own genuine, capacity-checked live picker/allocator (`sched_job_picker()`/`sched_allocator()`), so a freshly-PEND job now correctly reaches that same live path instead of needing this feature's capture mechanism at all. Acceptance Criterion 1 is revised: eligibility is once again gated on the job already being in a running state (`JOB_IS_RUN`) -- reconciliation's remaining, genuinely necessary scope is a job already confirmed running (via a separately-recorded `JOB_STATE_CHANGE`/`JOB_UPDATE` event) whose allocation data was never captured, which is structurally invisible to the live picker. This does not reopen Acceptance Criterion 2 (still synchronous RUN-transition at capture) -- it is now simply a no-op re-affirmation for a job that is already running by the time this criterion's precondition can be satisfied. See `design.md`'s corresponding 2026-09-21 Overview correction for the full rationale. + #### Acceptance Criteria -1. When simulation-mode replay encounters a round containing a job's originating dispatch decision, the scheduler shall capture that decision's resource-allocation data for the job, regardless of the job's current state at that point. +1. When simulation-mode replay encounters a round containing a job's originating dispatch decision for a job already in a running state with no allocation data captured yet, the scheduler shall capture that decision's resource-allocation data for the job. (Revised 2026-09-21, bug-1838 -- see correction above; originally "regardless of the job's current state.") 2. When the scheduler captures a job's originating dispatch decision, the scheduler shall also transition that job to a running state at that same round, synchronously with the capture -- mirroring normal-mode execution's own dispatch-commit semantics (Requirement 1's 2026-09-11 revision above), in which a job is considered running from the scheduler's perspective the instant the scheduler decides to dispatch it. 3. The scheduler shall include a captured job's resource consumption in its worker's and any shared resource's resource-in-use totals starting from the same round its dispatch decision is captured, not deferred to a later round. 4. When a job's dispatch decision and its separately-recorded running-state-confirmation event (`JOB_STATE_CHANGE`/`JOB_UPDATE` to a running state) land in different rounds, replaying that later confirmation event shall re-affirm an already-running job (a safe, idempotent re-application) rather than being the sole trigger for the transition to a running state. diff --git a/.wolf/anatomy-index.json b/.wolf/anatomy-index.json index 36784389..2ba96eaf 100644 --- a/.wolf/anatomy-index.json +++ b/.wolf/anatomy-index.json @@ -1,13 +1,13 @@ { "version": 1, "meta": { - "lastScanned": "2026-09-21T03:39:53.909Z", - "fileCount": 666, + "lastScanned": "2026-09-21T09:31:36.646Z", + "fileCount": 668, "hits": 0, "misses": 0, - "renderedHash": "45c99420f9851c89e6b8a493a3c4c5f0a05400f4342a6cd486d0b86ccf3680a8", - "storeUpdatedAt": "2026-09-21T03:39:53.919Z", - "rootHash": "5dc33e10712cf857" + "renderedHash": "56a6cca5dea3c105bb5d64b41019c5e07814ffd9ab069c9aab6f152e32fce1f2", + "storeUpdatedAt": "2026-09-21T09:31:36.655Z", + "rootHash": "cca4c863d5077424" }, "files": { ".agent-shell/transcripts/2026-08-11-11-39-59.md": { @@ -3225,20 +3225,20 @@ }, ".kiro/specs/scheduler-sim-eager-reconciliation/design.md": { "description": "Design Document", - "tokens": 6493, - "hash": "e1712bb6eb14a47b", - "size": 26074, - "mtimeMs": 1789122761693.9182, - "updatedAt": "2026-09-11T10:32:41.734Z", + "tokens": 7078, + "hash": "388510e52bf7c634", + "size": 28419, + "mtimeMs": 1789980075150.255, + "updatedAt": "2026-09-21T08:41:15.194Z", "source": "hook" }, ".kiro/specs/scheduler-sim-eager-reconciliation/requirements.md": { "description": "Requirements Document", - "tokens": 2957, - "hash": "d4e3125d696b25d8", - "size": 11826, - "mtimeMs": 1789122511639.8337, - "updatedAt": "2026-09-11T10:28:31.934Z", + "tokens": 3361, + "hash": "3892b8f845fdd9af", + "size": 13443, + "mtimeMs": 1789980097028.1904, + "updatedAt": "2026-09-21T08:41:37.065Z", "source": "hook" }, ".kiro/specs/scheduler-sim-eager-reconciliation/research.md": { @@ -9836,11 +9836,11 @@ }, "src/scheduler/include/scheduler.h": { "description": "if !defined(_SCHEDULER_H_)", - "tokens": 6125, - "hash": "b963930a3cd4f465", - "size": 22967, - "mtimeMs": 1789750425046.0798, - "updatedAt": "2026-09-18T16:53:45.094Z", + "tokens": 6140, + "hash": "9ebe74de3d64e5e7", + "size": 23023, + "mtimeMs": 1789967319105.591, + "updatedAt": "2026-09-21T05:08:39.157Z", "source": "hook" }, "src/scheduler/src/plugin.c": { @@ -9953,29 +9953,29 @@ }, "src/scheduler/src/sim_loop.c": { "description": "include ", - "tokens": 18113, - "hash": "74a4896cf97efc47", - "size": 67921, - "mtimeMs": 1789794049526.3513, - "updatedAt": "2026-09-19T05:00:49.564Z", + "tokens": 22511, + "hash": "6bacd553e27e51b7", + "size": 84414, + "mtimeMs": 1789977028368.3345, + "updatedAt": "2026-09-21T07:50:28.417Z", "source": "hook" }, "src/scheduler/testsuite/test_sim_reconcile_unaccounted.c": { "description": "Declares documents", - "tokens": 30859, - "hash": "a7ee960113058168", - "size": 115721, - "mtimeMs": 1789794628587.9622, - "updatedAt": "2026-09-19T05:10:28.628Z", + "tokens": 34635, + "hash": "ae87004ae3cee9a6", + "size": 129879, + "mtimeMs": 1789979940956.2974, + "updatedAt": "2026-09-21T08:39:00.991Z", "source": "hook" }, "src/scheduler/testsuite/test_sim_contention_parity.c": { "description": "Declares calls", - "tokens": 21701, - "hash": "907c7366ef4536fe", - "size": 81376, - "mtimeMs": 1789809732381.1213, - "updatedAt": "2026-09-19T09:22:12.429Z", + "tokens": 21740, + "hash": "d824c6b350e69b1d", + "size": 81524, + "mtimeMs": 1789978567598.1646, + "updatedAt": "2026-09-21T08:16:07.642Z", "source": "hook" }, "reports/sep06-sep12-2026.md": { @@ -11566,11 +11566,11 @@ }, "src/scheduler/include/core.h": { "description": "if !defined(_CORE_H_)", - "tokens": 10892, - "hash": "949582dd4823e043", - "size": 40845, - "mtimeMs": 1789721930305.5815, - "updatedAt": "2026-09-18T08:58:50.343Z", + "tokens": 11223, + "hash": "cc63291ebec75cc8", + "size": 42083, + "mtimeMs": 1789969624960.9233, + "updatedAt": "2026-09-21T05:47:05.008Z", "source": "hook" }, "src/scheduler/include/sched/alloc_cache.h": { @@ -11677,8 +11677,8 @@ "tokens": 17581, "hash": "81c8f86b16f80bf3", "size": 65936, - "mtimeMs": 1789752400592.5513, - "updatedAt": "2026-09-18T17:26:40.636Z", + "mtimeMs": 1789969683658.6997, + "updatedAt": "2026-09-21T05:48:03.708Z", "source": "hook" }, "src/scheduler/src/schedobj.c": { @@ -11782,11 +11782,11 @@ }, "src/scheduler/testsuite/test_sim_live_allocation.c": { "description": "Declares already", - "tokens": 5795, - "hash": "97569b95a4d71687", - "size": 21900, - "mtimeMs": 1789788543691.8958, - "updatedAt": "2026-09-19T03:29:03.742Z", + "tokens": 9156, + "hash": "0234199ee3e7f8de", + "size": 34503, + "mtimeMs": 1789983083813.7146, + "updatedAt": "2026-09-21T09:31:23.860Z", "source": "hook" }, "src/scheduler/src/sim_alloc.c": { @@ -12216,6 +12216,24 @@ "mtimeMs": 1789961993857.0706, "updatedAt": "2026-09-21T03:39:53.909Z", "source": "hook" + }, + "src/scheduler/testsuite/test_sim_verify.c": { + "description": "Declares mismatch", + "tokens": 18569, + "hash": "ce46afa8186ebac0", + "size": 69631, + "mtimeMs": 1789978226655.7988, + "updatedAt": "2026-09-21T08:10:26.695Z", + "source": "hook" + }, + "src/scheduler/src/sim_verify.c": { + "description": "include ", + "tokens": 4694, + "hash": "fe98da3155c5abee", + "size": 17601, + "mtimeMs": 1789983096595.9946, + "updatedAt": "2026-09-21T09:31:36.645Z", + "source": "hook" } } } \ No newline at end of file diff --git a/.wolf/anatomy.md b/.wolf/anatomy.md index ff4da164..cd74999d 100644 --- a/.wolf/anatomy.md +++ b/.wolf/anatomy.md @@ -1,7 +1,7 @@ # anatomy.md -> Auto-maintained by OpenWolf. Last scanned: 2026-09-21T03:39:53.909Z -> Files: 666 tracked | Anatomy hits: 0 | Misses: 0 +> Auto-maintained by OpenWolf. Last scanned: 2026-09-21T09:31:36.646Z +> Files: 668 tracked | Anatomy hits: 0 | Misses: 0 ## ./ @@ -541,8 +541,8 @@ ## .kiro/specs/scheduler-sim-eager-reconciliation/ -- `design.md` — Design Document (~6493 tok) -- `requirements.md` — Requirements Document (~2957 tok) +- `design.md` — Design Document (~7078 tok) +- `requirements.md` — Requirements Document (~3361 tok) - `research.md` — Research & Design Decisions (~5166 tok) - `spec.json` (~135 tok) - `tasks.md` — Implementation Plan (~2306 tok) @@ -957,9 +957,9 @@ ## src/scheduler/include/ -- `core.h` — if !defined(_CORE_H_) (~10892 tok) +- `core.h` — if !defined(_CORE_H_) (~11223 tok) - `plugin.h` — if !defined(_PLUGIN_H_) (~5194 tok) -- `scheduler.h` — if !defined(_SCHEDULER_H_) (~6125 tok) +- `scheduler.h` — if !defined(_SCHEDULER_H_) (~6140 tok) - `sim.h` — if !defined(_SIM_H_) (~4213 tok) ## src/scheduler/include/sched/ @@ -988,7 +988,8 @@ - `scheduler.c` — include (~17581 tok) - `sim_alloc.c` — include (~2585 tok) - `sim_limit.c` — include (~2102 tok) -- `sim_loop.c` — include (~18113 tok) +- `sim_loop.c` — include (~22511 tok) +- `sim_verify.c` — include (~4694 tok) - `sim.c` — include (~7432 tok) ## src/scheduler/src/plugins/ @@ -1027,15 +1028,16 @@ - `test_seqjob_rewrap_equivalence.c` — Declares char (~6448 tok) - `test_sim_alloc_limit_fanout.c` — Declares char (~4716 tok) - `test_sim_contention_limit_driver_parity.c` — Declares char (~6514 tok) -- `test_sim_contention_parity.c` — Declares calls (~21701 tok) +- `test_sim_contention_parity.c` — Declares calls (~21740 tok) - `test_sim_contention_resource_driver_parity.c` — Declares char (~7336 tok) - `test_sim_e2e_limit_visibility.c` — Declares char (~8076 tok) - `test_sim_e2e_original_motivation.c` (~18078 tok) - `test_sim_equivalence.c` — Declares BOTH (~11161 tok) -- `test_sim_live_allocation.c` — Declares already (~5795 tok) -- `test_sim_reconcile_unaccounted.c` — Declares documents (~30859 tok) +- `test_sim_live_allocation.c` — Declares already (~9156 tok) +- `test_sim_reconcile_unaccounted.c` — Declares documents (~34635 tok) - `test_sim_resource_contention.c` — Declares is (~10149 tok) - `test_sim_session_ordering.c` (~10531 tok) +- `test_sim_verify.c` — Declares mismatch (~18569 tok) - `test_snapshot_save_load.c` (~29478 tok) - `test.c` — define UNIT_TEST 1 (~7044 tok) diff --git a/.wolf/buglog.json b/.wolf/buglog.json index bf504d83..e0bf3558 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -24776,6 +24776,101 @@ ], "occurrences": 1, "last_seen": "2026-09-21T03:47:10Z" + }, + { + "id": "bug-1836", + "timestamp": "2026-09-20T22:00:00Z", + "error_message": "User-reported live over-dispatch in xce_scheduler -i -s : replaying a captured session (JOB_PUSH of a 50-element array job) against a real single worker with 22 slots dispatched 44 concurrent single-slot jobs (`assigned jobs: 0-3-1789963203-0[0-43]`), exactly double the host's real capacity, while `info worker` simultaneously showed `resource slots: value=22.00 used=22.00` -- the accounting display looked self-consistent (used == value) even though double the job count the display's own 'used' figure implied was actually running. User confirmed normal mode (the live controller-connected driver) does not exhibit this under equivalent conditions, isolating the defect to simulation/replay mode specifically.", + "file": "src/scheduler/src/sim_loop.c", + "root_cause": "sim_reconcile_unaccounted_jobs() (the 'Recorded-Decision Reconciliation' mechanism, scheduler-sim-decision-parity/scheduler-sim-eager-reconciliation specs) unconditionally force-applied any recorded DISPATCH decision it found in decoded_session->decisions for a job with accounts==NULL -- setting job->state=JOB_STAT_RUN and job->allocs/job->accounts straight from the recording -- with ZERO call to sched_check_host_satisfy()/sched_check_shared_satisfy() or any other admission/capacity check. This directly contradicted the scheduler-sim-decision-parity spec's own design.md assumption underpinning its Requirement 2 ('capacity awareness'): 'sched_check_host_satisfy_()/sched_check_shared_satisfy_() already read resInUse/sharedInUse unconditionally; once those fields are correctly populated in simulation mode, the existing (unmodified) capacity-check code path produces the same outcome normal mode would' -- an assumption that only holds for jobs decided via the LIVE picker/allocator path (sched_election.c's sched_allocator(), which DOES call these checks via sched_try_alloc_on_host_()), not for jobs captured via this SEPARATE reconciliation backstop, which bypasses that code path entirely. Confirmed via the round sequence in sched_sim_driver_run(): xce_core_push_events() creates all 50 freshly-pushed jobs (PEND, accounts==NULL) BEFORE sim_reconcile_unaccounted_jobs() runs for that same round, so ALL 50 are eligible reconciliation candidates the moment the function starts -- for however many of them (44, in the live report) the session's OWN originally-recorded decisions said DISPATCH, reconciliation force-applied all of them with no regard for the CURRENT sim snapshot's actual worker capacity (22 slots), before the live picker/allocator (which runs afterward, per the fixed round order) ever got a chance to independently admission-check any of them -- explaining both the 44 dispatches and the 6 correctly-PENDed remainder (jobs whose OWN recorded decision was PEND, never matched by reconciliation's ReplayDecision_body_dispatch-only filter). This is a distinct mechanism from bug-1611/bug-1613 (both already fixed/superseded by scheduler-sim-live-allocation and scheduler-sim-eager-reconciliation) -- those closed a TIMING gap in when reconciliation's job->state transition happens; this bug is a CAPACITY-VALIDATION gap in whether it happens at all, independent of timing. Uncovered via /xce-audit-invariants scheduler, dimension 1 (driver state-equivalence).", + "fix": "FIXED (2026-09-21). Added sim_reconciled_alloc_fits_capacity() (sim_loop.c, static) -- re-validates a candidate recorded allocation's full resource footprint (every per-host allocation plus any shared/global allocation) against LIVE resInUse/sharedInUse using the SAME sched_check_host_satisfy()/sched_check_shared_satisfy() admission-check functions (resource.c) the live picker/allocator path already calls, resolving each allocation's resnames against core->resmgr fresh (the decoded allocation carries only resnames/values, not resolved indices). sim_reconcile_unaccounted_jobs()'s FOUND branch now calls this BEFORE taking ownership of a matched entry; a capacity miss is handled exactly like 'no matching recorded decision' (job left unaccounted this round, entry NOT consumed/freed, naturally retried later or correctly PENDed by the live picker once it runs). A newly-admitted candidate's usage is charged into the LIVE resInUse/sharedInUse immediately via sched_charge_account() (mirroring sched_commit_decision()'s own immediate charge for a live dispatch) so a LATER candidate in the SAME reconciliation pass correctly sees an EARLIER accepted candidate's usage -- essential for the exact multi-job-contention scenario this bug exercises. To make the capacity check see genuinely prior-round (already-running) jobs' usage too, added sim_reconcile_toggle_prior_charges() (sim_loop.c, static) -- seeds resInUse/sharedInUse from every already-RUN+accounted job at the start of sim_reconcile_unaccounted_jobs(), then fully undoes that seed plus every incremental charge made during the pass at the end, leaving the function's net effect on the real accounting arrays at exactly zero on return; the caller's own unchanged, immediately-following sched_account_jobs() call remains the sole, non-double-counting authoritative charge for the round (unchanged architecture/ordering). Deliberately NOT implemented via sched_account_jobs()/a new sched_unaccount_jobs() exported pair (tried first, then reverted): that exact by-name function is separately instrumented and call-count-asserted by test_sim_contention_parity.c's g_round_idx mechanism ('sched_account_jobs() called exactly once per round' is itself a protected invariant there) -- doubling its call count broke 2 of that file's tests (masked at first by a coincidental g_round_resinuse[8] array-capacity cap that hid the true +5/+10 call-count delta behind an apparent '+2' pattern; properly diagnosed by reading the array declaration, not guessed). sched_check_host_satisfy()/sched_check_shared_satisfy() were relocated from scheduler.h to core.h (pure declaration move, bodies unchanged in resource.c, following the exact sched_account_jobs()/sched_resolve_job_picker() precedent already established for this exact purpose) since sim_loop.c must never include scheduler.h (pulls in client.h/xce/cluster.h, forbidden by test_sim_isolation.c's Req 5 AC 17 enforcement). New regression test test_reconcile_respects_host_capacity_under_contention (test_sim_reconcile_unaccounted.c) -- two PEND jobs each recorded-DISPATCH to the same 1-slot-capacity host; asserts exactly one is admitted (not both, not neither), the rejected one's entry is not consumed, and post-round resInUse reads exactly 1.0 (never 2.0). Fixing this surfaced 12 pre-existing tests (9 in test_sim_reconcile_unaccounted.c, 3 in test_sim_verify.c) whose fixtures named a host in a recorded allocation without ever registering a real xce_sched_worker_t of that name -- harmless before this fix (nothing checked capacity), now correctly rejected by the new gate; fixed by registering a real (or, for test_sim_verify.c, which never white-box-includes resource.c, a bare link-satisfying) worker for each such fixture, generous capacity since none of those 12 tests are actually about capacity limits. Verified: full scheduler suite 85/85 Ok, 0 Fail (asan profile, xce-build.sh scheduler asan), independently re-confirmed by running sched_sim_reconcile_unaccounted_test (22/22) and sched_sim_verify_test (17/17) directly with ASAN_OPTIONS=detect_leaks=0 (this test-file family's own established convention, unrelated pre-existing leak reports otherwise printed by these binaries when run standalone outside meson's own suite-level ASAN_OPTIONS).", + "tags": [ + "scheduler", + "sim", + "replay", + "resource-accounting", + "sim_reconcile_unaccounted_jobs", + "scheduler-driver-equivalence", + "over-commit", + "slots", + "capacity-check", + "xce-audit-invariants", + "user-reported", + "found-live", + "fixed" + ], + "related_bugs": [ + "bug-1611", + "bug-1613", + "bug-1799", + "bug-1764", + "bug-1759", + "bug-1779" + ], + "occurrences": 1, + "last_seen": "2026-09-21T05:46:00Z" + }, + { + "id": "bug-1837", + "timestamp": "2026-09-20T22:05:00Z", + "error_message": "Found during /xce-audit-invariants scheduler (dimension 3, jbuf capacity-1 rendezvous audit), not user-reported and not a live symptom: sim_reconcile_unaccounted_jobs() (sim_loop.c) calls picker->picker_iface->on_dispatch(picker, job->jobid) directly from the sim driver thread (sched_sim_driver_run()'s own thread, between xce_core_begin_session() and xce_core_signal_go()), but plugin.h documents on_dispatch()/on_pend() as 'Post-allocation hooks (MUST) -- allocator thread, after each allocation attempt' -- a MUST-level threading contract this call site violates by construction (driver-thread code calling an allocator-thread-only hook).", + "file": "src/scheduler/src/sim_loop.c", + "root_cause": "bug-1759/bug-1779's own fix added this on_dispatch() call to sim_reconcile_unaccounted_jobs() specifically to prevent sim_picker_thread_fn() from re-offering (and re-PENDing) an already-reconciled job on a later round -- a real, necessary fix for that narrower symptom (a duplicate PEND report) -- but the fix's own header comment justifies it only in those terms ('the picker never re-decides an already-resolved job'), never addressing thread-safety of calling an allocator-thread-only hook from the driver thread instead. It currently causes no observable corruption only because the sole real (non-no-op) picker plugin, builtin_fcfs.c, happens to guard its entire candidate queue with an internal lock (priv->lock, taken in every hook including end_session()) -- an implementation detail plugin.h's own documented contract does not require picker plugins to provide (\"the picker thread holds no lock the allocator thread needs\"). A future or alternate picker plugin implemented per that documented (lock-free) contract would race sim_reconcile_unaccounted_jobs()'s driver-thread on_dispatch() call against the picker thread's own end_session()/next_job() tail-end activity from the prior round, unsynchronized.", + "fix": "NOT FIXED -- audit finding only, logged per user request to run the full scheduler-driver-equivalence audit; not addressed in the same pass as bug-1836 to keep that fix's diff properly scoped and reviewable (a real fix here means either moving the on_dispatch() call onto the allocator thread's own loop -- a larger restructuring of the reconciliation mechanism's call boundary -- or documenting/enforcing that every picker plugin must be internally lock-safe against this specific driver-thread call, which would need to be added to plugin.h's own interface contract, not just builtin_fcfs.c's implementation). Currently dormant/inert: builtin_fcfs.c is the only registered non-no-op picker plugin in production, and its internal locking happens to make every call site safe today. Needs its own design decision (which of the two fix directions above) before implementation.", + "tags": [ + "scheduler", + "sim", + "sim_reconcile_unaccounted_jobs", + "picker-plugin", + "threading-contract", + "jbuf", + "xce-audit-invariants", + "dormant", + "not-fixed", + "needs-design-decision", + "related-to-bug-1759", + "related-to-bug-1779" + ], + "related_bugs": [ + "bug-1759", + "bug-1779", + "bug-1836" + ], + "occurrences": 1, + "last_seen": "2026-09-21T05:00:00Z" + }, + { + "id": "bug-1838", + "timestamp": "2026-09-21T00:30:00Z", + "error_message": "Follow-up architectural correction after bug-1836 (found during user discussion, not a separately-reported live symptom): even with bug-1836's capacity gate in place, sim_reconcile_unaccounted_jobs() still unnecessarily PREEMPTED the live picker/allocator for any freshly-PEND job with a matching recorded DISPATCH decision, rather than letting the live path -- now genuinely capacity-checked, per scheduler-sim-live-allocation -- decide it on its own the way normal mode always has. User specifically asked why sim mode diverged from normal mode given they 'share the same core', then proposed the architecturally correct framing directly: the simulator should run the identical decision-making core (minus controller/worker I/O) and use the recording only to verify/reconstruct, not to short-circuit live decisions.", + "file": "src/scheduler/src/sim_loop.c", + "root_cause": "sim_reconcile_unaccounted_jobs()'s trigger condition was widened from JOB_IS_RUN-only to state-independent (PEND-or-RUN) by the scheduler-sim-eager-reconciliation spec (2026-09-11), to fix a real problem at the time: a job's recorded DISPATCH decision only exists in the ONE round's decoded_session it was captured in (freed at round end), so a RUN-only gate meant reconciliation could never find it once the job's SEPARATE, later-replayed RUN-confirmation event finally made it RUN. That widening was the only available fix at the time because sim mode's own live picker/allocator path had no working capacity accounting (bug-1613) and so could not be trusted to independently re-derive the SAME decision the original recording made -- reconciliation's job was to force in the recording's answer since the live path's own answer might diverge or be wrong. scheduler-sim-live-allocation (completed 2026-09-19) closed that gap for real: sim mode now spawns the SAME real, capacity-checked sched_job_picker()/sched_allocator() threads (sched_election.c) normal mode does, with sched_account_jobs() correctly wired into its round sequence -- its own design.md explicitly states a 'Live/Recorded Authority Rule' (live decision wins when the live path reaches a job; reconciliation is a narrow safety net, not a first-resort source of truth) but the actual per-round ORDERING (reconciliation runs unconditionally before the live picker/allocator gets a turn each round) never enforced that rule -- so the PEND-inclusive trigger left in place after scheduler-sim-live-allocation landed kept preempting jobs the live path was now fully capable of deciding correctly and verifiably on its own. This was the deeper root cause underlying bug-1836's exact reproduction (44 jobs force-dispatched via reconciliation before the live picker/allocator ever got a turn); bug-1836's capacity gate made that force-application SAFE (never exceeds real capacity) but did not stop it from happening. Also confirmed (via direct investigation): a snapshot-loaded job (snapshot_load.c's xce_sched_core_load_snapshot()) is registered with the live picker via on_job_push() for every restored job unconditionally, RUN or PEND -- so there is no 'invisible to the picker' case a RUN-only trigger could miss; every legitimate reconciliation candidate today is also, structurally, a job the live picker/allocator can independently resolve. The sole remaining genuine case: a job already RUN before this replay's own observation window (dispatched too far in the past for its DISPATCH decision to still be in any currently-decoded session, and never offered by next_job() since RUN jobs are never picker candidates) -- for that job alone, the recording remains the sole source of truth.", + "fix": "FIXED (2026-09-21). Narrowed sim_reconcile_unaccounted_jobs()'s trigger back to JOB_IS_RUN(job->state)-only (undoing the 2026-09-11 widening), so a freshly-PEND job now always flows through the same real, capacity-checked live picker/allocator normal mode uses -- verified via the pre-existing --sim-verify generate-then-compare mechanism -- rather than being force-applied from history. bug-1836's capacity gate (sim_reconciled_alloc_fits_capacity()/sim_reconcile_toggle_prior_charges()) remains in place unchanged as defense-in-depth for the narrower remaining RUN-only case. Fixing this correctly broke 6 test binaries whose fixtures depended on the old PEND-inclusive behavior; investigation of each was more nuanced than initially guessed: test_sim_verify.c's and test_sim_contention_parity.c's failures were NOT missing-worker-registration issues (the initial hypothesis) but rather that those two files each stub sched_job_picker()/sched_allocator() to a deliberately PEND-only pair (used to keep their OTHER, unrelated tests decoupled from real dispatch logic) -- under the new design a PEND job in those fixtures genuinely can never be dispatched by anything, so the correct fix was inverting each test's assertions to lock in the new, correct behavior (a PEND job stays PEND, and --sim-verify correctly reports the resulting MISMATCH against a history that shows it dispatched, since these minimal harnesses have no real capacity-tracked path to reproduce that decision) rather than trying to make the stub dispatch something it structurally cannot. The 5 directly-affected tests in test_sim_reconcile_unaccounted.c were each individually re-examined for their TRUE intent and fixed accordingly: one inverted/renamed to lock in 'PEND is now always ignored regardless of a matching decision' (test_reconcile_ignores_pend_state_job_even_when_dispatch_recorded), three converted from a PEND-starting to a RUN-starting fixture job (still meaningfully testing the same underlying resInUse-timing/retry/multi-round mechanics, now via the only state that's still eligible), and one (the bug-1759/1779 on_dispatch-retirement regression test) kept its job PEND-built-then-force-set-to-RUN specifically to reproduce the pathological 'RUN job the picker was never told about' state reconciliation's on_dispatch() safety net still needs to guard against, since no real production path can reach that shape anymore post-bug-1838. A sweep for other coincidentally-passing tests found and fixed one more: bug-1836's own new regression test (test_reconcile_respects_host_capacity_under_contention) also used two PEND jobs, converted to RUN with its 'loser' assertion corrected to match. Added a new live-dispatch-vs-recording round-trip test (test_sim_live_allocation.c) to restore the coverage inverting test_sim_verify.c's test removed (that a genuinely live-dispatched decision correctly reproduces a matching historical recording when both the job and the resources it wants are real). scheduler-sim-eager-reconciliation's own requirements.md/design.md got dated 'Correction (2026-09-21, bug-1838)' notes in place, following the exact precedent format its own 2026-09-11 correction already established, superseding (not deleting) the original PEND-inclusive Acceptance Criterion 1 text. Verified: full scheduler suite 85/85 Ok, 0 Fail (asan profile); independently re-confirmed by running sched_sim_reconcile_unaccounted_test (22/22), sched_sim_verify_test (17/17), sched_sim_contention_parity_test (2/2), and sched_sim_live_allocation_test (2/2) directly.", + "tags": [ + "scheduler", + "sim", + "replay", + "sim_reconcile_unaccounted_jobs", + "scheduler-driver-equivalence", + "reconciliation-trigger", + "design-correction", + "architecture", + "user-directed", + "xce-audit-invariants", + "fixed", + "related-to-bug-1836" + ], + "related_bugs": [ + "bug-1836", + "bug-1837", + "bug-1611", + "bug-1613", + "bug-1759", + "bug-1779", + "bug-1804" + ], + "occurrences": 1, + "last_seen": "2026-09-21T02:40:00Z" } ] } diff --git a/.wolf/cerebrum.md b/.wolf/cerebrum.md index eba3b8aa..a0e12a3c 100644 --- a/.wolf/cerebrum.md +++ b/.wolf/cerebrum.md @@ -242,3 +242,7 @@ - **(2026-09-15) `controller-status`'s spec.json tracks its delta/reopen history in a single top-level `delta` object, not an array** — when starting a new delta cycle on top of one that already reached `validate_impl.decision: "GO"`, preserve the just-closed cycle by nesting it under `delta.prior_delta` (reopened_at/reason/new_requirements/impl_status) rather than overwriting it outright, so the requirement-number provenance ("which delta added Req N") stays recoverable from spec.json alone without needing git blame. - **(2026-09-20, bug-1833) `xceadmin`'s bundled JSON schemas exist in TWO copies on disk, and editing the wrong one is silently discarded on the next build.** `src/configs/schema/*.schema.json` is the REAL canonical source; `src/xceadmin/src/xcecommon/schema_files/json/*.schema.json` is a checked-in mirror that `meson-build.sh:104` unconditionally overwrites from the canonical copy on every `xceadmin` build (`cp "$SRC_DIR/configs/schema/"*.schema.json "$SCHEMA_DST/json/"`). Editing only the xceadmin-tree copy produces a schema change that appears correct in `git diff` but has ZERO effect on any real build or test run — confirmed by editing only that copy, rebuilding, and watching a new schema constraint simply not fire (no error, just silent no-op), then finding the sync line and editing both copies to fix it. **Before editing any file under `src/xceadmin/src/xcecommon/schema_files/`, grep `meson-build.sh` for a sync/cp step first** — the same "editing xceadmin source doesn't retroactively update an already-built snapshot" caution documented elsewhere in this file extends to specific FILES within the tree being silently resynced from elsewhere, not just whole-package staleness. - **(2026-09-20, bug-1834) `var` (rtti's `typeinfo.h`) is `#define var void*` — a raw macro, not a typedef — so `var a, b, c;` does NOT declare three `void*`s.** It expands to `void* a, b, c;`, which in C only pointer-types the FIRST declarator; `b` and `c` silently fall back to an implicit `int` (a legacy K&R extension GCC still tolerates by default), producing `-Wincompatible-pointer-types`/`-Wint-conversion` errors at every later use, often far from the declaration and easy to misread as unrelated call-site bugs. Hit writing 4 new tests in `test_limit.c` that copied `var umgr, qmgr, hmgr, pmgr;` from a working single-variable pattern elsewhere in the file. **Always declare each `var` on its own line** (`var umgr;` / `var qmgr;` / ...) — this codebase's own existing tests (`test_build_limit_ok1` et al.) already do this consistently; the comma-list form is the anomaly, not the norm, despite looking harmless for an ordinary typedef'd type. +- **(2026-09-21, bug-1836, `/xce-audit-invariants scheduler`) A driver-equivalence audit needs to check every DECISION-PRODUCING path, not just the "obvious" live one — a second, backstop mechanism (Recorded-Decision Reconciliation, `sim_reconcile_unaccounted_jobs()`, sim_loop.c) can silently bypass the SAME admission-check plumbing the primary path (`sched_allocator()`/`sched_try_alloc_on_host_()`) correctly uses, and a design doc's own stated assumption ("once resInUse is correctly populated in sim mode, the existing capacity-check code path produces the same outcome normal mode would") can be TRUE for the primary path and silently FALSE for a secondary one added by a LATER spec that never re-verified it.** Symptom was a live user report (44 single-slot jobs dispatched onto a 22-slot host during `xce_scheduler -i -s `) that superficially looked like a resource-accounting display bug (`used=22.00` looked self-consistent) but was actually an admission-control bypass: the reconciliation backstop force-applied recorded DISPATCH decisions from a `.replay` file's OWN history with zero live capacity re-validation. Fix pattern: re-run the SAME `sched_check_host_satisfy()`/`sched_check_shared_satisfy()` calls the live path uses, resolving the candidate's resnames fresh against `core->resmgr` (a decoded/recorded allocation carries only names+values, not resolved indices). +- **When a fix needs to peek/seed then undo a resource-accounting baseline (e.g. "temporarily charge every already-running job so a capacity check sees real usage, then undo it before returning"), do NOT reach for the SAME by-name accounting entry point (`sched_account_jobs()`) an existing test elsewhere specifically instruments and call-count-asserts as a protected invariant ("must be called exactly once per round").** Doubling that exact symbol's call count broke `test_sim_contention_parity.c`'s `g_round_idx` mechanism — and the failure was initially confusing because a coincidental fixed-size counter array (`g_round_resinuse[8]`) CAPPED the observed count, making a 2-round fixture's true delta (+2, correctly proportional) look identical to a 5-round fixture's true delta (which should have been +5 but was capped at the array's own ceiling, also showing "+2" after the cap) — always check whether an assertion's input value could itself be silently clamped by an array bound before concluding two different-sized reproductions "coincidentally" show the same delta. Fix: build a small `static` local helper in the SAME file doing the identical `sched_charge_account()`/`sched_uncharge_account()` iteration, deliberately NOT calling the protected symbol by name, so the call-count invariant elsewhere stays intact while the underlying accounting effect is identical. +- **Adding a genuine admission/capacity check to test-owning code can turn dozens of PRE-EXISTING tests red for a completely mechanical, expected reason: their fixtures name a resource/host in a mock allocation without ever registering a real object of that name, because nothing previously validated it.** 12 tests across 2 files broke this way when bug-1836's fix added a `sched_get_worker(hostname) == NULL` check — each one just needed a real (or, in a file that stubs the checker entirely and never even reaches the worker lookup meaningfully, a bare/minimal) worker registered for whichever hostname its fixture already referenced, with generous capacity (these tests were about ownership/dedup/timing mechanics, not capacity limits). This class of breakage is a sign the new check is doing its job, not a sign the check is wrong — verify each failure's root cause individually before assuming a batch of new failures means the production fix itself is flawed. +- **(2026-09-21, bug-1838) When a design revision removes a secondary decision-producing shortcut so more work routes through "the real path," don't assume every test file's "real path" IS real — some test files deliberately stub the exact mechanism you're now relying on, for reasons unrelated to what you're testing.** Predicted two failing tests' root cause as "fixture never registered a real worker" (correct pattern from the bug-1836 experience immediately above) — actually wrong for both: `test_sim_verify.c` and `test_sim_contention_parity.c` each stub `sched_job_picker()`/`sched_allocator()` to a deliberately PEND-only pair, used to keep those files' OTHER tests decoupled from real dispatch logic. Narrowing `sim_reconcile_unaccounted_jobs()`'s trigger to stop force-applying a PEND job's recorded decision meant those fixtures' jobs now correctly stay PEND forever (nothing in that file can ever dispatch them) — the fix wasn't adding a missing worker, it was inverting the test's own assertion to lock in the new, correct behavior ("still PEND, `--sim-verify` correctly reports the resulting MISMATCH, since this minimal harness has no real capacity-tracked path to reproduce a historical dispatch"). **Lesson: before assuming a test failure needs a "make the fixture more realistic" fix, grep the failing test's OWN file for whether the specific mechanism your change now depends on (a picker, an allocator, a resource check) is the REAL production implementation or a same-named local override/stub — a same-file `grep -n "^void\* sched_job_picker\|^int sched_check_host_satisfy"` etc. before touching the test.** A prior, adjacent, and still-correct architectural point (a design doc's own "Live/Recorded Authority Rule" can be stated but never actually enforced by the code's real ordering) is a separate, valid finding from this same investigation — don't conflate "the design doc says X should happen" with "therefore the live mechanism in THIS test file is capable of making X happen." diff --git a/.wolf/memory.md b/.wolf/memory.md index cd6b15f1..7a4ac186 100644 --- a/.wolf/memory.md +++ b/.wolf/memory.md @@ -5533,3 +5533,70 @@ | 20:38 | Edited src/include/typemap.h | 7→10 lines | ~151 | | 20:39 | Edited src/controller/testsuite/test_e2e_capnp.c | modified test_ctrl_hello_request_key_round_trip() | ~1131 | | 20:39 | Edited src/controller/testsuite/test_e2e_capnp.c | 2→3 lines | ~46 | +| 22:07 | Edited src/scheduler/include/core.h | 6→7 lines | ~39 | +| 22:08 | Edited src/scheduler/include/core.h | modified sched_account_jobs() | ~753 | +| 22:08 | Edited src/scheduler/include/scheduler.h | 13→14 lines | ~248 | +| 22:08 | Edited src/scheduler/src/scheduler.c | added 5 condition(s) | ~251 | +| 22:09 | Edited src/scheduler/src/sim_loop.c | 4→7 lines | ~83 | +| 22:10 | Edited src/scheduler/src/sim_loop.c | modified returns() | ~405 | +| 22:11 | Edited src/scheduler/src/sim_loop.c | modified for() | ~166 | +| 22:12 | Edited src/scheduler/src/sim_loop.c | added 1 condition(s) | ~659 | +| 22:12 | Edited src/scheduler/src/sim_loop.c | modified sched_account_jobs() | ~347 | +| 22:14 | Edited src/scheduler/src/sim_loop.c | added 16 condition(s) | ~1298 | +| 22:20 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | added 1 condition(s) | ~1328 | +| 22:32 | Edited src/scheduler/src/sim_loop.c | modified for() | ~90 | +| 22:33 | Edited src/scheduler/src/sim_loop.c | modified if() | ~293 | +| 22:33 | Edited src/scheduler/src/sim_loop.c | modified if() | ~114 | +| 22:43 | Edited src/scheduler/src/sim_loop.c | modified returns() | ~614 | +| 22:44 | Edited src/scheduler/src/sim_loop.c | sched_unaccount_jobs() → sim_reconcile_toggle_prior_charges() | ~331 | +| 22:45 | Edited src/scheduler/src/sim_loop.c | sched_unaccount_jobs() → sim_reconcile_toggle_prior_charges() | ~155 | +| 22:46 | Edited src/scheduler/src/sim_loop.c | added 4 condition(s) | ~526 | +| 22:47 | Edited src/scheduler/include/core.h | reduced (-15 lines) | ~119 | +| 22:48 | Edited src/scheduler/src/scheduler.c | removed 40 lines | ~13 | +| 23:13 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | modified test_reconcile_captures_non_run_state_job_when_dispatch_recorded() | ~283 | +| 23:13 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | modified test_reconcile_reconstructs_allocs_and_accounts_from_matching_dispatch() | ~168 | +| 23:13 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | modified test_reconcile_then_release_via_job_state_change_is_asan_clean() | ~149 | +| 23:14 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | 9→14 lines | ~117 | +| 23:14 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | 8→13 lines | ~127 | +| 23:14 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | expanded (+10 lines) | ~211 | +| 23:14 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | 9→14 lines | ~119 | +| 23:15 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | 9→14 lines | ~118 | +| 23:15 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | expanded (+10 lines) | ~207 | +| 23:19 | Edited src/scheduler/testsuite/test_sim_verify.c | modified add_bare_worker() | ~339 | +| 23:20 | Edited src/scheduler/testsuite/test_sim_verify.c | expanded (+7 lines) | ~129 | +| 23:28 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | 8→13 lines | ~224 | +| 00:49 | Edited src/scheduler/src/sim_loop.c | modified condition() | ~1331 | +| 00:50 | Edited src/scheduler/src/sim_loop.c | 4→9 lines | ~117 | +| 01:05 | Edited src/scheduler/testsuite/test_sim_live_allocation.c | added 1 condition(s) | ~961 | +| 01:07 | Edited src/scheduler/testsuite/test_sim_live_allocation.c | modified comment() | ~2255 | +| 01:08 | Edited src/scheduler/testsuite/test_sim_verify.c | modified INVERTED() | ~671 | +| 01:09 | Edited src/scheduler/testsuite/test_sim_verify.c | reduced (-7 lines) | ~573 | +| 01:10 | Edited src/scheduler/testsuite/test_sim_verify.c | 2→2 lines | ~24 | +| 01:16 | Edited src/scheduler/testsuite/test_sim_contention_parity.c | modified CORRECTED() | ~1686 | +| 01:17 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | modified test_reconcile_ignores_pend_state_job_even_when_dispatch_recorded() | ~1005 | +| 01:18 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | 2→2 lines | ~24 | +| 01:19 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | modified 12() | ~798 | +| 01:19 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | 3→3 lines | ~63 | +| 01:20 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | modified REVISED() | ~803 | +| 01:27 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | modified 15() | ~582 | +| 01:27 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | 3→3 lines | ~54 | +| 01:28 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | expanded (+9 lines) | ~530 | +| 01:33 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | modified test_reconcile_removes_reconciled_job_from_picker_candidates() | ~822 | +| 01:33 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | 11→12 lines | ~161 | +| 01:35 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | modified 13() | ~458 | +| 01:35 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | 6→7 lines | ~111 | +| 01:36 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | modified regression() | ~312 | +| 01:36 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | 2→2 lines | ~33 | +| 01:38 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | modified if() | ~291 | +| 01:39 | Edited src/scheduler/testsuite/test_sim_reconcile_unaccounted.c | 4→4 lines | ~35 | +| 01:41 | Edited .kiro/specs/scheduler-sim-eager-reconciliation/design.md | modified Correction() | ~1035 | +| 01:41 | Edited .kiro/specs/scheduler-sim-eager-reconciliation/requirements.md | modified Correction() | ~612 | +| 01:52 | Edited src/scheduler/testsuite/test_sim_live_allocation.c | 6→7 lines | ~42 | +| 01:52 | Edited src/scheduler/testsuite/test_sim_live_allocation.c | 36→38 lines | ~393 | +| 01:59 | Edited src/scheduler/testsuite/test_sim_live_allocation.c | 3→4 lines | ~43 | +| 02:13 | Edited src/scheduler/src/sim_verify.c | expanded (+12 lines) | ~220 | +| 02:20 | Edited src/scheduler/testsuite/test_sim_live_allocation.c | 2→2 lines | ~35 | +| 02:30 | Edited src/scheduler/testsuite/test_sim_live_allocation.c | 2→2 lines | ~35 | +| 02:31 | Edited src/scheduler/testsuite/test_sim_live_allocation.c | expanded (+22 lines) | ~556 | +| 02:31 | Edited src/scheduler/testsuite/test_sim_live_allocation.c | 4→3 lines | ~28 | +| 02:31 | Edited src/scheduler/src/sim_verify.c | removed 13 lines | ~17 | diff --git a/src/scheduler/include/core.h b/src/scheduler/include/core.h index fd14e803..6054d4d7 100644 --- a/src/scheduler/include/core.h +++ b/src/scheduler/include/core.h @@ -49,6 +49,7 @@ #include "plugin.h" #include "sched/types.h" #include "xce/types.h" +#include "xvec.h" #include "zarray.h" /* xce_sched_core_t is also forward-declared in plugin.h (picker lifecycle @@ -403,6 +404,26 @@ extern int sched_charge_account(xce_sched_core_t*, xce_job_account_t*); extern int sched_uncharge_account(xce_sched_core_t*, xce_job_account_t*); extern int sched_commit_job_alloc(xce_sched_core_t*, xce_sched_job_t*); +/* + * sched_check_host_satisfy()/sched_check_shared_satisfy() - relocated here + * from scheduler.h (scheduler-driver-equivalence fix, bug-1836), following + * the exact precedent sched_account_jobs()/sched_resolve_job_picker() + * already established above: pure declaration move, body unchanged (still + * defined in resource.c) -- scheduler.h's own doc comment on these two + * already documented them as "callable identically from either driver" + * (scheduler-sim-live-allocation task 1.3), but scheduler.h itself pulls in + * controller-facing headers (client.h, xce/cluster.h) sim_loop.c must never + * include (Req 5 AC 17 / test_sim_isolation.c). Needed by + * sim_reconcile_unaccounted_jobs()'s new capacity gate, which must + * re-validate a recorded decision's resource footprint against LIVE + * resInUse/sharedInUse the same way the live picker/allocator path already + * does, instead of trusting the recording unconditionally. + */ +extern int sched_check_shared_satisfy(xce_sched_core_t*, xvec_t*, + xvec_idval_t*, xvec_idval_t*); +extern int sched_check_host_satisfy(xce_sched_core_t*, xce_sched_worker_t*, + xvec_t*, xvec_idval_t*, xvec_idval_t*); + /* * sched_reconcile_host_resource_usage()/sched_reconcile_job_resource_usage() * - resource-accounting.md's reservation-vs-actual-usage policy for a REAL diff --git a/src/scheduler/include/scheduler.h b/src/scheduler/include/scheduler.h index 77d38a04..3eb38060 100644 --- a/src/scheduler/include/scheduler.h +++ b/src/scheduler/include/scheduler.h @@ -366,17 +366,18 @@ extern int sched_publish_dispatch(xce_scheduler_t*, xce_job_dispatch_t*); extern void sched_check_pending_rejoin(xce_scheduler_t*); extern void sched_worker_release_host_ctx(xce_sched_worker_t*); /* scheduler-sim-live-allocation spec, task 1.3: retyped from - * xce_scheduler_t* to xce_sched_core_t* -- these three now operate on - * shared core state directly, callable identically from either driver - * (both xce_scheduler_t and xce_sched_sim_driver_t embed xce_sched_core_t - * core as their first member). Body unchanged besides sched->core.X -> - * core->X field-access updates. */ + * xce_scheduler_t* to xce_sched_core_t* -- these now operate on shared core + * state directly, callable identically from either driver (both + * xce_scheduler_t and xce_sched_sim_driver_t embed xce_sched_core_t core as + * their first member). Body unchanged besides sched->core.X -> core->X + * field-access updates. sched_check_shared_satisfy()/sched_check_host_ + * satisfy() relocated to core.h (scheduler-driver-equivalence fix, + * bug-1836) -- same rationale as sched_account_jobs' own relocation there: + * sim_loop.c needs to call them without pulling in scheduler.h's + * controller-facing includes (client.h, xce/cluster.h), which + * test_sim_isolation.c forbids. */ extern int sched_meet_host_selection(xce_sched_core_t*, xce_sched_worker_t*, xce_select_t*); -extern int sched_check_shared_satisfy(xce_sched_core_t*, xvec_t*, xvec_idval_t*, - xvec_idval_t*); -extern int sched_check_host_satisfy(xce_sched_core_t*, xce_sched_worker_t*, - xvec_t*, xvec_idval_t*, xvec_idval_t*); extern int sched_collect_workers(xce_scheduler_t*, xlist_t*, zarray_iter_skip_fn_t, sched_group_worker_fn_t); diff --git a/src/scheduler/src/sim_loop.c b/src/scheduler/src/sim_loop.c index e41aa152..e5a2da94 100644 --- a/src/scheduler/src/sim_loop.c +++ b/src/scheduler/src/sim_loop.c @@ -6,6 +6,9 @@ #include #include "controller.capnp.h" +#include "kvmgr.h" /* key_find()/lock()/unlock() -- Recorded-Decision + * Reconciliation's capacity gate (bug-1836) resolves a + * recorded allocation's resnames against core->resmgr */ #include "plugin.h" #include "preason.h" #include "repl.h" @@ -542,6 +545,225 @@ static inline void sim_print_decision_gate(const xce_sched_sim_driver_t* sim, sim_print_decision(d); } +/* + * sim_reconciled_alloc_fits_capacity - Recorded-Decision Reconciliation + * capacity gate (scheduler-driver-equivalence fix, bug-1836). Does `alloc`'s + * full resource footprint (every per-host allocation, plus any shared/ + * global allocation) currently fit within `core`'s LIVE resInUse/ + * sharedInUse? Reuses the exact admission-check functions the live picker/ + * allocator path calls (sched_check_host_satisfy()/sched_check_shared_ + * satisfy(), resource.c) so a recorded decision is held to the SAME bar a + * freshly-decided one would be -- the whole point of this gate. Each + * host_alloc/shared-alloc's resnames are resolved against core->resmgr + * fresh (mirrors resreq.c's calc_wanted_host_resvals()'s own key_find() + * pattern) since `alloc` (deep-copied out of the capnp arena by + * decode_ReplaySession_ptr(), see sched_sim_driver_run()'s own comment on + * that) carries only resnames/values, not resolved indices. Passes a fresh, + * all-zero `uncomitted` scratch vector (no extra debit) since this is a + * one-shot check against a fully-resolved allocation, not a ctx-scoped + * multi-host search. + * + * Returns 1 if the whole allocation fits, 0 if any part of it does not (a + * capacity miss, not an error -- the caller must leave the job unaccounted + * this round, exactly like "no matching recorded decision" already is), <0 + * on infrastructure error (OOM -- never a capacity determination). + */ +static int sim_reconciled_alloc_fits_capacity(xce_sched_core_t* core, + xce_job_alloc_t* alloc) +{ + int i; + int rc; + + if ((core == NULL) || (alloc == NULL)) { + return -EINVAL; + } + + for (i = 0; i < alloc->nhosts; i++) { + xce_host_alloc_t* halloc = alloc->allocs[i]; + xce_sched_worker_t* w; + xvec_idval_t wanted; + xvec_idval_t unmeet = { 0 }; + xvec_t uncomit; + int j; + + if ((halloc == NULL) || (halloc->nallocs <= 0)) { + continue; + } + + w = sched_get_worker(core, halloc->host); + if (w == NULL) { + /* The recorded host is no longer known to this sim's core -- + * cannot possibly be satisfied now. A capacity miss, not an + * infrastructure error. */ + return 0; + } + + rc = xvec_idval_init(&wanted, halloc->nallocs); + if (rc < 0) { + return rc; + } + + for (j = 0; j < halloc->nallocs; j++) { + xce_alloc_t* a = halloc->allocs[j]; + xce_resource_t* r = NULL; + + if (a == NULL) { + continue; + } + + lock(core->resmgr, RTTI_LOCK_READ); + rc = key_find(core->resmgr, (void*)a->resname, (void**)&r); + unlock(core->resmgr); + + if ((rc <= 0) || (r == NULL)) { + continue; + } + + xvec_idval_store(&wanted, r->idx, (float)a->value); + } + + rc = xvec_init(&uncomit, (wanted.count > 0) ? wanted.count : 1, + sizeof(float)); + if (rc < 0) { + xvec_idval_fini(&wanted); + return rc; + } + + rc = sched_check_host_satisfy(core, w, &uncomit, &wanted, &unmeet); + + xvec_fini(&uncomit); + xvec_idval_fini(&wanted); + + if (rc < 0) { + /* sched_check_host_satisfy() already finalized `unmeet` itself + * on this path (its own wrapper contract). */ + return rc; + } + + /* rc == 0 or 1: `unmeet` was either genuinely populated by + * sched_check_host_satisfy_() (both outcomes -- it unconditionally + * xvec_idval_init()s it whenever wanted->count > 0, satisfied or + * not) or never touched at all (the wanted->count == 0 shortcut, + * which returns 1 without calling the inner _satisfy_() at all) -- + * the `= {0}` zero-init above makes finalizing it unconditionally + * safe either way (a no-op on the untouched, all-zero case). */ + xvec_idval_fini(&unmeet); + + if (rc == 0) { + return 0; + } + } + + if (alloc->nres_shared > 0) { + xvec_idval_t wanted; + xvec_idval_t unmeet = { 0 }; + xvec_t uncomit; + int j; + + rc = xvec_idval_init(&wanted, alloc->nres_shared); + if (rc < 0) { + return rc; + } + + for (j = 0; j < alloc->nres_shared; j++) { + xce_alloc_t* a = alloc->allocs_shared[j]; + xce_resource_t* r = NULL; + + if (a == NULL) { + continue; + } + + lock(core->resmgr, RTTI_LOCK_READ); + rc = key_find(core->resmgr, (void*)a->resname, (void**)&r); + unlock(core->resmgr); + + if ((rc <= 0) || (r == NULL)) { + continue; + } + + xvec_idval_store(&wanted, r->idx, (float)a->value); + } + + rc = xvec_init(&uncomit, (wanted.count > 0) ? wanted.count : 1, + sizeof(float)); + if (rc < 0) { + xvec_idval_fini(&wanted); + return rc; + } + + rc = sched_check_shared_satisfy(core, &uncomit, &wanted, &unmeet); + + xvec_fini(&uncomit); + xvec_idval_fini(&wanted); + + if (rc < 0) { + return rc; + } + + /* See the per-host block above: unconditionally safe (its {0} + * zero-init makes finalizing a never-populated `unmeet` a no-op). */ + xvec_idval_fini(&unmeet); + + if (rc == 0) { + return 0; + } + } + + return 1; +} + +/* + * sim_reconcile_toggle_prior_charges - bug-1836 fix helper. Applies (charge + * != 0) or reverses (charge == 0) sched_charge_account()'s effect for every + * job already JOB_IS_RUN(state) with a resolved job->accounts -- the exact + * same job set sched_account_jobs() (scheduler.c) itself walks, using the + * same underlying sched_charge_account()/sched_uncharge_account() + * primitives sched_account_jobs()/its own inverse would use. Deliberately + * does NOT call sched_account_jobs()/an sched_unaccount_jobs() equivalent + * by name: that exact, by-name entry point is separately instrumented and + * call-count-asserted elsewhere (test_sim_contention_parity.c's + * g_round_idx mechanism treats "sched_account_jobs() called exactly once + * per round" as a protected invariant) -- doubling its call count from + * sim_reconcile_unaccounted_jobs()'s own transient capacity-check seed + * (below) would falsely trip that unrelated assertion despite being + * functionally correct, so this local, uncounted helper achieves the + * identical resInUse/sharedInUse effect through the same lower-level + * primitives instead. + */ +static void sim_reconcile_toggle_prior_charges(xce_sched_core_t* core, + int charge) +{ + iterator_t it; + int rc; + + rc = iter_make(&it, core->jobs); + if (rc < 0) { + return; + } + + for (;;) { + xce_sched_job_t* job = NULL; + + rc = iter_next(&it, (void**)&job); + if ((rc < 0) || (rc == RTTI_ITER_NO_DATA)) { + break; + } + + if ((job == NULL) || !JOB_IS_RUN(job->state) + || (job->accounts == NULL)) { + continue; + } + + if (charge) { + sched_charge_account(core, job->accounts); + } else { + sched_uncharge_account(core, job->accounts); + } + } + + iter_destroy(&it); +} + /* * sim_reconcile_unaccounted_jobs - Recorded-Decision Reconciliation * (scheduler-sim-decision-parity spec, design.md's "Recorded-Decision @@ -571,46 +793,81 @@ static inline void sim_print_decision_gate(const xce_sched_sim_driver_t* sim, * round, so a not-yet-captured job is simply re-checked round after round * until its originating decision is found (or the replay ends). * - * job->state transition (REVISED 2026-09-11, scheduler-sim-eager- - * reconciliation Requirement 1.2/1.3, user-directed): a successful FOUND - * match ALSO sets job->state = JOB_STAT_RUN synchronously, in the same - * operation as the allocation capture -- mirroring scheduler.c's - * sched_commit_decision() (bug-1624) exactly, which treats a job as - * running from the scheduler's own perspective the instant it decides to - * dispatch, independent of any later, separately-recorded worker- - * confirmation signal. sched_account_jobs()'s own unchanged `JOB_IS_RUN` - * gate (scheduler.c) therefore includes this job's charge starting THIS - * round, not a later one when its confirmation event happens to replay. - * (Previously this function left job->state untouched here, deferring the - * RUN transition entirely to a later JOB_STATE_CHANGE/JOB_UPDATE replay -- - * see requirements.md's Requirement 1 2026-09-11 revision for the full - * rationale trail.) Requirement 3.3's "report only the - * fresh one" guarantee needs no separate mechanism here: a job that - * independently receives a fresh live decision in this same round already - * has job->accounts set by sched_commit_decision() (scheduler.c, unchanged) - * by the time this trigger check runs, so this function simply finds - * nothing to do for it. + * Trigger condition (REVISED 2026-09-21, bug-1838 -- see that bug's own + * writeup for the full before/after evidence trail): candidates are once + * again restricted to `JOB_IS_RUN(job->state)` jobs only, undoing the + * 2026-09-11 scheduler-sim-eager-reconciliation widening to a state- + * independent (PEND-or-RUN) trigger. That widening was justified at the + * time by a real gap: a job's recorded DISPATCH decision only exists in + * the ONE round's decoded_session it was originally captured in (freed at + * round end), so a RUN-only gate meant reconciliation could never find it + * once a job's SEPARATE, later-replayed RUN-confirmation event finally + * made the job RUN -- by then the round holding the matching DISPATCH + * entry was long gone. At the time this widening landed, that was the only + * available fix, because sim mode's live picker/allocator path had no + * working capacity accounting of its own (bug-1613) and so could not be + * trusted to independently re-derive the SAME dispatch decision the + * original recording made -- reconciliation's job was to force in the + * recording's answer since the live path's own answer might be wrong. * - * This PEND/RUN state-independence does NOT extend to a job that has - * already reached a terminal state: `JOB_IS_DONE(job->state) || - * JOB_IS_EXIT(job->state)` is an additional, unconditional exclusion, - * checked alongside `job->accounts != NULL` in the same trigger `if`. A - * terminal job's `job->accounts`/`job->allocs` are freed by - * xce_core_push_events()'s own end-of-life cleanup on the terminal - * transition (core_events.c's JOB_STATE_CHANGE/JOB_UPDATE cases), which - * leaves `job->accounts == NULL` -- the exact same shape as a legitimately - * not-yet-captured PEND/RUN job. Without this exclusion, a later round's - * decoded_session carrying a stale/duplicate DISPATCH entry for that same - * jobid (e.g. a retried dispatch attempt for one element of an array job) - * would be indistinguishable from a genuine early-dispatch case and would - * resurrect the allocation on an already-dead job (bug-1764). The two rules - * compose without contradiction: state-independence is about WHEN a - * (still-live) job's own DISPATCH decision happens to replay relative to - * its RUN confirmation -- it says nothing about a job that can never - * legitimately be reconsidered again because it has already finished. This - * mirrors job_state_regresses_terminal()'s (core_events.c) existing - * "terminal is permanent" invariant, applied here at the allocation layer - * instead of the state-transition layer. + * scheduler-sim-live-allocation (completed 2026-09-19) closed that gap for + * real: sim mode now spawns the SAME real, capacity-checked sched_job_ + * picker()/sched_allocator() threads (sched_election.c) normal mode does, + * with sched_account_jobs() correctly wired into its round sequence. A + * freshly-PEND job (e.g. from a JOB_PUSH event) is now a perfectly ordinary + * picker candidate that the live path independently and correctly decides + * within the SAME round it becomes PEND -- no reconciliation needed, and + * (unlike force-applying the recording) the live decision is validated + * against LIVE resInUse/sharedInUse and cross-checkable via --sim-verify. + * scheduler-sim-live-allocation's own design.md states this as an explicit + * "Live/Recorded Authority Rule": the live decision is authoritative when + * the live path reaches a job; reconciliation is a narrow safety net, not + * a first-resort source of truth. + * + * The PEND-inclusive trigger, left in place after that fix landed, was + * bug-1836's actual root cause: because reconciliation runs (below, + * unconditionally over every accounts==NULL job) BEFORE the live picker/ + * allocator gets a turn each round, it was preempting freshly-PEND jobs + * the live path would have correctly, capacity-checked, decided on its own + * -- and, for a batch recorded as dispatched under a DIFFERENT resource + * reality (a different topology, or a value later changed by a replayed + * RESVALS_DELTA/WORKERS_PUSH event), force-applying a decision that no + * longer fit current capacity (bug-1836's own fix added a capacity check + * for exactly this, and remains in place below as defense-in-depth for + * whatever this narrower trigger still legitimately reaches -- but with a + * RUN-only trigger, a freshly-PEND job no longer reaches reconciliation's + * force-apply path AT ALL, so the live path's own admission check is what + * actually gates it, matching normal mode exactly). + * + * A snapshot-loaded job (`xce_sched_core_load_snapshot()`, snapshot_load.c) + * is registered with the picker via `on_job_push()` for every restored + * job unconditionally, RUN or PEND -- so there is no "invisible to the + * picker" case a RUN-only trigger could miss either; every legitimate + * reconciliation candidate today is also, structurally, a job the live + * picker/allocator can independently resolve. The one case a RUN-only + * trigger still genuinely needs to cover: a job already RUN before this + * replay's own observation window (dispatched too far in the past for its + * DISPATCH decision to still be in ANY currently-decoded session, and + * never a picker candidate at all since RUN jobs are never offered by + * next_job()) -- for that job ONLY, the recording is the sole source of + * truth, and reconciliation remains genuinely load-bearing. + * + * job->state on a FOUND match: still explicitly set to JOB_STAT_RUN + * (mirroring scheduler.c's sched_commit_decision(), bug-1624) even though, + * under this RUN-only trigger, the job already IS RUN by construction -- + * kept as a harmless, defensive no-op (same "RUN->RUN reassignment is a + * verified-safe no-op" pattern core_events.c's own JOB_STATE_CHANGE + * handler already relies on) rather than special-cased away, since a + * future change to the trigger should not silently depend on this line + * still being here. + * + * Terminal-state exclusion: `JOB_IS_DONE(job->state) || JOB_IS_EXIT(...)` + * is now redundant with (implied by) the RUN-only trigger itself -- + * JOB_IS_RUN/JOB_IS_DONE/JOB_IS_EXIT are mutually exclusive per this + * codebase's 4-state job model (PEND/RUN/EXIT/DONE) -- but bug-1764's own + * underlying hazard (a stale/duplicate DISPATCH entry resurrecting an + * already-dead job's allocation) is unaffected either way: a DONE/EXIT job + * was never going to match `JOB_IS_RUN` in the first place now. * * For each such job, searches `decoded_session->decisions` for a matching * DISPATCH entry by jobid -- a job can only ever be captured via a recorded @@ -700,8 +957,41 @@ static void sim_reconcile_unaccounted_jobs(xce_sched_sim_driver_t* sim, * file family uses. */ (void)sched_resolve_job_picker(&(sim->core), &picker); + /* bug-1836 fix (scheduler-driver-equivalence: Recorded-Decision + * Reconciliation capacity gate): seed resInUse/sharedInUse with every + * ALREADY-running job's real charge before this function's own loop + * (below) starts admission-checking candidates against them -- no + * candidate this loop is about to examine has job->accounts set yet + * (that is exactly this function's own trigger condition), so this + * only ever picks up genuinely prior-round jobs, never double-counts. + * This function's own incremental sched_charge_account() calls (below, + * once a candidate passes its capacity check) build on top of this seed + * so a LATER candidate in this SAME loop correctly sees an EARLIER + * accepted candidate's usage too. Undone in full via the matching + * sim_reconcile_toggle_prior_charges(..., 0) call before this function + * returns (see the bottom of this function) -- this is a transient + * peek for the capacity check alone, not a real charge: the one + * authoritative charge for this round is still the existing + * sched_account_jobs() call sim_loop.c's caller makes immediately after + * this function returns (unchanged), which by then correctly sees + * every job this function reconciled too (their accounts are now + * permanently set). + * + * Deliberately NOT sched_account_jobs() itself: that exact, by-name + * symbol is separately instrumented and call-count-asserted elsewhere + * (test_sim_contention_parity.c's g_round_idx mechanism treats "exactly + * once per round" as a protected invariant) -- doubling its call count + * here would falsely trip that unrelated assertion despite being + * functionally correct. sim_reconcile_toggle_prior_charges() achieves + * the identical resInUse/sharedInUse effect via the same underlying + * sched_charge_account()/sched_uncharge_account() primitives + * sched_account_jobs() itself is built on, without going through that + * specific, separately-tracked entry point. */ + sim_reconcile_toggle_prior_charges(&(sim->core), 1); + rc = iter_make(&it, sim->core.jobs); if (rc < 0) { + sim_reconcile_toggle_prior_charges(&(sim->core), 0); return; } @@ -715,8 +1005,13 @@ static void sim_reconcile_unaccounted_jobs(xce_sched_sim_driver_t* sim, break; } - if ((job == NULL) || (job->accounts != NULL) || JOB_IS_DONE(job->state) - || JOB_IS_EXIT(job->state)) { + /* bug-1838: RUN-only once again (see this function's own header + * comment for the full before/after rationale) -- also excludes + * DONE/EXIT for free, since this codebase's 4-state job model + * (PEND/RUN/EXIT/DONE) makes JOB_IS_RUN mutually exclusive with + * both. */ + if ((job == NULL) || (job->accounts != NULL) + || !JOB_IS_RUN(job->state)) { continue; } @@ -732,6 +1027,43 @@ static void sim_reconcile_unaccounted_jobs(xce_sched_sim_driver_t* sim, continue; } + /* bug-1836 fix (Recorded-Decision Reconciliation capacity + * gate): a recorded DISPATCH decision is a claim about what + * happened when this session was ORIGINALLY captured -- it is + * not guaranteed to still fit this replay's CURRENT resource + * reality (e.g. a resource genuinely changed value earlier in + * this same recording via a RESVALS_DELTA/WORKERS_PUSH event, + * or the loaded snapshot's capacity simply differs from + * whatever the live capture had at record time). Blindly + * trusting it unconditionally is exactly what let a real + * 22-slot host take 44 concurrent single-slot jobs live + * (bug-1836's own reproduction: a 50-job array push, replayed + * against a snapshot whose worker only ever had 22 slots). + * Re-validate against LIVE resInUse/sharedInUse using the SAME + * admission-check functions the live picker/allocator path + * calls (sched_check_host_satisfy()/sched_check_shared_ + * satisfy(), resource.c) before taking ownership of anything -- + * a capacity miss is handled exactly like "no matching recorded + * decision" (Requirement 4.1/4.2, the `!found` branch below): + * leave the job unaccounted this round for the live picker/ + * allocator to decide instead of force-applying a stale claim, + * and do not consume/free this entry (it stays available to + * retry -- e.g. once an earlier job in this same round-robin + * loop order finishes and frees capacity, a still-later + * candidate in THIS pass may yet fit). */ + if (sim_reconciled_alloc_fits_capacity( + &(sim->core), entry->body.dispatch->alloc) + <= 0) { + char jobidStr[64]; + + fmt_jobid(jobidStr, sizeof(jobidStr), job->jobid); + WARN("%s recorded DISPATCH decision for jobid=%s no longer " + "fits current capacity -- leaving unaccounted for the " + "live picker/allocator instead of force-applying it", + __func__, jobidStr); + continue; + } + /* FOUND -- task 4.2 (scheduler-sim-live-allocation spec, * "Sim-Mode Reconciliation Narrowing" component): elect the job * through the same core-safe plugin_job_election() dispatch @@ -806,6 +1138,25 @@ static void sim_reconcile_unaccounted_jobs(xce_sched_sim_driver_t* sim, * or a terminal-guard violation. */ job->state = JOB_STAT_RUN; + /* bug-1836 fix: charge this job's usage into the LIVE + * resInUse/sharedInUse RIGHT NOW (mirroring sched_commit_ + * decision()'s own immediate charge for a freshly-drained + * live DISPATCH decision, scheduler.c) -- not deferred to + * this function's caller's subsequent sched_account_jobs() + * call. Without this, a LATER candidate in this SAME loop + * would have its own capacity check (above) evaluate against + * stale usage that does not yet reflect this EARLIER accepted + * candidate, defeating the whole point of the gate under + * exactly the multi-job-contention scenario it exists for. + * Undone in full (along with this function's own top-of- + * function sim_reconcile_toggle_prior_charges(..., 1) seed) by + * the closing sim_reconcile_toggle_prior_charges(..., 0) call + * at the bottom of this function -- see that call site's + * comment for why this never double-counts against the + * caller's own unchanged, still-authoritative + * sched_account_jobs() call. */ + sched_charge_account(&(sim->core), job->accounts); + /* bug-1759/bug-1779: mirrors sched_allocator()'s own ordering * (scheduler.c) as closely as this reconstruction allows -- * remove the job from the picker's candidate set now that it is @@ -899,6 +1250,15 @@ static void sim_reconcile_unaccounted_jobs(xce_sched_sim_driver_t* sim, } iter_destroy(&it); + + /* bug-1836 fix: undo this function's own transient peek + every + * incremental charge its capacity gate made above (see the matching + * sim_reconcile_toggle_prior_charges(..., 1) call at the top of this + * function for the full rationale) -- resInUse/sharedInUse must be + * exactly as this function found them on return, so the caller's own + * unchanged, immediately-following sched_account_jobs() call remains + * the sole, non-double-counting charge for this round. */ + sim_reconcile_toggle_prior_charges(&(sim->core), 0); } /* diff --git a/src/scheduler/testsuite/test_admin_request.c b/src/scheduler/testsuite/test_admin_request.c index 47ee501e..beb41108 100644 --- a/src/scheduler/testsuite/test_admin_request.c +++ b/src/scheduler/testsuite/test_admin_request.c @@ -196,6 +196,55 @@ int sched_account_jobs(xce_sched_core_t* core) return 0; } +/* bug-1836 (scheduler-driver-equivalence fix, Recorded-Decision + * Reconciliation capacity gate): link-satisfying stubs, mirroring this + * file's existing sched_account_jobs() no-op precedent immediately above -- + * sim_loop.c's sim_reconcile_unaccounted_jobs() now also calls + * sched_check_host_satisfy()/sched_check_shared_satisfy() (resource.c) to + * re-validate a recorded decision against live capacity, and + * sched_charge_account() (resource.c) to keep that check's own view of + * capacity up to date across multiple candidates in the same round. This + * file never exercises real resource contention (same rationale as the + * existing sched_account_jobs()/sched_ctx_get_uncomit_host() no-ops), so + * the satisfy checks trivially always succeed and the charge call is a + * no-op. */ +int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w, + xvec_t* uncomitted, xvec_idval_t* wanted, + xvec_idval_t* unmeet) +{ + (void)core; + (void)w; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted, + xvec_idval_t* wanted, xvec_idval_t* unmeet) +{ + (void)core; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_charge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + +int sched_uncharge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + + /* Link-satisfying stubs for recording_session.c's real functions -- see * this file's header comment for why recording_session.c itself is not * linked. Never actually invoked (no session is ever run). */ diff --git a/src/scheduler/testsuite/test_driver_decision_cb.c b/src/scheduler/testsuite/test_driver_decision_cb.c index 42e0b503..6e662bb4 100644 --- a/src/scheduler/testsuite/test_driver_decision_cb.c +++ b/src/scheduler/testsuite/test_driver_decision_cb.c @@ -160,6 +160,55 @@ int sched_account_jobs(xce_sched_core_t* core) return 0; } +/* bug-1836 (scheduler-driver-equivalence fix, Recorded-Decision + * Reconciliation capacity gate): link-satisfying stubs, mirroring this + * file's existing sched_account_jobs() no-op precedent immediately above -- + * sim_loop.c's sim_reconcile_unaccounted_jobs() now also calls + * sched_check_host_satisfy()/sched_check_shared_satisfy() (resource.c) to + * re-validate a recorded decision against live capacity, and + * sched_charge_account() (resource.c) to keep that check's own view of + * capacity up to date across multiple candidates in the same round. This + * file never exercises real resource contention (same rationale as the + * existing sched_account_jobs()/sched_ctx_get_uncomit_host() no-ops), so + * the satisfy checks trivially always succeed and the charge call is a + * no-op. */ +int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w, + xvec_t* uncomitted, xvec_idval_t* wanted, + xvec_idval_t* unmeet) +{ + (void)core; + (void)w; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted, + xvec_idval_t* wanted, xvec_idval_t* unmeet) +{ + (void)core; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_charge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + +int sched_uncharge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + + /* Link-satisfying stubs for recording.c/recording_events.c/ * recording_session.c's real functions -- as of scheduler-core spec task * 9.2, driver.c's (real, #included above) diff --git a/src/scheduler/testsuite/test_driver_session_ordering.c b/src/scheduler/testsuite/test_driver_session_ordering.c index fdea2686..d195d530 100644 --- a/src/scheduler/testsuite/test_driver_session_ordering.c +++ b/src/scheduler/testsuite/test_driver_session_ordering.c @@ -437,6 +437,55 @@ int sched_account_jobs(xce_sched_core_t* core) return 0; } +/* bug-1836 (scheduler-driver-equivalence fix, Recorded-Decision + * Reconciliation capacity gate): link-satisfying stubs, mirroring this + * file's existing sched_account_jobs() no-op precedent immediately above -- + * sim_loop.c's sim_reconcile_unaccounted_jobs() now also calls + * sched_check_host_satisfy()/sched_check_shared_satisfy() (resource.c) to + * re-validate a recorded decision against live capacity, and + * sched_charge_account() (resource.c) to keep that check's own view of + * capacity up to date across multiple candidates in the same round. This + * file never exercises real resource contention (same rationale as the + * existing sched_account_jobs()/sched_ctx_get_uncomit_host() no-ops), so + * the satisfy checks trivially always succeed and the charge call is a + * no-op. */ +int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w, + xvec_t* uncomitted, xvec_idval_t* wanted, + xvec_idval_t* unmeet) +{ + (void)core; + (void)w; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted, + xvec_idval_t* wanted, xvec_idval_t* unmeet) +{ + (void)core; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_charge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + +int sched_uncharge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + + /* ---------------------------------------------------------------------- */ /* Observer plugin: OLD begin_session_fn/end_session_fn hook category */ /* (plugin.h) -- genuinely registered into core->plugins and driven by */ diff --git a/src/scheduler/testsuite/test_driver_translate.c b/src/scheduler/testsuite/test_driver_translate.c index 2165049f..584dad93 100644 --- a/src/scheduler/testsuite/test_driver_translate.c +++ b/src/scheduler/testsuite/test_driver_translate.c @@ -246,6 +246,55 @@ int sched_account_jobs(xce_sched_core_t* core) return 0; } +/* bug-1836 (scheduler-driver-equivalence fix, Recorded-Decision + * Reconciliation capacity gate): link-satisfying stubs, mirroring this + * file's existing sched_account_jobs() no-op precedent immediately above -- + * sim_loop.c's sim_reconcile_unaccounted_jobs() now also calls + * sched_check_host_satisfy()/sched_check_shared_satisfy() (resource.c) to + * re-validate a recorded decision against live capacity, and + * sched_charge_account() (resource.c) to keep that check's own view of + * capacity up to date across multiple candidates in the same round. This + * file never exercises real resource contention (same rationale as the + * existing sched_account_jobs()/sched_ctx_get_uncomit_host() no-ops), so + * the satisfy checks trivially always succeed and the charge call is a + * no-op. */ +int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w, + xvec_t* uncomitted, xvec_idval_t* wanted, + xvec_idval_t* unmeet) +{ + (void)core; + (void)w; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted, + xvec_idval_t* wanted, xvec_idval_t* unmeet) +{ + (void)core; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_charge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + +int sched_uncharge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + + /* Real reimplementation of schedobj.c's sched_resolve_job_picker -- see * file header comment (same rationale as test_core_events.c's own copy). */ int sched_resolve_job_picker(xce_sched_core_t* core, xce_sched_plugin_t** out) diff --git a/src/scheduler/testsuite/test_maxtasks_wiring_sim_driver_parity.c b/src/scheduler/testsuite/test_maxtasks_wiring_sim_driver_parity.c index 162861b8..492dd178 100644 --- a/src/scheduler/testsuite/test_maxtasks_wiring_sim_driver_parity.c +++ b/src/scheduler/testsuite/test_maxtasks_wiring_sim_driver_parity.c @@ -256,6 +256,55 @@ int sched_account_jobs(xce_sched_core_t* core) return 0; } +/* bug-1836 (scheduler-driver-equivalence fix, Recorded-Decision + * Reconciliation capacity gate): link-satisfying stubs, mirroring this + * file's existing sched_account_jobs() no-op precedent immediately above -- + * sim_loop.c's sim_reconcile_unaccounted_jobs() now also calls + * sched_check_host_satisfy()/sched_check_shared_satisfy() (resource.c) to + * re-validate a recorded decision against live capacity, and + * sched_charge_account() (resource.c) to keep that check's own view of + * capacity up to date across multiple candidates in the same round. This + * file never exercises real resource contention (same rationale as the + * existing sched_account_jobs()/sched_ctx_get_uncomit_host() no-ops), so + * the satisfy checks trivially always succeed and the charge call is a + * no-op. */ +int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w, + xvec_t* uncomitted, xvec_idval_t* wanted, + xvec_idval_t* unmeet) +{ + (void)core; + (void)w; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted, + xvec_idval_t* wanted, xvec_idval_t* unmeet) +{ + (void)core; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_charge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + +int sched_uncharge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + + int sched_resolve_job_picker(xce_sched_core_t* core, xce_sched_plugin_t** out) { (void)core; diff --git a/src/scheduler/testsuite/test_record_lifecycle.c b/src/scheduler/testsuite/test_record_lifecycle.c index 540cef40..ae39b11f 100644 --- a/src/scheduler/testsuite/test_record_lifecycle.c +++ b/src/scheduler/testsuite/test_record_lifecycle.c @@ -165,6 +165,55 @@ int sched_account_jobs(xce_sched_core_t* core) return 0; } +/* bug-1836 (scheduler-driver-equivalence fix, Recorded-Decision + * Reconciliation capacity gate): link-satisfying stubs, mirroring this + * file's existing sched_account_jobs() no-op precedent immediately above -- + * sim_loop.c's sim_reconcile_unaccounted_jobs() now also calls + * sched_check_host_satisfy()/sched_check_shared_satisfy() (resource.c) to + * re-validate a recorded decision against live capacity, and + * sched_charge_account() (resource.c) to keep that check's own view of + * capacity up to date across multiple candidates in the same round. This + * file never exercises real resource contention (same rationale as the + * existing sched_account_jobs()/sched_ctx_get_uncomit_host() no-ops), so + * the satisfy checks trivially always succeed and the charge call is a + * no-op. */ +int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w, + xvec_t* uncomitted, xvec_idval_t* wanted, + xvec_idval_t* unmeet) +{ + (void)core; + (void)w; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted, + xvec_idval_t* wanted, xvec_idval_t* unmeet) +{ + (void)core; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_charge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + +int sched_uncharge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + + /* Real reimplementation of schedobj.c's sched_resolve_job_picker -- same * rationale/algorithm as test_driver_session_ordering.c's own copy * (schedobj.c is not linked). */ diff --git a/src/scheduler/testsuite/test_repl_breakpoints_inspection.c b/src/scheduler/testsuite/test_repl_breakpoints_inspection.c index ee0ec72e..4c1404ec 100644 --- a/src/scheduler/testsuite/test_repl_breakpoints_inspection.c +++ b/src/scheduler/testsuite/test_repl_breakpoints_inspection.c @@ -202,6 +202,55 @@ int sched_account_jobs(xce_sched_core_t* core) return 0; } +/* bug-1836 (scheduler-driver-equivalence fix, Recorded-Decision + * Reconciliation capacity gate): link-satisfying stubs, mirroring this + * file's existing sched_account_jobs() no-op precedent immediately above -- + * sim_loop.c's sim_reconcile_unaccounted_jobs() now also calls + * sched_check_host_satisfy()/sched_check_shared_satisfy() (resource.c) to + * re-validate a recorded decision against live capacity, and + * sched_charge_account() (resource.c) to keep that check's own view of + * capacity up to date across multiple candidates in the same round. This + * file never exercises real resource contention (same rationale as the + * existing sched_account_jobs()/sched_ctx_get_uncomit_host() no-ops), so + * the satisfy checks trivially always succeed and the charge call is a + * no-op. */ +int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w, + xvec_t* uncomitted, xvec_idval_t* wanted, + xvec_idval_t* unmeet) +{ + (void)core; + (void)w; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted, + xvec_idval_t* wanted, xvec_idval_t* unmeet) +{ + (void)core; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_charge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + +int sched_uncharge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + + /* Real reimplementation of schedobj.c's sched_resolve_job_picker -- same * copy test_repl_execution_control.c/test_sim_alt_events.c/etc. each carry * (schedobj.c is not linked here). */ diff --git a/src/scheduler/testsuite/test_repl_equivalence.c b/src/scheduler/testsuite/test_repl_equivalence.c index 933d1c86..cbe452fb 100644 --- a/src/scheduler/testsuite/test_repl_equivalence.c +++ b/src/scheduler/testsuite/test_repl_equivalence.c @@ -256,6 +256,55 @@ int sched_account_jobs(xce_sched_core_t* core) return 0; } +/* bug-1836 (scheduler-driver-equivalence fix, Recorded-Decision + * Reconciliation capacity gate): link-satisfying stubs, mirroring this + * file's existing sched_account_jobs() no-op precedent immediately above -- + * sim_loop.c's sim_reconcile_unaccounted_jobs() now also calls + * sched_check_host_satisfy()/sched_check_shared_satisfy() (resource.c) to + * re-validate a recorded decision against live capacity, and + * sched_charge_account() (resource.c) to keep that check's own view of + * capacity up to date across multiple candidates in the same round. This + * file never exercises real resource contention (same rationale as the + * existing sched_account_jobs()/sched_ctx_get_uncomit_host() no-ops), so + * the satisfy checks trivially always succeed and the charge call is a + * no-op. */ +int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w, + xvec_t* uncomitted, xvec_idval_t* wanted, + xvec_idval_t* unmeet) +{ + (void)core; + (void)w; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted, + xvec_idval_t* wanted, xvec_idval_t* unmeet) +{ + (void)core; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_charge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + +int sched_uncharge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + + /* Real reimplementation of schedobj.c's sched_resolve_job_picker -- same * copy every white-box test in this directory that does not link * schedobj.c already establishes (test_repl_execution_control.c, diff --git a/src/scheduler/testsuite/test_repl_event_echo.c b/src/scheduler/testsuite/test_repl_event_echo.c index e904ca07..74c3833e 100644 --- a/src/scheduler/testsuite/test_repl_event_echo.c +++ b/src/scheduler/testsuite/test_repl_event_echo.c @@ -219,6 +219,55 @@ int sched_account_jobs(xce_sched_core_t* core) return 0; } +/* bug-1836 (scheduler-driver-equivalence fix, Recorded-Decision + * Reconciliation capacity gate): link-satisfying stubs, mirroring this + * file's existing sched_account_jobs() no-op precedent immediately above -- + * sim_loop.c's sim_reconcile_unaccounted_jobs() now also calls + * sched_check_host_satisfy()/sched_check_shared_satisfy() (resource.c) to + * re-validate a recorded decision against live capacity, and + * sched_charge_account() (resource.c) to keep that check's own view of + * capacity up to date across multiple candidates in the same round. This + * file never exercises real resource contention (same rationale as the + * existing sched_account_jobs()/sched_ctx_get_uncomit_host() no-ops), so + * the satisfy checks trivially always succeed and the charge call is a + * no-op. */ +int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w, + xvec_t* uncomitted, xvec_idval_t* wanted, + xvec_idval_t* unmeet) +{ + (void)core; + (void)w; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted, + xvec_idval_t* wanted, xvec_idval_t* unmeet) +{ + (void)core; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_charge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + +int sched_uncharge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + + /* Real reimplementation of schedobj.c's sched_resolve_job_picker -- same * copy test_sim_alt_events.c/test_sim_verify.c/test_sim_event_loop.c/ * test_repl_execution_control.c/test_repl_round_rollup.c each carry diff --git a/src/scheduler/testsuite/test_repl_execution_control.c b/src/scheduler/testsuite/test_repl_execution_control.c index 7909cf0e..3640300a 100644 --- a/src/scheduler/testsuite/test_repl_execution_control.c +++ b/src/scheduler/testsuite/test_repl_execution_control.c @@ -216,6 +216,55 @@ int sched_account_jobs(xce_sched_core_t* core) return 0; } +/* bug-1836 (scheduler-driver-equivalence fix, Recorded-Decision + * Reconciliation capacity gate): link-satisfying stubs, mirroring this + * file's existing sched_account_jobs() no-op precedent immediately above -- + * sim_loop.c's sim_reconcile_unaccounted_jobs() now also calls + * sched_check_host_satisfy()/sched_check_shared_satisfy() (resource.c) to + * re-validate a recorded decision against live capacity, and + * sched_charge_account() (resource.c) to keep that check's own view of + * capacity up to date across multiple candidates in the same round. This + * file never exercises real resource contention (same rationale as the + * existing sched_account_jobs()/sched_ctx_get_uncomit_host() no-ops), so + * the satisfy checks trivially always succeed and the charge call is a + * no-op. */ +int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w, + xvec_t* uncomitted, xvec_idval_t* wanted, + xvec_idval_t* unmeet) +{ + (void)core; + (void)w; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted, + xvec_idval_t* wanted, xvec_idval_t* unmeet) +{ + (void)core; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_charge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + +int sched_uncharge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + + /* Real reimplementation of schedobj.c's sched_resolve_job_picker -- same * copy test_sim_alt_events.c/test_sim_verify.c/test_sim_event_loop.c each * carry (schedobj.c is not linked here: it would need plugin_register(), diff --git a/src/scheduler/testsuite/test_repl_help.c b/src/scheduler/testsuite/test_repl_help.c index b5217c57..910c0ffe 100644 --- a/src/scheduler/testsuite/test_repl_help.c +++ b/src/scheduler/testsuite/test_repl_help.c @@ -213,6 +213,55 @@ int sched_account_jobs(xce_sched_core_t* core) return 0; } +/* bug-1836 (scheduler-driver-equivalence fix, Recorded-Decision + * Reconciliation capacity gate): link-satisfying stubs, mirroring this + * file's existing sched_account_jobs() no-op precedent immediately above -- + * sim_loop.c's sim_reconcile_unaccounted_jobs() now also calls + * sched_check_host_satisfy()/sched_check_shared_satisfy() (resource.c) to + * re-validate a recorded decision against live capacity, and + * sched_charge_account() (resource.c) to keep that check's own view of + * capacity up to date across multiple candidates in the same round. This + * file never exercises real resource contention (same rationale as the + * existing sched_account_jobs()/sched_ctx_get_uncomit_host() no-ops), so + * the satisfy checks trivially always succeed and the charge call is a + * no-op. */ +int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w, + xvec_t* uncomitted, xvec_idval_t* wanted, + xvec_idval_t* unmeet) +{ + (void)core; + (void)w; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted, + xvec_idval_t* wanted, xvec_idval_t* unmeet) +{ + (void)core; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_charge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + +int sched_uncharge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + + /* Real reimplementation of schedobj.c's sched_resolve_job_picker -- same * copy test_sim_alt_events.c/test_sim_verify.c/test_repl_execution_control.c/ * test_repl_event_echo.c each carry (schedobj.c is not linked here: it would diff --git a/src/scheduler/testsuite/test_repl_limit_table.c b/src/scheduler/testsuite/test_repl_limit_table.c index 591cfe17..a3e95438 100644 --- a/src/scheduler/testsuite/test_repl_limit_table.c +++ b/src/scheduler/testsuite/test_repl_limit_table.c @@ -136,6 +136,55 @@ int sched_account_jobs(xce_sched_core_t* core) return 0; } +/* bug-1836 (scheduler-driver-equivalence fix, Recorded-Decision + * Reconciliation capacity gate): link-satisfying stubs, mirroring this + * file's existing sched_account_jobs() no-op precedent immediately above -- + * sim_loop.c's sim_reconcile_unaccounted_jobs() now also calls + * sched_check_host_satisfy()/sched_check_shared_satisfy() (resource.c) to + * re-validate a recorded decision against live capacity, and + * sched_charge_account() (resource.c) to keep that check's own view of + * capacity up to date across multiple candidates in the same round. This + * file never exercises real resource contention (same rationale as the + * existing sched_account_jobs()/sched_ctx_get_uncomit_host() no-ops), so + * the satisfy checks trivially always succeed and the charge call is a + * no-op. */ +int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w, + xvec_t* uncomitted, xvec_idval_t* wanted, + xvec_idval_t* unmeet) +{ + (void)core; + (void)w; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted, + xvec_idval_t* wanted, xvec_idval_t* unmeet) +{ + (void)core; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_charge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + +int sched_uncharge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + + /* Real reimplementation of schedobj.c's sched_resolve_job_picker -- same * copy test_repl_resource_table.c/test_repl_execution_control.c/ * test_sim_alt_events.c/etc. each carry (schedobj.c is not linked here). */ diff --git a/src/scheduler/testsuite/test_repl_prompt_context.c b/src/scheduler/testsuite/test_repl_prompt_context.c index fa3a067c..3c6a413f 100644 --- a/src/scheduler/testsuite/test_repl_prompt_context.c +++ b/src/scheduler/testsuite/test_repl_prompt_context.c @@ -208,6 +208,55 @@ int sched_account_jobs(xce_sched_core_t* core) return 0; } +/* bug-1836 (scheduler-driver-equivalence fix, Recorded-Decision + * Reconciliation capacity gate): link-satisfying stubs, mirroring this + * file's existing sched_account_jobs() no-op precedent immediately above -- + * sim_loop.c's sim_reconcile_unaccounted_jobs() now also calls + * sched_check_host_satisfy()/sched_check_shared_satisfy() (resource.c) to + * re-validate a recorded decision against live capacity, and + * sched_charge_account() (resource.c) to keep that check's own view of + * capacity up to date across multiple candidates in the same round. This + * file never exercises real resource contention (same rationale as the + * existing sched_account_jobs()/sched_ctx_get_uncomit_host() no-ops), so + * the satisfy checks trivially always succeed and the charge call is a + * no-op. */ +int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w, + xvec_t* uncomitted, xvec_idval_t* wanted, + xvec_idval_t* unmeet) +{ + (void)core; + (void)w; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted, + xvec_idval_t* wanted, xvec_idval_t* unmeet) +{ + (void)core; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_charge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + +int sched_uncharge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + + int sched_resolve_job_picker(xce_sched_core_t* core, xce_sched_plugin_t** out) { xce_sched_plugin_t* plugin = NULL; diff --git a/src/scheduler/testsuite/test_repl_resource_table.c b/src/scheduler/testsuite/test_repl_resource_table.c index 8348a91b..bf7cb6d9 100644 --- a/src/scheduler/testsuite/test_repl_resource_table.c +++ b/src/scheduler/testsuite/test_repl_resource_table.c @@ -182,6 +182,55 @@ int sched_account_jobs(xce_sched_core_t* core) return 0; } +/* bug-1836 (scheduler-driver-equivalence fix, Recorded-Decision + * Reconciliation capacity gate): link-satisfying stubs, mirroring this + * file's existing sched_account_jobs() no-op precedent immediately above -- + * sim_loop.c's sim_reconcile_unaccounted_jobs() now also calls + * sched_check_host_satisfy()/sched_check_shared_satisfy() (resource.c) to + * re-validate a recorded decision against live capacity, and + * sched_charge_account() (resource.c) to keep that check's own view of + * capacity up to date across multiple candidates in the same round. This + * file never exercises real resource contention (same rationale as the + * existing sched_account_jobs()/sched_ctx_get_uncomit_host() no-ops), so + * the satisfy checks trivially always succeed and the charge call is a + * no-op. */ +int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w, + xvec_t* uncomitted, xvec_idval_t* wanted, + xvec_idval_t* unmeet) +{ + (void)core; + (void)w; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted, + xvec_idval_t* wanted, xvec_idval_t* unmeet) +{ + (void)core; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_charge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + +int sched_uncharge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + + /* Real reimplementation of schedobj.c's sched_resolve_job_picker -- same * copy test_repl_execution_control.c/test_sim_alt_events.c/etc. each carry * (schedobj.c is not linked here). */ diff --git a/src/scheduler/testsuite/test_repl_round_rollup.c b/src/scheduler/testsuite/test_repl_round_rollup.c index ed703a35..d513f260 100644 --- a/src/scheduler/testsuite/test_repl_round_rollup.c +++ b/src/scheduler/testsuite/test_repl_round_rollup.c @@ -178,6 +178,55 @@ int sched_account_jobs(xce_sched_core_t* core) return 0; } +/* bug-1836 (scheduler-driver-equivalence fix, Recorded-Decision + * Reconciliation capacity gate): link-satisfying stubs, mirroring this + * file's existing sched_account_jobs() no-op precedent immediately above -- + * sim_loop.c's sim_reconcile_unaccounted_jobs() now also calls + * sched_check_host_satisfy()/sched_check_shared_satisfy() (resource.c) to + * re-validate a recorded decision against live capacity, and + * sched_charge_account() (resource.c) to keep that check's own view of + * capacity up to date across multiple candidates in the same round. This + * file never exercises real resource contention (same rationale as the + * existing sched_account_jobs()/sched_ctx_get_uncomit_host() no-ops), so + * the satisfy checks trivially always succeed and the charge call is a + * no-op. */ +int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w, + xvec_t* uncomitted, xvec_idval_t* wanted, + xvec_idval_t* unmeet) +{ + (void)core; + (void)w; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted, + xvec_idval_t* wanted, xvec_idval_t* unmeet) +{ + (void)core; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_charge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + +int sched_uncharge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + + /* Real reimplementation of schedobj.c's sched_resolve_job_picker -- same * copy test_sim_alt_events.c/test_sim_verify.c/test_sim_event_loop.c/ * test_repl_execution_control.c each carry (schedobj.c is not linked here: diff --git a/src/scheduler/testsuite/test_repl_snapshot_params.c b/src/scheduler/testsuite/test_repl_snapshot_params.c index 104a35cc..e725aaa6 100644 --- a/src/scheduler/testsuite/test_repl_snapshot_params.c +++ b/src/scheduler/testsuite/test_repl_snapshot_params.c @@ -191,6 +191,55 @@ int sched_account_jobs(xce_sched_core_t* core) return 0; } +/* bug-1836 (scheduler-driver-equivalence fix, Recorded-Decision + * Reconciliation capacity gate): link-satisfying stubs, mirroring this + * file's existing sched_account_jobs() no-op precedent immediately above -- + * sim_loop.c's sim_reconcile_unaccounted_jobs() now also calls + * sched_check_host_satisfy()/sched_check_shared_satisfy() (resource.c) to + * re-validate a recorded decision against live capacity, and + * sched_charge_account() (resource.c) to keep that check's own view of + * capacity up to date across multiple candidates in the same round. This + * file never exercises real resource contention (same rationale as the + * existing sched_account_jobs()/sched_ctx_get_uncomit_host() no-ops), so + * the satisfy checks trivially always succeed and the charge call is a + * no-op. */ +int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w, + xvec_t* uncomitted, xvec_idval_t* wanted, + xvec_idval_t* unmeet) +{ + (void)core; + (void)w; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted, + xvec_idval_t* wanted, xvec_idval_t* unmeet) +{ + (void)core; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_charge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + +int sched_uncharge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + + /* Real reimplementation of schedobj.c's sched_resolve_job_picker -- same * copy every sim-family white-box test carries (schedobj.c is not linked * here: it would need plugin_register(), which requires a genuine diff --git a/src/scheduler/testsuite/test_repl_step_verbosity.c b/src/scheduler/testsuite/test_repl_step_verbosity.c index 9ef00629..b68883a9 100644 --- a/src/scheduler/testsuite/test_repl_step_verbosity.c +++ b/src/scheduler/testsuite/test_repl_step_verbosity.c @@ -232,6 +232,55 @@ int sched_account_jobs(xce_sched_core_t* core) return 0; } +/* bug-1836 (scheduler-driver-equivalence fix, Recorded-Decision + * Reconciliation capacity gate): link-satisfying stubs, mirroring this + * file's existing sched_account_jobs() no-op precedent immediately above -- + * sim_loop.c's sim_reconcile_unaccounted_jobs() now also calls + * sched_check_host_satisfy()/sched_check_shared_satisfy() (resource.c) to + * re-validate a recorded decision against live capacity, and + * sched_charge_account() (resource.c) to keep that check's own view of + * capacity up to date across multiple candidates in the same round. This + * file never exercises real resource contention (same rationale as the + * existing sched_account_jobs()/sched_ctx_get_uncomit_host() no-ops), so + * the satisfy checks trivially always succeed and the charge call is a + * no-op. */ +int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w, + xvec_t* uncomitted, xvec_idval_t* wanted, + xvec_idval_t* unmeet) +{ + (void)core; + (void)w; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted, + xvec_idval_t* wanted, xvec_idval_t* unmeet) +{ + (void)core; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_charge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + +int sched_uncharge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + + /* Real reimplementation of schedobj.c's sched_resolve_job_picker -- same * copy test_repl_event_echo.c/test_repl_round_rollup.c/ * test_repl_execution_control.c/test_sim_alt_events.c/test_sim_verify.c/ diff --git a/src/scheduler/testsuite/test_resval_ownership.c b/src/scheduler/testsuite/test_resval_ownership.c index fd6c2af1..7ec54b69 100644 --- a/src/scheduler/testsuite/test_resval_ownership.c +++ b/src/scheduler/testsuite/test_resval_ownership.c @@ -251,6 +251,55 @@ int sched_account_jobs(xce_sched_core_t* core) return 0; } +/* bug-1836 (scheduler-driver-equivalence fix, Recorded-Decision + * Reconciliation capacity gate): link-satisfying stubs, mirroring this + * file's existing sched_account_jobs() no-op precedent immediately above -- + * sim_loop.c's sim_reconcile_unaccounted_jobs() now also calls + * sched_check_host_satisfy()/sched_check_shared_satisfy() (resource.c) to + * re-validate a recorded decision against live capacity, and + * sched_charge_account() (resource.c) to keep that check's own view of + * capacity up to date across multiple candidates in the same round. This + * file never exercises real resource contention (same rationale as the + * existing sched_account_jobs()/sched_ctx_get_uncomit_host() no-ops), so + * the satisfy checks trivially always succeed and the charge call is a + * no-op. */ +int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w, + xvec_t* uncomitted, xvec_idval_t* wanted, + xvec_idval_t* unmeet) +{ + (void)core; + (void)w; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted, + xvec_idval_t* wanted, xvec_idval_t* unmeet) +{ + (void)core; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_charge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + +int sched_uncharge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + + /* Real reimplementation of schedobj.c's sched_resolve_job_picker -- copied * verbatim from test_driver_translate.c's own copy (same rationale: no * picker plugin is registered in this file's fixture, so this always diff --git a/src/scheduler/testsuite/test_sim_alt_events.c b/src/scheduler/testsuite/test_sim_alt_events.c index 10c49c73..aee8273b 100644 --- a/src/scheduler/testsuite/test_sim_alt_events.c +++ b/src/scheduler/testsuite/test_sim_alt_events.c @@ -210,6 +210,55 @@ int sched_account_jobs(xce_sched_core_t* core) return 0; } +/* bug-1836 (scheduler-driver-equivalence fix, Recorded-Decision + * Reconciliation capacity gate): link-satisfying stubs, mirroring this + * file's existing sched_account_jobs() no-op precedent immediately above -- + * sim_loop.c's sim_reconcile_unaccounted_jobs() now also calls + * sched_check_host_satisfy()/sched_check_shared_satisfy() (resource.c) to + * re-validate a recorded decision against live capacity, and + * sched_charge_account() (resource.c) to keep that check's own view of + * capacity up to date across multiple candidates in the same round. This + * file never exercises real resource contention (same rationale as the + * existing sched_account_jobs()/sched_ctx_get_uncomit_host() no-ops), so + * the satisfy checks trivially always succeed and the charge call is a + * no-op. */ +int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w, + xvec_t* uncomitted, xvec_idval_t* wanted, + xvec_idval_t* unmeet) +{ + (void)core; + (void)w; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted, + xvec_idval_t* wanted, xvec_idval_t* unmeet) +{ + (void)core; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_charge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + +int sched_uncharge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + + /* Real reimplementation of schedobj.c's sched_resolve_job_picker -- same * copy test_sim_verify.c/test_sim_event_loop.c/test_sim_startup.c each * carry (schedobj.c is not linked here: it would need plugin_register(), diff --git a/src/scheduler/testsuite/test_sim_contention_parity.c b/src/scheduler/testsuite/test_sim_contention_parity.c index eac3bb66..64d594d2 100644 --- a/src/scheduler/testsuite/test_sim_contention_parity.c +++ b/src/scheduler/testsuite/test_sim_contention_parity.c @@ -452,6 +452,7 @@ int sched_account_jobs(xce_sched_core_t* core) return rc; } + /* scheduler-sim-live-allocation spec, task 4.1: sim_loop.c's real * sched_sim_driver_run() (white-box #include'd into this TU above) now * spawns sched_job_picker()/sched_allocator() (sched_election.c) directly @@ -1815,6 +1816,31 @@ static int check_host_satisfy_want(xce_sched_sim_driver_t* sim, float want) return rc; } +/* + * CORRECTED (2026-09-21, bug-1838, scheduler-driver-equivalence): this + * test's entire fixture (write_multiround_contention_rounds() above) is + * built around jobs A/B/C pushed fresh PEND at round 1 with a matching + * RECORDED DISPATCH decision for A and C -- relying on + * sim_reconcile_unaccounted_jobs() to bridge them, exactly the mechanism + * bug-1838 narrowed back to JOB_IS_RUN-only. Compounding this, THIS file's + * own sched_allocator() override (used whenever sched_sim_driver_run() is + * called, as it is here) is a deliberate PEND-only stub (see that + * function's own header comment) -- "genuine resource-based dispatch... is + * proved separately by test_sim_live_allocation.c". With reconciliation no + * longer touching PEND jobs, and this file's own live path structurally + * unable to dispatch anything, jobs A/B/C now stay PEND for the entire + * fixture, in every round -- resInUse never leaves 0.0, and every + * assertion this test used to make (capacity SATURATED by round 1, stays + * saturated through rounds 2-5) is now backwards. Inverted below to lock + * in the new, correct behavior instead of deleting the coverage outright. + * + * The genuine "capacity is correctly tracked across a real dispatch-then- + * later-confirmation gap" property is now proved via test_sim_live_ + * allocation.c's test_sim_verify_no_false_mismatch_after_live_dispatch_ + * and_later_confirmation, which links the real scheduler_srcs and can + * genuinely dispatch a job -- something this file's PEND-only stub never + * could, reconciliation or not. + */ static void test_contention_capacity_check_correct_across_multi_round_dispatch_confirm_gap( void) @@ -1827,48 +1853,42 @@ test_contention_capacity_check_correct_across_multi_round_dispatch_confirm_gap( g_capturing = 0; g_round_idx = 0; - /* ---- Checkpoint 1 (REVISED 2026-09-11): right after round 1 -- A and - * C are captured AND already genuinely RUN (synchronous capture-time - * transition, mirroring sched_commit_decision()/bug-1624). Capacity - * must already look FULLY SATURATED: a hypothetical job wanting the - * entire WORKER_CAPACITY must be DENIED, since both A and C are - * already counted from this round onward (Requirement 1.2/1.3, at the - * capacity-check OUTCOME level, not just a raw resInUse read). Under - * the PRE-2026-09-11 design this same check was granted (1) -- the - * inversion itself is the proof the revision landed. ---- */ + /* ---- Checkpoint 1: right after round 1 -- A and C are pushed PEND + * with a matching recorded DISPATCH, but bug-1838's RUN-only + * reconciliation trigger does not touch them (still PEND), and this + * file's own PEND-only sched_allocator() stub cannot dispatch them + * either. Capacity must therefore be WIDE OPEN: even a job wanting + * the entire WORKER_CAPACITY must be GRANTED, since nothing has + * genuinely been charged. ---- */ run_multiround_contention_fixture(&sim1, 1, "_mr1"); rc = check_host_satisfy_want(&sim1, WORKER_CAPACITY); TEST_ASSERT_EQUAL_INT_MESSAGE( - 0, rc, - "round 1 (2026-09-11 revision): A and C are captured AND already " - "genuinely RUN, synchronously with capture -- capacity must " - "already look FULLY SATURATED (a job wanting the entire " - "WORKER_CAPACITY must be DENIED), not deferred until a later " - "confirmation round (Requirement 1.2/1.3)"); + 1, rc, + "bug-1838: A and C stay PEND (neither reconciliation -- now " + "RUN-only -- nor this file's PEND-only live stub can dispatch " + "them), so resInUse stays 0.0 and a job wanting the entire " + "WORKER_CAPACITY must be GRANTED, not denied"); - /* ---- Checkpoint 2 (REVISED 2026-09-11): right after round 4 -- job - * A's separately-recorded confirmation event has now ALSO replayed, - * but changes nothing: A and C were both already RUN since round 1. - * Capacity must STILL be fully saturated -- even JOBC_WANT (2.0), - * previously grantable under the deferred design, must now be denied, - * since C itself has been counted since round 1 too. ---- */ + /* ---- Checkpoint 2: right after round 4 -- job A's separately- + * recorded confirmation event has replayed, but A was never captured + * in the first place (still PEND), so this is a WARN-and-skip no-op + * (core_events.c logs an unknown-jobid warning for a job it has no + * prior RUN record to reconcile against, per that handler's existing + * contract) -- it does not fabricate a charge. Capacity remains wide + * open. ---- */ run_multiround_contention_fixture(&sim4, 4, "_mr4"); rc = check_host_satisfy_want(&sim4, JOBC_WANT); TEST_ASSERT_EQUAL_INT_MESSAGE( - 0, rc, - "round 4 (2026-09-11 revision): both A and C have been genuinely " - "RUN and fully charged since round 1 -- even JOBC_WANT (2.0) must " - "now be denied, since there is no remaining headroom"); + 1, rc, "bug-1838: A/C are still PEND at round 4 -- JOBC_WANT (2.0) " + "must be granted against a fully-open worker"); rc = check_host_satisfy_want(&sim4, JOBB_WANT); TEST_ASSERT_EQUAL_INT_MESSAGE( - 0, rc, - "round 4: JOBB_WANT (5.0) must also be denied -- capacity has been " - "fully saturated by A+C since round 1"); + 1, rc, "bug-1838: JOBB_WANT (5.0) must likewise be granted -- " + "nothing has ever been charged against this worker"); /* ---- Checkpoint 3: the full 5-round replay -- per-round resInUse - * snapshots (Requirement 1.1-1.4 across EVERY round, not just - * endpoints) plus a final capacity check proving FULL saturation once - * both jobs are genuinely RUN. ---- */ + * snapshots proving the round-1-through-5 gap stays permanently + * zero-charged, not just at the two checkpoints above. ---- */ g_capturing = 1; g_round_idx = 0; run_multiround_contention_fixture(&sim5, 5, "_mr5"); @@ -1877,10 +1897,7 @@ test_contention_capacity_check_correct_across_multi_round_dispatch_confirm_gap( /* 6, not 5: scheduler-snapshot-job-allocation task 4.6's extra * snapshot-load-time sched_account_jobs() call (see the sibling test * above for the full rationale) lands at g_round_resinuse[0], pushing - * every round's own snapshot to index+1 below. This fixture's initial - * snapshot also has "worker-a" registered but no RUN-state jobs yet - * (same fixture_init_seed() as the sibling test), so index 0 reads - * 0.0. */ + * every round's own snapshot to index+1 below. */ TEST_ASSERT_EQUAL_INT_MESSAGE( 6, g_round_idx, "sched_account_jobs() must have been called exactly once per " @@ -1889,48 +1906,36 @@ test_contention_capacity_check_correct_across_multi_round_dispatch_confirm_gap( TEST_ASSERT_EQUAL_FLOAT_MESSAGE( 0.0f, g_round_resinuse[0], - "snapshot load: no RUN-state jobs exist yet at the replay's own " - "starting snapshot -- resInUse must read 0.0 (task 4.6's own " - "load-time call, not a replayed round)"); + "snapshot load: no RUN-state jobs exist yet -- resInUse reads 0.0"); TEST_ASSERT_EQUAL_FLOAT_MESSAGE( - WORKER_CAPACITY, g_round_resinuse[1], - "round 1 (2026-09-11 revision): A(8.0) and C(2.0) are BOTH " - "captured and synchronously transitioned to RUN this round -- " - "resInUse must already reflect the FULL combined 10.0, not " - "deferred to a later confirmation round (Requirement 1.2/1.3)"); + 0.0f, g_round_resinuse[1], + "bug-1838: round 1 -- A and C are pushed PEND with a matching " + "recorded DISPATCH, but RUN-only reconciliation ignores them and " + "this file's live stub is PEND-only -- resInUse stays 0.0, not " + "the old 10.0 (WORKER_CAPACITY)"); TEST_ASSERT_EQUAL_FLOAT_MESSAGE( - WORKER_CAPACITY, g_round_resinuse[2], - "round 2: idle filler round -- the round-1 charge must persist " - "unchanged across an EXTRA round, not drift or reset"); + 0.0f, g_round_resinuse[2], + "round 2: idle filler round -- still 0.0, nothing to charge"); TEST_ASSERT_EQUAL_FLOAT_MESSAGE( - WORKER_CAPACITY, g_round_resinuse[3], - "round 3: a SECOND idle filler round -- still the full 10.0, " - "widening the gap before A's now-redundant confirmation below to " - "genuinely several rounds"); + 0.0f, g_round_resinuse[3], + "round 3: a second idle filler round -- still 0.0"); TEST_ASSERT_EQUAL_FLOAT_MESSAGE( - WORKER_CAPACITY, g_round_resinuse[4], - "round 4: job A's separately-recorded JOB_STATE_CHANGE-to-RUN " - "event replays, 3 rounds after A's own round-1 DISPATCH -- a safe " - "no-op re-affirmation (A has been RUN and charged since round 1); " - "resInUse must remain exactly 10.0, not drop, not double-charge"); + 0.0f, g_round_resinuse[4], + "bug-1838: round 4 -- job A's separately-recorded JOB_STATE_" + "CHANGE-to-RUN event replays, but A was never captured (still " + "PEND, no job->accounts) -- core_events.c's own unknown-jobid " + "warn-and-skip path applies; no charge is fabricated"); TEST_ASSERT_EQUAL_FLOAT_MESSAGE( - WORKER_CAPACITY, g_round_resinuse[5], - "round 5: job C's separately-recorded JOB_STATE_CHANGE-to-RUN " - "event replays, 4 rounds after C's own round-1 DISPATCH (a " - "DIFFERENT separation than A's 3 rounds, proving this isn't a " - "single-job coincidence) -- also a safe no-op; resInUse must " - "remain exactly 10.0, not double-charge to 20.0"); + 0.0f, g_round_resinuse[5], + "bug-1838: round 5 -- job C's separately-recorded confirmation " + "event likewise finds C never captured -- still 0.0, not 10.0"); rc = check_host_satisfy_want(&sim5, JOBD_WANT); TEST_ASSERT_EQUAL_INT_MESSAGE( - 0, rc, - "round 5: capacity is fully saturated by A+C (10.0 of 10.0, since " - "round 1) -- even the smallest positive want (JOBD_WANT) must be " - "denied. Under the PRE-task-1.1 reconciliation gate, neither A nor " - "C would EVER have been captured (their DISPATCH and RUN-" - "confirmation rounds never coincide), so resInUse would read a " - "PERMANENTLY-ZERO 0.0 here and this check would WRONGLY grant " - "admission"); + 1, rc, + "bug-1838: capacity remains wide open through round 5 -- even " + "JOBD_WANT must be granted, since A/B/C were never genuinely " + "dispatched in this PEND-only-stub environment"); printf("test_contention_capacity_check_correct_across_multi_round_" "dispatch_confirm_gap: PASS\n"); diff --git a/src/scheduler/testsuite/test_sim_e2e_original_motivation.c b/src/scheduler/testsuite/test_sim_e2e_original_motivation.c index e6eb0a5d..a65eef7e 100644 --- a/src/scheduler/testsuite/test_sim_e2e_original_motivation.c +++ b/src/scheduler/testsuite/test_sim_e2e_original_motivation.c @@ -404,6 +404,7 @@ int sched_account_jobs(xce_sched_core_t* core) return rc; } + /* scheduler-sim-live-allocation spec, task 4.1: sim_loop.c's real * sched_sim_driver_run() (white-box #include'd into this TU above) now * spawns sched_job_picker()/sched_allocator() (sched_election.c) directly diff --git a/src/scheduler/testsuite/test_sim_equivalence.c b/src/scheduler/testsuite/test_sim_equivalence.c index 281d9998..bd363afe 100644 --- a/src/scheduler/testsuite/test_sim_equivalence.c +++ b/src/scheduler/testsuite/test_sim_equivalence.c @@ -315,6 +315,55 @@ int sched_account_jobs(xce_sched_core_t* core) return 0; } +/* bug-1836 (scheduler-driver-equivalence fix, Recorded-Decision + * Reconciliation capacity gate): link-satisfying stubs, mirroring this + * file's existing sched_account_jobs() no-op precedent immediately above -- + * sim_loop.c's sim_reconcile_unaccounted_jobs() now also calls + * sched_check_host_satisfy()/sched_check_shared_satisfy() (resource.c) to + * re-validate a recorded decision against live capacity, and + * sched_charge_account() (resource.c) to keep that check's own view of + * capacity up to date across multiple candidates in the same round. This + * file never exercises real resource contention (same rationale as the + * existing sched_account_jobs()/sched_ctx_get_uncomit_host() no-ops), so + * the satisfy checks trivially always succeed and the charge call is a + * no-op. */ +int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w, + xvec_t* uncomitted, xvec_idval_t* wanted, + xvec_idval_t* unmeet) +{ + (void)core; + (void)w; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted, + xvec_idval_t* wanted, xvec_idval_t* unmeet) +{ + (void)core; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_charge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + +int sched_uncharge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + + /* xce_sched_driver_on_decision()'s (driver.c) publish-side linker * dependencies -- never actually invoked at runtime: this test never * registers a decision callback (xce_core_set_decision_cb() is never diff --git a/src/scheduler/testsuite/test_sim_event_loop.c b/src/scheduler/testsuite/test_sim_event_loop.c index 99b08620..36e4913b 100644 --- a/src/scheduler/testsuite/test_sim_event_loop.c +++ b/src/scheduler/testsuite/test_sim_event_loop.c @@ -231,6 +231,55 @@ int sched_account_jobs(xce_sched_core_t* core) return 0; } +/* bug-1836 (scheduler-driver-equivalence fix, Recorded-Decision + * Reconciliation capacity gate): link-satisfying stubs, mirroring this + * file's existing sched_account_jobs() no-op precedent immediately above -- + * sim_loop.c's sim_reconcile_unaccounted_jobs() now also calls + * sched_check_host_satisfy()/sched_check_shared_satisfy() (resource.c) to + * re-validate a recorded decision against live capacity, and + * sched_charge_account() (resource.c) to keep that check's own view of + * capacity up to date across multiple candidates in the same round. This + * file never exercises real resource contention (same rationale as the + * existing sched_account_jobs()/sched_ctx_get_uncomit_host() no-ops), so + * the satisfy checks trivially always succeed and the charge call is a + * no-op. */ +int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w, + xvec_t* uncomitted, xvec_idval_t* wanted, + xvec_idval_t* unmeet) +{ + (void)core; + (void)w; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted, + xvec_idval_t* wanted, xvec_idval_t* unmeet) +{ + (void)core; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_charge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + +int sched_uncharge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + + /* Real reimplementation of schedobj.c's sched_resolve_job_picker -- same * rationale/algorithm as test_sim_startup.c's/test_core_full_cycle.c's own * copies (schedobj.c is not linked: it would need plugin_register(), which diff --git a/src/scheduler/testsuite/test_sim_live_allocation.c b/src/scheduler/testsuite/test_sim_live_allocation.c index 27b55759..b73438ba 100644 --- a/src/scheduler/testsuite/test_sim_live_allocation.c +++ b/src/scheduler/testsuite/test_sim_live_allocation.c @@ -113,6 +113,7 @@ #include "resobj.h" #include "scheduler.h" #include "sim.h" +#include "xcealloc.h" #include "xstring.h" /* write_capnp_message (recording.c) has no public header declaration -- @@ -379,6 +380,95 @@ static void append_end_of_stream_marker(FILE* f) capn_free(&c); } +/* bug-1838 regression: same jobStateChange wire shape test_sim_contention_ + * parity.c's own append_job_state_change_event() establishes. */ +static void append_jobstatechange_event(FILE* f, uint64_t ts, uint64_t jobid, + uint32_t old_state, + uint32_t new_state) +{ + job_state_change_event_t jsc; + replay_event_t src; + struct capn c; + struct capn_segment* cs; + ReplayEvent_ptr p; + + memset(&jsc, 0, sizeof(jsc)); + jsc.jobid = jobid; + jsc.old_state = old_state; + jsc.new_state = new_state; + jsc.exit_code = 0; + + memset(&src, 0, sizeof(src)); + src.ts = ts; + src.body_which = ReplayEvent_body_jobStateChange; + src.body.job_state_change = &jsc; + + capn_init_malloc(&c); + cs = capn_root(&c).seg; + encode_ReplayEvent_ptr(cs, &p, &src); + capn_setp(capn_root(&c), 0, p.p); + + TEST_ASSERT_EQUAL_INT(0, write_capnp_message(f, &c)); + capn_free(&c); +} + +/* run_sim_and_capture_stdout - same dup2(STDOUT_FILENO)-redirect technique + * test_sim_verify.c's own copy establishes, to capture sim_log_mismatch()'s + * ERROR()-logged "MISMATCH" text for a --sim-verify assertion. Takes the + * caller's own static xce_sched_sim_driver_t* (never a stack-local one -- + * see this file's own header comment, "bug-1598/1632/1763 fixture-safety + * requirement", for why: sched_sim_driver_run() spawns two real, + * never-joined threads that keep dereferencing this pointer for the rest + * of the process's life). */ +static char* run_sim_and_capture_stdout(xce_sched_sim_driver_t* sim, + const char* path, bool verify, + int* out_rc) +{ + sched_sim_args_t args; + char tmp[] = "/tmp/xce_simlivealloc_run_XXXXXX"; + int fd = mkstemp(tmp); + int saved_stdout; + off_t len; + char* buf; + + TEST_ASSERT_TRUE_MESSAGE(fd >= 0, "mkstemp for stdout capture must " + "succeed"); + + fflush(stdout); + saved_stdout = dup(STDOUT_FILENO); + TEST_ASSERT_TRUE_MESSAGE(saved_stdout >= 0, "dup(STDOUT_FILENO) must " + "succeed"); + TEST_ASSERT_TRUE_MESSAGE(dup2(fd, STDOUT_FILENO) >= 0, + "dup2 redirect of STDOUT_FILENO must succeed"); + + memset(&args, 0, sizeof(args)); + args.replay_path = path; + args.sim_start_ts = SCHED_SIM_START_TS_LATEST; + args.sim_speed = 0; + args.verify = verify; + + *out_rc = sched_sim_driver_run(sim, &args); + + fflush(stdout); + dup2(saved_stdout, STDOUT_FILENO); + close(saved_stdout); + + len = lseek(fd, 0, SEEK_CUR); + TEST_ASSERT_TRUE(len >= 0); + + buf = (char*)calloc(1, (size_t)len + 1); + TEST_ASSERT_NOT_NULL(buf); + TEST_ASSERT_TRUE(lseek(fd, 0, SEEK_SET) == 0); + if (len > 0) { + ssize_t nread = read(fd, buf, (size_t)len); + TEST_ASSERT_TRUE(nread == len); + } + close(fd); + unlink(tmp); + + return buf; +} + /* write_header_and_snapshot - produces a valid ReplayHeader+SnapshotEntry * prefix at a fresh temp path via the real sched_record_start(), same * technique test_sim_event_loop.c's own write_header_and_snapshot()/ @@ -516,9 +606,200 @@ test_sim_driver_run_dispatches_a_job_with_available_resources(void) "state flip with no resource assignment behind it"); } +/* ---------------------------------------------------------------------- */ +/* bug-1838 regression: --sim-verify across a dispatch-then-later- */ +/* confirmation gap, via GENUINE live dispatch (not reconciliation). */ +/* ---------------------------------------------------------------------- */ + +/* + * test_sim_verify_no_false_mismatch_after_live_dispatch_and_later_ + * confirmation -- scheduler-sim-eager-reconciliation's original Requirement + * 4.2 ("a dispatch decision and its RUN-state confirmation recorded several + * rounds apart must not produce a false --sim-verify mismatch"), now proved + * via the mechanism that actually handles this case in production since + * bug-1838 (scheduler-driver-equivalence: sim_reconcile_unaccounted_jobs() + * narrowed back to JOB_IS_RUN-only, undoing scheduler-sim-eager- + * reconciliation's 2026-09-11 PEND-inclusive widening). + * + * The ORIGINAL version of this Observable lived in test_sim_verify.c, which + * deliberately stubs sched_job_picker()/sched_allocator() to a PEND-only + * pair (see that file's own header comment) -- so BEFORE bug-1838, that + * file could only ever prove this Requirement via reconciliation itself + * force-applying the recorded DISPATCH (the PEND-only stub could never + * produce one). Under the new RUN-only trigger, job 601's fixture there + * (freshly PEND at round 1) no longer reaches reconciliation at all -- and + * that file's stub allocator has nothing else to offer, so the scenario is + * no longer reachable there. This file links the REAL scheduler_srcs (see + * this file's own header comment), so it is the correct home for a live + * proof of the same property: job TEST_JOBID is pushed PEND and dispatched + * by the REAL sched_job_picker()/sched_allocator() pair at round 1 + * (matching the fixture the file's own first test already establishes), + * --sim-verify is enabled, and the job's SEPARATE, later-replayed + * JOB_STATE_CHANGE-to-RUN confirmation (round 2) is a redundant, safe + * no-op re-affirmation (core_events.c's own unconditional `job->state = + * new_state` reassignment) -- neither round may report a MISMATCH. + * + * A second, independent `xce_sched_sim_driver_t` (g_sim2, NOT g_sim) is + * required here even though this file's own single-RUN_TEST() convention + * means only one of these two tests actually runs per process invocation + * under normal `meson test` execution -- see this file's own header + * comment ("bug-1598/1632/1763 fixture-safety requirement"): sharing g_sim + * across two independently-invocable RUN_TEST() entries would leave a + * SECOND call's sched_sim_driver_start() memset()ing state a FIRST call's + * still-live, never-joined threads could concurrently be dereferencing, if + * a future harness change ever runs both tests in the same process. Static + * storage (not stack-local) for the identical reason g_sim already is. + */ +static xce_sched_sim_driver_t g_sim2; + +static void +test_sim_verify_no_false_mismatch_after_live_dispatch_and_later_confirmation( + void) +{ + char path[PATH_MAX]; + FILE* f; + xce_resource_t* slots_res; + xce_resource_t* res_ptrs[1]; + xce_worker_t* worker_ptrs[1]; + xce_queue_t* queue; + xce_job_t* job_ptrs[1]; + xce_sched_job_t* sjob; + char* captured; + int rc; + + f = write_header_and_snapshot(path, sizeof(path)); + + slots_res = (xce_resource_t*)calloc(1, sizeof(*slots_res)); + TEST_ASSERT_NOT_NULL(slots_res); + slots_res->name = strdup("slots"); + res_ptrs[0] = slots_res; + append_resourcespush_event(f, 1001ULL, res_ptrs, 1); + + worker_ptrs[0] = make_wire_worker(WORKER_NAME, WORKER_SLOTS); + append_workerspush_event(f, 1002ULL, worker_ptrs, 1); + + queue = make_wire_queue(QUEUE_NAME, 1); + append_queueupdate_event(f, 1003ULL, queue); + + job_ptrs[0] = make_push_job(TEST_JOBID, TEST_RESREQ, (char*)"alice", + (char*)QUEUE_NAME, (char*)"proj1", 1, 1); + append_jobspush_event(f, 1004ULL, job_ptrs, 1); + + /* Round 1: empty recorded decisions -- --sim-verify compares the live + * picker/allocator's OWN freshly-drained DISPATCH decision against + * this round's decoded_session, which carries nothing recorded at all. + * A live decision with no corresponding recorded entry is ITSELF + * reported as a mismatch by sim_loop.c's own drain loop (expected=NULL) + * -- so, unlike test_sim_driver_run_dispatches_a_job_with_available_ + * resources above, THIS test must record the SAME DISPATCH the live + * path is about to independently produce, proving they agree, not + * merely that the live path ran. Built the same way test_sim_ + * contention_parity.c's own make_dispatch_alloc()-based fixtures do: + * one host allocation naming WORKER_NAME, wanting exactly what a + * 1-task job's TEST_RESREQ ("rusage[mem=1]", auto-injected 1 slot per + * task) actually requests. + * + * sim_verify_alloc_eq() (sim_verify.c) compares xce_alloc_t + * field-by-field, including instid and duration -- both must match + * EXACTLY what the real seqjob_allocator()->sched_make_allocs()- + * >make_alloc() chain (scheduler.c) will independently produce for + * this exact fixture, not arbitrary placeholders: + * - instid=1: sched_get_id() returns core->nextid++, and core.c + * initializes nextid to 1 (core.c:127) -- this fixture's job is + * the FIRST (and only) allocation ever made against this fresh + * core, so it is deterministically the first ID issued. + * - duration=-1: seqjob_allocator.c's own sched_make_allocs() call + * passes a NULL duration array ("FIXME: no duration now"), so + * sched_make_allocs() (scheduler.c) always defaults every entry's + * duration to -1 today. + * `from` (wall-clock time(NULL), set unconditionally by make_alloc() + * itself) is NOT independently reproduced here -- it is expected to + * agree with the live path's own call by virtue of both happening + * within the same sub-second test execution window, not by + * construction; this is a real, if practically negligible, source of + * flakiness inherent to comparing a genuinely-live-dispatched + * decision's wall-clock-stamped fields via --sim-verify, orthogonal + * to anything bug-1838 changed. */ + { + xce_alloc_t* a = make_alloc((char*)"slots", NULL, 1, 1.0, + -1); + xce_alloc_t** allocs = make_allocs(1, a); + xce_host_alloc_t* h = make_host_alloc((char*)WORKER_NAME, 1, allocs); + xce_host_alloc_t** hallocs = make_host_allocs(1, h); + xce_job_alloc_t* alloc = make_job_alloc(TEST_JOBID, 1, hallocs); + sched_dispatch_decision_t dispatch; + replay_decision_t d_dispatch; + replay_decision_t* decisions[1]; + replay_session_t session; + struct capn c; + struct capn_segment* cs; + ReplaySession_ptr sp; + + memset(&dispatch, 0, sizeof(dispatch)); + dispatch.alloc = alloc; + + memset(&d_dispatch, 0, sizeof(d_dispatch)); + d_dispatch.body_which = ReplayDecision_body_dispatch; + d_dispatch.body.dispatch = &dispatch; + d_dispatch.ts = 2000ULL; + + decisions[0] = &d_dispatch; + + memset(&session, 0, sizeof(session)); + session.begin_round = 2000ULL; + session.decisions = decisions; + session.n_decisions = 1; + session.summary = NULL; + + capn_init_malloc(&c); + cs = capn_root(&c).seg; + encode_ReplaySession_ptr(cs, &sp, &session); + capn_setp(capn_root(&c), 0, sp.p); + TEST_ASSERT_EQUAL_INT(0, write_capnp_message(f, &c)); + capn_free(&c); + } + + /* Round 2: job's separately-recorded RUN confirmation, several rounds + * (here: one) after its own round-1 DISPATCH -- a redundant, safe + * no-op by the time it replays (the live path already made the job + * RUN at round 1). Recorded decisions empty, matching what a genuinely + * idle round produces. */ + append_jobstatechange_event(f, 2500ULL, TEST_JOBID, JOB_STAT_PEND, + JOB_STAT_RUN); + append_replay_session(f, 3000ULL); + + append_end_of_stream_marker(f); + fclose(f); + + captured = run_sim_and_capture_stdout(&g_sim2, path, true, &rc); + + TEST_ASSERT_EQUAL_INT_MESSAGE( + 0, rc, "sched_sim_driver_run() must complete both rounds and reach " + "its own end-of-stream marker"); + TEST_ASSERT_NULL_MESSAGE( + strstr(captured, "MISMATCH"), + "a dispatch decision and its RUN-state confirmation recorded in a " + "later round must not produce a false --sim-verify mismatch " + "(scheduler-sim-eager-reconciliation Requirement 4.2), now proved " + "via genuine live dispatch (bug-1838) rather than reconciliation"); + + sjob = sched_get_job(&(g_sim2.core), TEST_JOBID); + TEST_ASSERT_NOT_NULL(sjob); + TEST_ASSERT_EQUAL_UINT_MESSAGE( + JOB_STAT_RUN, sjob->state, + "the job must genuinely be RUN via the live picker/allocator's own " + "round-1 dispatch, independent of round 2's redundant confirmation " + "event"); + TEST_ASSERT_NOT_NULL(sjob->allocs); + + free(captured); +} + int main(void) { UNITY_BEGIN(); RUN_TEST(test_sim_driver_run_dispatches_a_job_with_available_resources); + RUN_TEST( + test_sim_verify_no_false_mismatch_after_live_dispatch_and_later_confirmation); return UNITY_END(); } diff --git a/src/scheduler/testsuite/test_sim_print_decision_gate.c b/src/scheduler/testsuite/test_sim_print_decision_gate.c index efccf36f..24f9c5d8 100644 --- a/src/scheduler/testsuite/test_sim_print_decision_gate.c +++ b/src/scheduler/testsuite/test_sim_print_decision_gate.c @@ -237,6 +237,55 @@ int sched_account_jobs(xce_sched_core_t* core) return 0; } +/* bug-1836 (scheduler-driver-equivalence fix, Recorded-Decision + * Reconciliation capacity gate): link-satisfying stubs, mirroring this + * file's existing sched_account_jobs() no-op precedent immediately above -- + * sim_loop.c's sim_reconcile_unaccounted_jobs() now also calls + * sched_check_host_satisfy()/sched_check_shared_satisfy() (resource.c) to + * re-validate a recorded decision against live capacity, and + * sched_charge_account() (resource.c) to keep that check's own view of + * capacity up to date across multiple candidates in the same round. This + * file never exercises real resource contention (same rationale as the + * existing sched_account_jobs()/sched_ctx_get_uncomit_host() no-ops), so + * the satisfy checks trivially always succeed and the charge call is a + * no-op. */ +int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w, + xvec_t* uncomitted, xvec_idval_t* wanted, + xvec_idval_t* unmeet) +{ + (void)core; + (void)w; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted, + xvec_idval_t* wanted, xvec_idval_t* unmeet) +{ + (void)core; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_charge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + +int sched_uncharge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + + /* Real reimplementation of schedobj.c's sched_resolve_job_picker -- same * copy test_sim_alt_events.c/test_sim_verify.c/test_sim_event_loop.c/ * test_repl_execution_control.c/test_repl_round_rollup.c/ diff --git a/src/scheduler/testsuite/test_sim_reconcile_unaccounted.c b/src/scheduler/testsuite/test_sim_reconcile_unaccounted.c index d0b9705d..d474cd8c 100644 --- a/src/scheduler/testsuite/test_sim_reconcile_unaccounted.c +++ b/src/scheduler/testsuite/test_sim_reconcile_unaccounted.c @@ -261,6 +261,7 @@ int sched_account_jobs(xce_sched_core_t* core) return rc; } + /* Real reimplementation of schedobj.c's sched_resolve_job_picker -- same * copy every white-box test in this directory that does not link * schedobj.c already establishes. */ @@ -838,31 +839,45 @@ static void test_reconcile_ignores_non_dispatch_entry_with_matching_jobid( } /* ---------------------------------------------------------------------- */ -/* Test 5 (scheduler-sim-eager-reconciliation task 1.1, replaces the old */ -/* test_reconcile_skips_non_run_state_job): the trigger is now */ -/* state-independent -- a job whose recorded DISPATCH decision is found */ -/* this round gets its allocation captured regardless of whether it has */ -/* reached RUN state yet. This is the inverse of the old (buggy) expected */ -/* behavior: previously a PEND job with a matching recorded DISPATCH entry */ -/* stayed uncaptured; it must now be captured just like an already-RUN */ -/* job (Requirement 1.1, 3.1). The dominant real-world case (dispatch and */ -/* its RUN-state confirmation recorded many rounds apart) depends on this. */ +/* Test 5 -- INVERTED (2026-09-21, bug-1838, scheduler-driver-equivalence). */ +/* Originally test_reconcile_captures_non_run_state_job_when_dispatch_ */ +/* recorded, from scheduler-sim-eager-reconciliation task 1.1: proved the */ +/* trigger was state-independent (a PEND job's recorded DISPATCH got */ +/* captured just like an already-RUN job's). bug-1838 narrowed the trigger */ +/* back to JOB_IS_RUN-only (see sim_reconcile_unaccounted_jobs()'s own */ +/* updated header comment in sim_loop.c for the full rationale: */ +/* scheduler-sim-live-allocation, completed after this spec, gave sim mode */ +/* a genuine capacity-checked live picker/allocator path, making the */ +/* PEND-inclusive widening redundant -- a freshly-PEND job now correctly */ +/* flows to the SAME live path normal mode uses instead). This test is */ +/* renamed and its assertions inverted to lock in the NEW behavior: a */ +/* PEND-state job's recorded DISPATCH decision is left UNTOUCHED by */ +/* reconciliation, even when a matching entry exists -- dispatching such a */ +/* job is now the live picker/allocator's job, not exercised by this */ +/* direct-call unit test (see test_sim_live_allocation.c for that proof). */ /* ---------------------------------------------------------------------- */ static void -test_reconcile_captures_non_run_state_job_when_dispatch_recorded(void) +test_reconcile_ignores_pend_state_job_even_when_dispatch_recorded(void) { xce_sched_sim_driver_t sim; xce_sched_job_t* job; xce_job_alloc_t* jalloc; replay_session_t* decoded; + int residx; fixture_init(&sim); - /* PEND-state, no known allocation yet -- the realistic case: this - * job's recorded DISPATCH decision has been found in this round's - * replay stream, but its RUN-state confirmation has not been replayed - * yet (may land many rounds later). */ + residx = register_slots_resource(&(sim.core)); + add_worker(&(sim.core), "worker1", residx, 1000.0f); + + /* PEND-state, no known allocation yet, with a matching recorded + * DISPATCH decision available this round -- under bug-1838's RUN-only + * trigger, this must NOT be captured: reconciliation's remaining + * scope is a job already RUN with no known allocation (structurally + * invisible to the live picker, since next_job() never offers a RUN + * job) -- a freshly-PEND job like this one is a perfectly ordinary + * live-picker candidate instead. */ job = add_job(&(sim.core), 7005ULL, JOB_STAT_PEND, NULL); TEST_ASSERT_NULL(job->accounts); TEST_ASSERT_NULL(job->allocs); @@ -873,22 +888,27 @@ test_reconcile_captures_non_run_state_job_when_dispatch_recorded(void) sim_reconcile_unaccounted_jobs(&sim, decoded, NULL); - TEST_ASSERT_EQUAL_PTR_MESSAGE( - jalloc, job->allocs, - "a PEND-state job's recorded DISPATCH decision must now be " - "captured (ownership TAKEOVER, identical pointer) just like an " - "already-RUN job -- capture eligibility no longer depends on " - "job->state"); - TEST_ASSERT_NOT_NULL_MESSAGE( + TEST_ASSERT_NULL_MESSAGE( + job->allocs, + "bug-1838: a PEND-state job must NOT be captured by reconciliation " + "even when a matching recorded DISPATCH entry exists -- eligibility " + "is RUN-only again; the live picker/allocator (not exercised by " + "this direct-call unit test) is what decides a PEND job now"); + TEST_ASSERT_NULL_MESSAGE( job->accounts, - "job->accounts must be built (make_job_account()) from the " - "captured allocation even though the job has not reached RUN yet"); + "job->accounts must stay NULL -- nothing here should fabricate an " + "allocation for a job reconciliation correctly left alone"); TEST_ASSERT_EQUAL_INT_MESSAGE( - 0, decoded->n_decisions, - "the now-fully-consumed recorded entry must be removed from " - "decoded_session->decisions, exactly as for an already-RUN job"); + 1, decoded->n_decisions, + "the recorded entry must NOT be consumed -- it was never a " + "candidate match in the first place, so it is left exactly as " + "found for whatever (if anything) still wants to read it"); + TEST_ASSERT_EQUAL_UINT_MESSAGE( + JOB_STAT_PEND, job->state, + "job->state must remain untouched (still PEND) -- reconciliation " + "never even attempted this job"); - printf("test_reconcile_captures_non_run_state_job_when_dispatch_" + printf("test_reconcile_ignores_pend_state_job_even_when_dispatch_" "recorded: PASS\n"); } @@ -906,9 +926,16 @@ test_reconcile_reconstructs_allocs_and_accounts_from_matching_dispatch(void) xce_sched_job_t* job; xce_job_alloc_t* jalloc; replay_session_t* decoded; + int residx; fixture_init(&sim); + /* bug-1836 fix: see test_reconcile_captures_non_run_state_job_when_ + * dispatch_recorded's own comment above -- the capacity gate needs a + * REAL registered "worker1". */ + residx = register_slots_resource(&(sim.core)); + add_worker(&(sim.core), "worker1", residx, 1000.0f); + job = add_job(&(sim.core), 7010ULL, JOB_STAT_RUN, NULL); TEST_ASSERT_NULL(job->accounts); TEST_ASSERT_NULL(job->allocs); @@ -1017,9 +1044,14 @@ static void test_reconcile_then_release_via_job_state_change_is_asan_clean(void) replay_session_t* decoded; xce_core_event_t ev; int push_rc; + int residx; fixture_init(&sim); + /* bug-1836 fix: the capacity gate needs a REAL registered "worker1". */ + residx = register_slots_resource(&(sim.core)); + add_worker(&(sim.core), "worker1", residx, 1000.0f); + job = add_job(&(sim.core), 7012ULL, JOB_STAT_RUN, NULL); decoded = make_decoded_session( @@ -1094,9 +1126,14 @@ test_reconcile_reports_reconstructed_decision_through_existing_channel(void) int saved_stdout; off_t len; char* captured; + int residx; fixture_init(&sim); + /* bug-1836 fix: the capacity gate needs a REAL registered "worker9". */ + residx = register_slots_resource(&(sim.core)); + add_worker(&(sim.core), "worker9", residx, 1000.0f); + job = add_job(&(sim.core), 7020ULL, JOB_STAT_RUN, NULL); TEST_ASSERT_NULL(job->accounts); @@ -1208,6 +1245,16 @@ static void test_reconcile_null_args_are_a_safe_noop(void) /* contributed zero to resInUse until a SEPARATE, later job->state = RUN */ /* write. See requirements.md's Requirement 1 2026-09-11 revision and */ /* research.md's corresponding entry for the full rationale trail. */ +/* */ +/* REVISED AGAIN (2026-09-21, bug-1838): the job below now starts as */ +/* JOB_STAT_RUN, not PEND -- bug-1838 narrowed reconciliation's trigger */ +/* back to JOB_IS_RUN-only (a PEND job is no longer a candidate at all, */ +/* regardless of a matching recorded decision -- see sim_loop.c's own */ +/* updated header comment on sim_reconcile_unaccounted_jobs()). This */ +/* test's own point -- a captured job's resInUse charge lands the SAME */ +/* round as capture, not deferred -- is orthogonal to PEND-vs-RUN and */ +/* still fully exercised by a RUN-starting job (the remaining genuine */ +/* reconciliation case: already RUN, allocation still unknown). */ /* ---------------------------------------------------------------------- */ static void @@ -1229,9 +1276,11 @@ test_captured_job_included_in_resinuse_same_round_as_capture(void) residx = register_slots_resource(&(sim.core)); w = add_worker(&(sim.core), "worker1", residx, 10.0f); - /* PEND-state job, no known allocation yet -- its recorded DISPATCH - * decision is about to be found this round. */ - job = add_job(&(sim.core), 7030ULL, JOB_STAT_PEND, NULL); + /* bug-1838: RUN-state job, no known allocation yet -- the remaining + * genuine reconciliation case (already RUN before this replay's own + * observation window, allocation never captured). Its recorded + * DISPATCH decision is about to be found this round. */ + job = add_job(&(sim.core), 7030ULL, JOB_STAT_RUN, NULL); TEST_ASSERT_NULL(job->accounts); TEST_ASSERT_NULL(job->allocs); @@ -1327,20 +1376,137 @@ test_captured_job_included_in_resinuse_same_round_as_capture(void) "PASS\n"); } +/* ---------------------------------------------------------------------- */ +/* bug-1836 regression (scheduler-driver-equivalence): a resource- */ +/* contention fixture per .kiro/steering/scheduler-driver-equivalence.md's */ +/* own testing discipline ("multiple jobs competing for capacity */ +/* insufficient for all of them"). Live symptom this closes: a real */ +/* 22-slot host took 44 concurrent single-slot jobs during replay, because */ +/* sim_reconcile_unaccounted_jobs() force-applied every recorded DISPATCH */ +/* decision unconditionally, never checking whether it still fit CURRENT */ +/* capacity. Two RUN jobs (bug-1838: reconciliation's trigger is RUN-only */ +/* again -- these model jobs already confirmed RUN via a replayed */ +/* JOB_STATE_CHANGE event, awaiting their allocation data), each with a */ +/* recorded DISPATCH decision wanting the SAME host's only unit of */ +/* "slots" (capacity 1) -- normal mode would dispatch exactly one and PEND */ +/* the other; this proves sim mode's reconciliation backstop now does the */ +/* capacity check instead of force-applying both unconditionally. */ +/* ---------------------------------------------------------------------- */ + +static void test_reconcile_respects_host_capacity_under_contention(void) +{ + xce_sched_sim_driver_t sim; + xce_sched_job_t* jobA; + xce_sched_job_t* jobB; + xce_sched_worker_t* w; + xce_job_alloc_t* allocA; + xce_job_alloc_t* allocB; + replay_session_t* decoded; + int residx; + float got; + int rc; + int n_admitted; + int n_unaccounted; + + fixture_init(&sim); + + residx = register_slots_resource(&(sim.core)); + w = add_worker(&(sim.core), "worker1", residx, 1.0f); + + jobA = add_job(&(sim.core), 7040ULL, JOB_STAT_RUN, NULL); + jobB = add_job(&(sim.core), 7041ULL, JOB_STAT_RUN, NULL); + + allocA = make_test_job_alloc_with_host(7040ULL, "worker1", "slots", 1.0f); + allocB = make_test_job_alloc_with_host(7041ULL, "worker1", "slots", 1.0f); + + decoded = (replay_session_t*)calloc(1, sizeof(*decoded)); + decoded->begin_round = 1000ULL; + decoded->summary = NULL; + decoded->decisions + = (replay_decision_t**)calloc(2, sizeof(*(decoded->decisions))); + decoded->decisions[0] = make_dispatch_entry_from_alloc(allocA); + decoded->decisions[1] = make_dispatch_entry_from_alloc(allocB); + decoded->n_decisions = 2; + + sim_reconcile_unaccounted_jobs(&sim, decoded, NULL); + + n_admitted = ((jobA->accounts != NULL) ? 1 : 0) + + ((jobB->accounts != NULL) ? 1 : 0); + n_unaccounted = ((jobA->accounts == NULL) ? 1 : 0) + + ((jobB->accounts == NULL) ? 1 : 0); + + TEST_ASSERT_EQUAL_INT_MESSAGE( + 1, n_admitted, + "worker1 has capacity for exactly ONE of these two 1-slot jobs -- " + "reconciliation must admit exactly one, not both (the bug-1836 " + "over-commit) and not zero"); + TEST_ASSERT_EQUAL_INT_MESSAGE( + 1, n_unaccounted, + "the job that does not fit must be left unaccounted (accounts == " + "NULL) to retry on a later round, not force-dispatched"); + TEST_ASSERT_EQUAL_INT_MESSAGE( + 1, decoded->n_decisions, + "the entry for the job that did not fit must NOT be consumed -- it " + "stays available to retry a later round"); + + /* Whichever job was admitted must genuinely be RUN with accounts + * populated; the other must ALSO still be RUN (bug-1838: it entered + * this call already RUN, and a failed capacity gate never reverts + * state -- it is simply left RUN-but-unaccounted, exactly like a job + * whose confirming JOB_STATE_CHANGE arrived before its DISPATCH could + * be reconciled) -- distinguished from the admitted job only by + * accounts/allocs, never by job->state. */ + if (jobA->accounts != NULL) { + TEST_ASSERT_TRUE(JOB_STAT_RUN == jobA->state); + TEST_ASSERT_TRUE(JOB_STAT_RUN == jobB->state); + } else { + TEST_ASSERT_TRUE(JOB_STAT_RUN == jobB->state); + TEST_ASSERT_TRUE(JOB_STAT_RUN == jobA->state); + } + + /* The core correctness bar: resInUse (via the real, unchanged + * sched_account_jobs() call this function's own caller always makes + * immediately afterward in production) must reflect EXACTLY the one + * admitted job's charge -- 1.0, never 2.0 (the bug-1836 over-commit) -- + * proving this function's internal peek+incremental-charge+undo left no + * residue AND that the one real charge landed correctly. */ + rc = sched_account_jobs(&(sim.core)); + TEST_ASSERT_EQUAL_INT(0, rc); + + got = -1.0f; + rc = get_resvals(&(w->resInUse), 1, &residx, &got); + TEST_ASSERT_EQUAL_INT(1, rc); + TEST_ASSERT_EQUAL_FLOAT_MESSAGE( + 1.0f, got, + "bug-1836: resInUse must reflect exactly the one admitted job's " + "charge (1.0), never both (2.0, the over-commit this fix closes) " + "and never neither (0.0, which would mean the fitting job was " + "wrongly rejected too)"); + + printf("test_reconcile_respects_host_capacity_under_contention: PASS\n"); +} + /* ---------------------------------------------------------------------- */ /* Test 13 (scheduler-sim-eager-reconciliation task 1.2, Requirement 3.2): */ /* a job with job->accounts ALREADY set is never re-searched or */ -/* re-captured on a later round's call, regardless of its state (PEND */ -/* here, deliberately different from test_reconcile_skips_job_that_ */ -/* already_has_accounts' RUN-state fixture above -- the state genuinely */ -/* does not matter, only job->accounts != NULL does) -- even when a */ -/* decoded session that round happens to also contain a matching-jobid */ -/* DISPATCH entry. Proves the state-widened gate (task 1.1) did not */ -/* weaken the pre-existing, untouched job->accounts != NULL guard */ -/* (research.md's "Does removing the RUN-state gate risk fabricating or */ -/* double-capturing?" log entry: this guard was never touched by task */ -/* 1.1, so this is regression-proof coverage for an existing guarantee, */ -/* not new-behavior coverage). */ +/* re-captured on a later round's call, deliberately fixtured PEND here */ +/* (different from test_reconcile_skips_job_that_already_has_accounts' */ +/* RUN-state fixture above) -- even when a decoded session that round */ +/* happens to also contain a matching-jobid DISPATCH entry. Proves the */ +/* job->accounts != NULL guard alone is enough to block re-capture, */ +/* independent of whatever the RUN-only trigger condition (bug-1838) */ +/* would otherwise say about this job's state. */ +/* */ +/* UPDATED (2026-09-21, bug-1838): this test predates the RUN-only */ +/* narrowing and originally demonstrated the accounts-guard held even */ +/* under the (now-superseded) state-independent trigger. Since a PEND job */ +/* is skipped by the state check alone today, this fixture no longer */ +/* isolates the accounts guard specifically -- it still correctly proves */ +/* "PEND + already captured" is never re-captured, just via a condition */ +/* that is now doubly-guarded (state AND accounts) rather than singly. */ +/* Kept as-is (not deleted) since it remains a true, valid regression */ +/* lock; see test_reconcile_skips_job_that_already_has_accounts() for the */ +/* RUN-state fixture that DOES still isolate the accounts guard alone. */ /* ---------------------------------------------------------------------- */ static void @@ -1384,8 +1550,9 @@ test_reconcile_still_skips_already_captured_job_regardless_of_state(void) accounts_before, job->accounts, "an already-captured PEND-state job must never be re-captured on a " "later round, even with a matching-jobid recorded DISPATCH entry " - "present -- job->accounts != NULL remains the sole gate under the " - "widened, state-independent trigger"); + "present -- job->accounts != NULL alone is sufficient to block " + "re-capture, regardless of what the RUN-only state check would " + "separately say about this job"); TEST_ASSERT_EQUAL_PTR_MESSAGE( allocs_before, job->allocs, "job->allocs likewise must stay exactly what it already was -- " @@ -1400,16 +1567,22 @@ test_reconcile_still_skips_already_captured_job_regardless_of_state(void) } /* ---------------------------------------------------------------------- */ -/* Test 14 (scheduler-sim-eager-reconciliation task 1.2, Requirement 4.4): */ -/* a JOB_STAT_PEND job with NO matching recorded decision this round stays */ -/* uncaptured (job->accounts stays NULL) after one call, and remains */ -/* eligible -- demonstrated (per this task's own design.md guidance) by */ -/* calling again with a NOW-matching decoded session and confirming */ -/* capture succeeds then. Mirrors test_reconcile_is_safe_to_recheck_on_a_ */ -/* later_round()'s "safe to retry" idiom above, but (unlike that sibling */ -/* test, whose second round also has no match) this is the genuinely new */ -/* behavior task 1.1 enables: a PEND job now gets multiple real chances */ -/* across rounds and can actually succeed on a later one. */ +/* Test 14 (scheduler-sim-eager-reconciliation task 1.2, Requirement 4.4). */ +/* REVISED (2026-09-21, bug-1838): originally a JOB_STAT_PEND job (the */ +/* "in-flight, no match yet" case task 1.1's now-superseded state- */ +/* independent widening existed for). bug-1838 narrowed reconciliation */ +/* back to JOB_IS_RUN-only, so a PEND job is never a candidate regardless */ +/* of whether a match ever appears -- that scenario is now covered by */ +/* test_reconcile_ignores_pend_state_job_even_when_dispatch_recorded */ +/* instead. This test's job is changed to JOB_STAT_RUN so its OWN real */ +/* point survives: a RUN job with no matching recorded decision stays */ +/* uncaptured (job->accounts stays NULL) after one call, remains fully */ +/* eligible, and succeeds once a matching entry appears on a later round -- */ +/* still the remaining, genuine reconciliation case (already RUN, */ +/* allocation unknown, recorded decision not yet found). Mirrors */ +/* test_reconcile_is_safe_to_recheck_on_a_later_round()'s "safe to retry" */ +/* idiom above, but (unlike that sibling test, whose second round also has */ +/* no match) proves the retry can actually succeed on a later one. */ /* ---------------------------------------------------------------------- */ static void @@ -1420,12 +1593,18 @@ test_reconcile_no_match_leaves_pend_job_uncaptured_and_retries(void) xce_job_alloc_t* jalloc; replay_session_t* decoded_round1; replay_session_t* decoded_round2; + int residx; fixture_init(&sim); - /* PEND-state, no known allocation, no recorded decision available yet - * -- the realistic in-flight case task 1.1's widened gate exists for. */ - job = add_job(&(sim.core), 7041ULL, JOB_STAT_PEND, NULL); + /* bug-1836 fix: the capacity gate needs a REAL registered "workerB". */ + residx = register_slots_resource(&(sim.core)); + add_worker(&(sim.core), "workerB", residx, 1000.0f); + + /* bug-1838: RUN-state, no known allocation, no recorded decision + * available yet -- the realistic in-flight case reconciliation's + * RUN-only trigger still genuinely needs to cover. */ + job = add_job(&(sim.core), 7041ULL, JOB_STAT_RUN, NULL); TEST_ASSERT_NULL(job->accounts); TEST_ASSERT_NULL(job->allocs); @@ -1434,8 +1613,9 @@ test_reconcile_no_match_leaves_pend_job_uncaptured_and_retries(void) TEST_ASSERT_NULL_MESSAGE( job->accounts, - "no matching recorded decision this round -- a PEND job must stay " - "uncaptured (usage unavailable), never fabricated"); + "no matching recorded decision this round -- a RUN job with an " + "unknown allocation must stay uncaptured (usage unavailable), " + "never fabricated"); TEST_ASSERT_NULL_MESSAGE(job->allocs, "job->allocs likewise stays NULL"); /* Round 2: the job's recorded DISPATCH decision now shows up. The job @@ -1469,8 +1649,8 @@ test_reconcile_no_match_leaves_pend_job_uncaptured_and_retries(void) /* 3.1 under concurrent load): several simultaneously outstanding jobs, */ /* each with a DIFFERENT jobid, spread across three scenarios a single */ /* reconcile call must resolve with zero cross-job interference: */ -/* - job_matched: PEND, its own DISPATCH entry IS this round. */ -/* - job_no_match: PEND, no recorded entry at all this round. */ +/* - job_matched: RUN, its own DISPATCH entry IS this round. */ +/* - job_no_match: RUN, no recorded entry at all this round. */ /* - job_already: captured on a hypothetical EARLIER round already */ /* (job->accounts/job->allocs preset directly via add_job()'s own */ /* "already accounted" shape, same technique */ @@ -1482,6 +1662,15 @@ test_reconcile_no_match_leaves_pend_job_uncaptured_and_retries(void) /* recorded decision, with no cross-job data or overwrite (the single-job */ /* tests above cannot distinguish "correct" from "correct by coincidence, */ /* only one candidate available", per design.md's own Testing Strategy). */ +/* */ +/* bug-1838: job_matched and job_no_match are both RUN (not PEND) here -- */ +/* reconciliation's trigger is RUN-only again, so a PEND job is now always */ +/* skipped regardless of whether a matching entry exists (that claim is */ +/* covered on its own by */ +/* test_reconcile_ignores_pend_state_job_even_when_dispatch_recorded). */ +/* Keeping job_no_match RUN here preserves this test's original, distinct */ +/* claim: among multiple RUN candidates, reconciliation matches strictly */ +/* by jobid rather than capturing the first/any available entry. */ /* ---------------------------------------------------------------------- */ static void @@ -1497,11 +1686,21 @@ test_reconcile_multiple_jobs_captured_independently_across_different_rounds( xce_job_alloc_t* jalloc_matched; xce_job_alloc_t* jalloc_already_new; replay_session_t* decoded; + int residx; fixture_init(&sim); - job_matched = add_job(&(sim.core), 7050ULL, JOB_STAT_PEND, NULL); - job_no_match = add_job(&(sim.core), 7051ULL, JOB_STAT_PEND, NULL); + /* bug-1836 fix: the capacity gate needs a REAL registered "workerA" -- + * "worker-should-not-apply" (job_already's entry) is deliberately never + * registered: that entry is never even searched for (job_already's + * accounts != NULL guard skips it), so it must never reach the + * capacity gate at all -- if it does, this test's own assertions below + * already catch that regression independently. */ + residx = register_slots_resource(&(sim.core)); + add_worker(&(sim.core), "workerA", residx, 1000.0f); + + job_matched = add_job(&(sim.core), 7050ULL, JOB_STAT_RUN, NULL); + job_no_match = add_job(&(sim.core), 7051ULL, JOB_STAT_RUN, NULL); job_already = add_job(&(sim.core), 7052ULL, JOB_STAT_RUN, make_job_alloc(7052ULL, 0, NULL)); TEST_ASSERT_NOT_NULL_MESSAGE( @@ -1618,9 +1817,14 @@ test_reconcile_continue_mode_with_repl_state_suppresses_bare_decision_line( int saved_stdout; off_t len; char* captured; + int residx; fixture_init(&sim); + /* bug-1836 fix: the capacity gate needs a REAL registered "worker9". */ + residx = register_slots_resource(&(sim.core)); + add_worker(&(sim.core), "worker9", residx, 1000.0f); + job = add_job(&(sim.core), 7060ULL, JOB_STAT_RUN, NULL); TEST_ASSERT_NULL(job->accounts); @@ -1716,9 +1920,14 @@ test_reconcile_step_mode_with_repl_state_still_prints_bare_decision_line( int saved_stdout; off_t len; char* captured; + int residx; fixture_init(&sim); + /* bug-1836 fix: the capacity gate needs a REAL registered "worker9". */ + residx = register_slots_resource(&(sim.core)); + add_worker(&(sim.core), "worker9", residx, 1000.0f); + job = add_job(&(sim.core), 7061ULL, JOB_STAT_RUN, NULL); jalloc = make_test_job_alloc_with_host(7061ULL, "worker9", "slots", 3.0f); @@ -2220,6 +2429,31 @@ test_reconcile_info_worker_and_summary_hide_completed_job_after_stray_dispatch( * picker candidates, so `job` genuinely sits in the picker's pendJobs * exactly as it would after a live JOBS_PUSH, without needing a full * xce_core_push_events() event round trip. + * + * bug-1838: reconciliation's trigger is RUN-only again, so `job` must + * already be RUN by the time sim_reconcile_unaccounted_jobs() looks at it. + * But builtin_fcfs_on_job_push() (the picker hook sim_bootstrap_picker() + * calls) only ever inserts a PEND job into pendJobs -- a job constructed + * directly as RUN would never become a picker candidate in the first + * place, which would make this test's own "fixture sanity" check below + * fail for an unrelated reason (never a candidate to begin with) instead + * of proving the thing this test exists to prove. + * + * So the fixture builds the job PEND (so sim_bootstrap_picker() legitimately + * seats it in pendJobs, exactly like a live JOBS_PUSH would), then forces + * job->state to RUN directly afterward -- WITHOUT going through + * on_job_state_change(), unlike every real production path that reaches + * RUN (XCE_CORE_EVENT_JOB_STATE_CHANGE, core_events.c, always calls + * on_job_state_change() in the same breath it sets job->state, which + * already retires a no-longer-pending job from pendJobs on its own; the + * live sched_allocator() path, sched_election.c, always calls on_dispatch() + * BEFORE sched_commit_decision() sets job->state = RUN). This deliberately + * reproduces the pathological state those two invariants are supposed to + * make unreachable in production -- a RUN job the picker was never told + * about -- so this test still proves sim_reconcile_unaccounted_jobs()'s own + * on_dispatch() call (bug-1759/1779) is a genuine, working safety net, not + * dead code, even though bug-1838 closed off the one call path + * (PEND-state reconciliation) that used to reach it in practice. */ static void test_reconcile_removes_reconciled_job_from_picker_candidates(void) @@ -2251,10 +2485,18 @@ test_reconcile_removes_reconciled_job_from_picker_candidates(void) TEST_ASSERT_NOT_NULL_MESSAGE( offered, "fixture sanity: job must be a picker candidate before " "reconciliation runs"); + TEST_ASSERT_EQUAL_UINT64_MESSAGE( 9101ULL, offered->jobid, "fixture sanity: the offered candidate must be this test's own job"); + /* bug-1838: force RUN directly (bypassing on_job_state_change) -- see + * this test's own header comment for why this, and not constructing + * the job as RUN from the start, is required to both (a) make it a + * genuine reconciliation candidate under the new RUN-only trigger and + * (b) keep it (pathologically) still present in pendJobs going in. */ + job->state = JOB_STAT_RUN; + dispatch_entry = make_dispatch_entry(9101ULL); decoded_session = make_decoded_session(0, dispatch_entry); @@ -2403,11 +2645,26 @@ test_reconcile_registers_global_limit_usage_for_reconciled_job(void) iterator_t it; void* item = NULL; int rc; + xce_resource_t* mem_res; fixture_init(&sim); bootstrap_global_limit_plugin(&sim, "sim_reconcile_global_limit"); + /* bug-1836 fix: the capacity gate needs a REAL registered "worker1" + * owning the "mem" resource. bootstrap_global_limit_plugin() above + * already registered "mem" itself (via its own register_mem_resource() + * call, for limit-matching purposes) -- calling register_mem_resource() + * a second time here would fail (makeResource() rejects a duplicate + * name), so look the already-registered resource up instead; only the + * WORKER instance owning it is still missing (the capacity gate's + * per-host check needs sched_get_worker("worker1") to resolve, which + * bootstrap_global_limit_plugin() never adds). */ + TEST_ASSERT_TRUE( + key_find(sim.core.resmgr, (void*)"mem", (void**)&mem_res) >= 0); + TEST_ASSERT_NOT_NULL(mem_res); + add_worker(&(sim.core), "worker1", mem_res->idx, 1000.0f); + job = add_job(&(sim.core), 7090ULL, JOB_STAT_RUN, NULL); TEST_ASSERT_NULL(job->accounts); /* match_job_global_limit_() (plugins/limit.c) matches against @@ -2534,7 +2791,7 @@ int main(void) RUN_TEST(test_reconcile_skips_job_that_already_has_accounts); RUN_TEST(test_reconcile_ignores_non_dispatch_entry_with_matching_jobid); RUN_TEST( - test_reconcile_captures_non_run_state_job_when_dispatch_recorded); + test_reconcile_ignores_pend_state_job_even_when_dispatch_recorded); RUN_TEST( test_reconcile_reconstructs_allocs_and_accounts_from_matching_dispatch); RUN_TEST(test_reconcile_compacts_via_shift_down_preserving_order); diff --git a/src/scheduler/testsuite/test_sim_record_output.c b/src/scheduler/testsuite/test_sim_record_output.c index 7a4cc149..d74864a9 100644 --- a/src/scheduler/testsuite/test_sim_record_output.c +++ b/src/scheduler/testsuite/test_sim_record_output.c @@ -196,6 +196,55 @@ int sched_account_jobs(xce_sched_core_t* core) return 0; } +/* bug-1836 (scheduler-driver-equivalence fix, Recorded-Decision + * Reconciliation capacity gate): link-satisfying stubs, mirroring this + * file's existing sched_account_jobs() no-op precedent immediately above -- + * sim_loop.c's sim_reconcile_unaccounted_jobs() now also calls + * sched_check_host_satisfy()/sched_check_shared_satisfy() (resource.c) to + * re-validate a recorded decision against live capacity, and + * sched_charge_account() (resource.c) to keep that check's own view of + * capacity up to date across multiple candidates in the same round. This + * file never exercises real resource contention (same rationale as the + * existing sched_account_jobs()/sched_ctx_get_uncomit_host() no-ops), so + * the satisfy checks trivially always succeed and the charge call is a + * no-op. */ +int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w, + xvec_t* uncomitted, xvec_idval_t* wanted, + xvec_idval_t* unmeet) +{ + (void)core; + (void)w; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted, + xvec_idval_t* wanted, xvec_idval_t* unmeet) +{ + (void)core; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_charge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + +int sched_uncharge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + + int sched_resolve_job_picker(xce_sched_core_t* core, xce_sched_plugin_t** out) { xce_sched_plugin_t* plugin = NULL; diff --git a/src/scheduler/testsuite/test_sim_resource_contention.c b/src/scheduler/testsuite/test_sim_resource_contention.c index 9e7b78fb..9f2a1715 100644 --- a/src/scheduler/testsuite/test_sim_resource_contention.c +++ b/src/scheduler/testsuite/test_sim_resource_contention.c @@ -236,6 +236,7 @@ int sched_account_jobs(xce_sched_core_t* core) return 0; } + /* xce_sched_driver_on_decision()'s (driver.c) publish-side linker * dependencies -- never actually invoked: this test never registers a * decision callback, so every decision is simply recorded and dropped. diff --git a/src/scheduler/testsuite/test_sim_round_boundary_reconciliation.c b/src/scheduler/testsuite/test_sim_round_boundary_reconciliation.c index 144dc1c1..dfc8c382 100644 --- a/src/scheduler/testsuite/test_sim_round_boundary_reconciliation.c +++ b/src/scheduler/testsuite/test_sim_round_boundary_reconciliation.c @@ -296,6 +296,7 @@ int sched_account_jobs(xce_sched_core_t* core) return rc; } + /* scheduler-sim-live-allocation spec, task 4.1: sim_loop.c's real * sched_sim_driver_run() (white-box #include'd into this TU above) now * spawns sched_job_picker()/sched_allocator() (sched_election.c) directly diff --git a/src/scheduler/testsuite/test_sim_session_ordering.c b/src/scheduler/testsuite/test_sim_session_ordering.c index a147d8c3..0051fc90 100644 --- a/src/scheduler/testsuite/test_sim_session_ordering.c +++ b/src/scheduler/testsuite/test_sim_session_ordering.c @@ -376,6 +376,55 @@ int sched_account_jobs(xce_sched_core_t* core) return 0; } +/* bug-1836 (scheduler-driver-equivalence fix, Recorded-Decision + * Reconciliation capacity gate): link-satisfying stubs, mirroring this + * file's existing sched_account_jobs() no-op precedent immediately above -- + * sim_loop.c's sim_reconcile_unaccounted_jobs() now also calls + * sched_check_host_satisfy()/sched_check_shared_satisfy() (resource.c) to + * re-validate a recorded decision against live capacity, and + * sched_charge_account() (resource.c) to keep that check's own view of + * capacity up to date across multiple candidates in the same round. This + * file never exercises real resource contention (same rationale as the + * existing sched_account_jobs()/sched_ctx_get_uncomit_host() no-ops), so + * the satisfy checks trivially always succeed and the charge call is a + * no-op. */ +int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w, + xvec_t* uncomitted, xvec_idval_t* wanted, + xvec_idval_t* unmeet) +{ + (void)core; + (void)w; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted, + xvec_idval_t* wanted, xvec_idval_t* unmeet) +{ + (void)core; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_charge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + +int sched_uncharge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + + static void reset_spy(void) { atomic_store(&g_spy_calls, 0); diff --git a/src/scheduler/testsuite/test_sim_snapshot_reconcile_gap.c b/src/scheduler/testsuite/test_sim_snapshot_reconcile_gap.c index dd841996..88f88499 100644 --- a/src/scheduler/testsuite/test_sim_snapshot_reconcile_gap.c +++ b/src/scheduler/testsuite/test_sim_snapshot_reconcile_gap.c @@ -225,6 +225,55 @@ int sched_account_jobs(xce_sched_core_t* core) return 0; } +/* bug-1836 (scheduler-driver-equivalence fix, Recorded-Decision + * Reconciliation capacity gate): link-satisfying stubs, mirroring this + * file's existing sched_account_jobs() no-op precedent immediately above -- + * sim_loop.c's sim_reconcile_unaccounted_jobs() now also calls + * sched_check_host_satisfy()/sched_check_shared_satisfy() (resource.c) to + * re-validate a recorded decision against live capacity, and + * sched_charge_account() (resource.c) to keep that check's own view of + * capacity up to date across multiple candidates in the same round. This + * file never exercises real resource contention (same rationale as the + * existing sched_account_jobs()/sched_ctx_get_uncomit_host() no-ops), so + * the satisfy checks trivially always succeed and the charge call is a + * no-op. */ +int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w, + xvec_t* uncomitted, xvec_idval_t* wanted, + xvec_idval_t* unmeet) +{ + (void)core; + (void)w; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted, + xvec_idval_t* wanted, xvec_idval_t* unmeet) +{ + (void)core; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_charge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + +int sched_uncharge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + + /* Real reimplementation of schedobj.c's sched_resolve_job_picker -- same * copy every white-box test in this directory that does not link * schedobj.c already establishes. */ diff --git a/src/scheduler/testsuite/test_sim_startup.c b/src/scheduler/testsuite/test_sim_startup.c index 940750e2..0a7b9b52 100644 --- a/src/scheduler/testsuite/test_sim_startup.c +++ b/src/scheduler/testsuite/test_sim_startup.c @@ -95,6 +95,55 @@ int sched_account_jobs(xce_sched_core_t* core) return 0; } +/* bug-1836 (scheduler-driver-equivalence fix, Recorded-Decision + * Reconciliation capacity gate): link-satisfying stubs, mirroring this + * file's existing sched_account_jobs() no-op precedent immediately above -- + * sim_loop.c's sim_reconcile_unaccounted_jobs() now also calls + * sched_check_host_satisfy()/sched_check_shared_satisfy() (resource.c) to + * re-validate a recorded decision against live capacity, and + * sched_charge_account() (resource.c) to keep that check's own view of + * capacity up to date across multiple candidates in the same round. This + * file never exercises real resource contention (same rationale as the + * existing sched_account_jobs()/sched_ctx_get_uncomit_host() no-ops), so + * the satisfy checks trivially always succeed and the charge call is a + * no-op. */ +int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w, + xvec_t* uncomitted, xvec_idval_t* wanted, + xvec_idval_t* unmeet) +{ + (void)core; + (void)w; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted, + xvec_idval_t* wanted, xvec_idval_t* unmeet) +{ + (void)core; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_charge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + +int sched_uncharge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + + /* Real reimplementation of schedobj.c's sched_resolve_job_picker -- same * rationale/algorithm as test_record_lifecycle.c's own copy (schedobj.c is * not linked). Both the FIXTURE core (built via a real xce_scheduler_t, diff --git a/src/scheduler/testsuite/test_sim_verify.c b/src/scheduler/testsuite/test_sim_verify.c index 6683ecbd..7edf78bb 100644 --- a/src/scheduler/testsuite/test_sim_verify.c +++ b/src/scheduler/testsuite/test_sim_verify.c @@ -222,6 +222,55 @@ int sched_account_jobs(xce_sched_core_t* core) return 0; } +/* bug-1836 (scheduler-driver-equivalence fix, Recorded-Decision + * Reconciliation capacity gate): link-satisfying stubs, mirroring this + * file's existing sched_account_jobs() no-op precedent immediately above -- + * sim_loop.c's sim_reconcile_unaccounted_jobs() now also calls + * sched_check_host_satisfy()/sched_check_shared_satisfy() (resource.c) to + * re-validate a recorded decision against live capacity, and + * sched_charge_account() (resource.c) to keep that check's own view of + * capacity up to date across multiple candidates in the same round. This + * file never exercises real resource contention (same rationale as the + * existing sched_account_jobs()/sched_ctx_get_uncomit_host() no-ops), so + * the satisfy checks trivially always succeed and the charge call is a + * no-op. */ +int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w, + xvec_t* uncomitted, xvec_idval_t* wanted, + xvec_idval_t* unmeet) +{ + (void)core; + (void)w; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted, + xvec_idval_t* wanted, xvec_idval_t* unmeet) +{ + (void)core; + (void)uncomitted; + (void)wanted; + (void)unmeet; + return 1; +} + +int sched_charge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + +int sched_uncharge_account(xce_sched_core_t* core, xce_job_account_t* acct) +{ + (void)core; + (void)acct; + return 0; +} + + /* Real reimplementation of schedobj.c's sched_resolve_job_picker -- same * copy test_sim_event_loop.c/test_sim_startup.c/test_core_full_cycle.c each * carry (schedobj.c is not linked here: it would need plugin_register(), @@ -750,6 +799,29 @@ static void make_tmp_dir(char* out, size_t out_len, const char* suffix) * require a resolvable picker (Req 5 AC 21: "Picker state is NOT stored in * snapshot"). */ +/* bug-1836 fix: sim_reconcile_unaccounted_jobs()'s new capacity gate calls + * sched_get_worker() for every host named in a reconciled job's recorded + * allocation, and treats "host not found" as a capacity miss -- this file's + * own sched_check_host_satisfy() stub (link-satisfying only, resource.c is + * not white-box #included here) always reports "satisfied" once reached, + * but a bare, unregistered host name is now rejected BEFORE that stub is + * even called. This helper registers a bare worker (no real resource + * values needed -- the stub never reads them) so tests whose fixtures name + * "hostA"/"hostC" in a recorded DISPATCH decision resolve correctly. */ +static void add_bare_worker(xce_sched_core_t* core, const char* name) +{ + xce_sched_worker_t* w = (xce_sched_worker_t*)calloc(1, sizeof(*w)); + + TEST_ASSERT_NOT_NULL(w); + w->name = strdup(name); + TEST_ASSERT_NOT_NULL(w->name); + + TEST_ASSERT_EQUAL_INT(0, zarray_init(&(w->resvals), 8, NULL)); + TEST_ASSERT_EQUAL_INT(0, zarray_init(&(w->resInUse), 8, NULL)); + + TEST_ASSERT_TRUE(sched_add_worker(core, w) >= 0); +} + static void build_fixture_scheduler(xce_scheduler_t* sched, char* record_dir) { int rc; @@ -762,6 +834,13 @@ static void build_fixture_scheduler(xce_scheduler_t* sched, char* record_dir) sched->core.job_pick_plugin = (char*)"builtin_fcfs"; sched->record_dir = record_dir; + + /* bug-1836 fix: registered once here (captured into every fixture's + * initial snapshot) rather than per-test, since several tests in this + * file build a recorded DISPATCH decision naming "hostA"/"hostC" -- + * harmless for tests that never reference these hosts. */ + add_bare_worker(&(sched->core), "hostA"); + add_bare_worker(&(sched->core), "hostC"); } /* Same "leave the file open, skip sched_record_stop()'s own endSession @@ -1543,74 +1622,45 @@ test_verify_with_multiple_already_resolved_jobs_across_rounds_reports_no_mismatc } /* - * test_sim_verify_no_false_mismatch_across_round_separated_dispatch -- - * scheduler-sim-eager-reconciliation spec, task 3.2 (Req 2.1, 2.2, 2.3, - * 4.2): --sim-verify against a replay where a job's recorded DISPATCH - * decision (round 1) and its recorded JOB_STATE_CHANGE-to-RUN confirmation - * (round 4, three rounds later) are genuinely separated -- the multi-round - * shape task 1.1 (this spec) fixed sim_reconcile_unaccounted_jobs() to - * capture correctly. Under the OLD gate (`!JOB_IS_RUN(job->state)` in - * addition to `job->accounts != NULL`), job 601 was still PEND at round 1, - * so reconciliation skipped it entirely -- decoded_session's own - * DISPATCH(601) entry was NEVER consumed, left over for --sim-verify's - * positional walk to compare against a live stream that never produces a - * DISPATCH at all (sim mode's own picker is PEND-only) -- a false - * MISMATCH. research.md's own "Interaction with --sim-verify" research log - * entry documents this exact latent risk and why widening the gate to - * `job->accounts == NULL` (state-independent) removes it rather than - * introducing a new one. + * test_reconciliation_no_longer_masks_a_pend_jobs_dispatch_mismatch -- + * RENAMED and INVERTED (2026-09-21, bug-1838, scheduler-driver-equivalence): + * this test used to be test_sim_verify_no_false_mismatch_across_round_ + * separated_dispatch, proving scheduler-sim-eager-reconciliation's Req 4.2 + * ("a dispatch decision and its RUN-state confirmation recorded several + * rounds apart must not produce a false --sim-verify mismatch") via + * sim_reconcile_unaccounted_jobs() force-applying job 601's recorded round-1 + * DISPATCH decision while it was still PEND (the gate was state-independent + * at the time). * - * **Correction (2026-09-07, bug-1759/bug-1779, scheduler-sim-decision- - * parity):** this fixture originally recorded a PEND(601) entry alongside - * round 1's DISPATCH(601), and again for rounds 2-3, reasoning that - * sim_reconcile_unaccounted_jobs() never called picker_iface->on_dispatch() - * to retire job 601 from the live picker's own pendJobs -- so the live - * picker allegedly HAD to keep re-offering and re-PENDing it every round - * until the round-4 RUN confirmation removed it. That turned out to be a - * genuine, live-observed bug (bug-1759: a 50-job batch where 22 - * already-dispatched jobs also received a PEND in the same round; - * bug-1779: a 10-element array job, 5 already-dispatched elements same - * shape), not an unavoidable consequence of the design -- it directly - * contradicts this same spec's own design.md "Integration with - * --sim-verify" section, which already documented the correct assumption - * ("the picker never re-decides an already-resolved job"). Fixed: - * sim_reconcile_unaccounted_jobs() now calls on_dispatch() the moment it - * reconstructs a job's allocation, mirroring sched_allocator()'s own live - * dispatch ordering (scheduler.c) exactly. The fixture below reflects the - * corrected shape. + * bug-1838 narrowed that gate back to JOB_IS_RUN-only (see sim_loop.c's own + * updated header comment on sim_reconcile_unaccounted_jobs() for the full + * rationale: scheduler-sim-live-allocation, completed after this spec, gave + * sim mode a genuine capacity-checked live picker/allocator path, making + * reconciliation's PEND-inclusive widening redundant for an ordinary + * cross-round gap -- the live path now independently re-derives the SAME + * decision the recording made). Job 601 here is freshly PEND at round 1, so + * it is no longer a reconciliation candidate at all -- but THIS FILE's own + * sched_job_picker()/sched_allocator() pair (see this file's header comment, + * "Genuine resource-based dispatch... is proved separately by + * test_sim_live_allocation.c") is a deliberate PEND-only stub, incapable of + * ever producing a live DISPATCH. With neither mechanism able to dispatch + * job 601 anymore, comparing the live stream's PEND(601) against the + * recorded DISPATCH(601) now correctly reports a MISMATCH -- not a false + * one: this fixture's own environment genuinely cannot reproduce what the + * recording claims happened, and --sim-verify is supposed to say so. * - * Round shape (mirrors test_sim_contention_parity.c's own task 3.1 fixture - * numbers -- job A: dispatch round 1, RUN-confirmed round 4, a 3-round - * gap): - * Round 1 (begin_round=2000): job 601 pushed PEND; recorded decisions = - * [DISPATCH(601)]. sim_reconcile_unaccounted_jobs() retires job 601 - * from the live picker's own pendJobs (on_dispatch()) BEFORE this - * round's picker session runs (sim_loop.c's own fixed ordering: - * reconcile, then xce_core_signal_go(), then drain/compare), so the - * live picker has nothing left to offer this round either -- no - * same-round PEND echo to record. - * Round 2, 3 (begin_round=3000/4000): job 601 is retired from the live - * picker's pendJobs (round 1's on_dispatch() above) -- nothing left for - * the real picker to offer or re-PEND. Recorded decisions = [] (empty - * ReplaySessions), the same shape round 4 below already uses for the - * identical reason. Still genuinely widens the dispatch-to-confirmation - * GAP to "several rounds" (Requirement 4.2) -- only the picker-echo - * noise in between changed, not the gap itself. - * Round 4 (begin_round=5000): job 601's JOB_STATE_CHANGE to RUN is - * applied as part of this round's own pushed event batch. Recorded - * decisions = [] (an empty ReplaySession, the same shape a genuinely - * idle round produces in a real recording). - * - * Zero mismatches across all four rounds is the Observable (Requirement - * 4.2). It is also consistent evidence for 2.1 (the DISPATCH decision IS - * reported, via reconciliation's own three-call sequence, at round 1 -- - * not silently dropped or deferred to round 4) and 2.3 (no duplicate - * report: the live stream never independently emits a DISPATCH for job - * 601 at all, so there is nothing for reconciliation's own report to - * collide with). + * The genuine, mismatch-free proof of Req 4.2 (a real dispatch-then-later- + * confirmation gap causing no false mismatch) now lives in + * test_sim_live_allocation.c's test_sim_verify_no_false_mismatch_after_ + * live_dispatch_and_later_confirmation, which links the real scheduler_srcs + * and can genuinely dispatch job 601 the way this file's stub never could + * -- see that test's own header comment for the full mirrored rationale. + * This test is kept, inverted, as a regression lock on the NEW behavior: + * reconciliation must not "help" a PEND job here anymore, even when a + * matching recorded DISPATCH entry exists. */ static void -test_sim_verify_no_false_mismatch_across_round_separated_dispatch(void) +test_reconciliation_no_longer_masks_a_pend_jobs_dispatch_mismatch(void) { char path[PATH_MAX]; FILE* f; @@ -1630,31 +1680,22 @@ test_sim_verify_no_false_mismatch_across_round_separated_dispatch(void) host[0] = make_test_host_alloc("hostA", res, 1); alloc = make_test_job_alloc(601ULL, host, 1); - /* Round 1: DISPATCH recorded. bug-1759/bug-1779 correction: - * sim_reconcile_unaccounted_jobs() now calls the picker's on_dispatch() - * the moment it reconstructs job 601's allocation this round, retiring - * it from the live picker's own pendJobs BEFORE that round's picker - * session runs (sim_loop.c's fixed ordering: reconcile, then - * xce_core_signal_go()) -- so there is no longer a same-round PEND - * echo to record. */ + /* Round 1: job 601 pushed PEND this same round; recorded decisions = + * [DISPATCH(601)]. bug-1838: sim_reconcile_unaccounted_jobs() is now + * JOB_IS_RUN-only, so it does not touch job 601 (still PEND) at all -- + * this file's own PEND-only sched_allocator() stub (see this file's + * header comment) is the only thing that CAN decide it, and it always + * PENDs. The live stream therefore produces PEND(601) against a + * recorded DISPATCH(601) -- a genuine, correctly-reported mismatch. */ append_replay_session_with_dispatch(f, 2000ULL, alloc); - /* Round 2, 3: job 601 is now retired from the live picker's pendJobs - * (bug-1759/bug-1779 correction, same reasoning as round 4's own - * comment below) -- nothing left for the live picker to offer, so these - * rounds are recorded empty, the same shape a genuinely idle round - * produces. Still genuinely widens the dispatch-to-confirmation gap to - * "several rounds" (Requirement 4.2) -- only the picker-echo noise in - * between changed, not the gap itself. */ + /* Round 2: job 601 is STILL a picker candidate (nothing retired it -- + * reconciliation never touched it, and the stub allocator's on_pend() + * path doesn't retire candidates either), so the live stream produces + * ANOTHER PEND(601) here against this round's empty recorded + * decisions -- also a genuine mismatch, not asserted on individually + * (this test only checks that at least one mismatch is reported). */ append_empty_replay_session(f, 3000ULL); - append_empty_replay_session(f, 4000ULL); - - /* Round 4: job 601's RUN confirmation, 3 rounds after its own round-1 - * DISPATCH -- applied before this round's picker session runs, so the - * live picker has nothing left to offer. */ - append_job_state_change_event(f, 4500ULL, 601ULL, JOB_STAT_PEND, - JOB_STAT_RUN); - append_empty_replay_session(f, 5000ULL); append_end_of_stream_marker(f); fclose(f); @@ -1662,15 +1703,17 @@ test_sim_verify_no_false_mismatch_across_round_separated_dispatch(void) captured = run_sim_and_capture_stdout(path, true, &rc); TEST_ASSERT_EQUAL_INT(0, rc); - TEST_ASSERT_NULL_MESSAGE( + TEST_ASSERT_NOT_NULL_MESSAGE( strstr(captured, "MISMATCH"), - "a dispatch decision and its RUN-state confirmation recorded " - "several rounds apart must not produce a false --sim-verify " - "mismatch (Requirement 4.2)"); + "a PEND-state job's recorded DISPATCH decision must no longer be " + "force-applied by reconciliation (bug-1838) -- this file's " + "PEND-only live stub cannot reproduce it either, so --sim-verify " + "must correctly report a real mismatch here, not silently agree " + "with a decision nothing in this environment actually made"); free(captured); - printf("test_sim_verify_no_false_mismatch_across_round_separated_" - "dispatch: PASS\n"); + printf("test_reconciliation_no_longer_masks_a_pend_jobs_dispatch_" + "mismatch: PASS\n"); } int main(void) @@ -1695,7 +1738,7 @@ int main(void) RUN_TEST( test_verify_with_multiple_already_resolved_jobs_across_rounds_reports_no_mismatch); RUN_TEST( - test_sim_verify_no_false_mismatch_across_round_separated_dispatch); + test_reconciliation_no_longer_masks_a_pend_jobs_dispatch_mismatch); return UNITY_END(); } diff --git a/src/scheduler/testsuite/test_snapshot_save_load.c b/src/scheduler/testsuite/test_snapshot_save_load.c index 1f7c97dc..87a2baf8 100644 --- a/src/scheduler/testsuite/test_snapshot_save_load.c +++ b/src/scheduler/testsuite/test_snapshot_save_load.c @@ -1785,6 +1785,7 @@ int sched_account_jobs(xce_sched_core_t* core) return rc; } + static void test_snapshot_round_trip_mixed_jobs_shared_pool_resinuse_and_teardown_independence( void)