From 91f29a2a90a0041d1d85759241b5a2b49ebbc4b0 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Fri, 9 Oct 2026 08:51:51 +0000 Subject: [PATCH] docs(buglog): bug-1931 fixed (e3b87d28) Co-Authored-By: Claude Opus 5.5 (1M context) --- .wolf/buglog.json | 12 +++++++----- .wolf/bugs/bug-1931.md | 23 ++++++++++++++++------- 2 files changed, 23 insertions(+), 12 deletions(-) diff --git a/.wolf/buglog.json b/.wolf/buglog.json index ba2ee66d..08168680 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -17669,9 +17669,9 @@ }, { "id": "bug-1931", - "status": "open", + "status": "fixed", "status_updated": "2026-10-09", - "status_note": "newly logged", + "status_note": "", "component": "scheduler", "severity": "wrong-result", "timestamp": "2026-10-09T07:23:49Z", @@ -17679,8 +17679,8 @@ "occurrences": 1, "error_message": "xce_scheduler parjob allocator ignores minTasks: a tasks 2..5 job pends though 2 slots are free\n\n**Observed:** in a `xce_scheduler --sim` session, element `[5]` of an array job requesting `tasks 2..5` pends although its only candidate host, pc13v, has 2 free slots (22 total, 20 in use by elements [1]-[4], 5 tasks each). The decision trace shows that only 5-task partitions are tried:\n```\n[trace] job 0-5-1791529946-0[5] session 23 position 5 level full\n requirement single tasks 2..5 requests mem=409600.000000 (400.00 KB)\n plan 1 hosts 1 tasks per position [5]\n host pc13v rejected (per-host check): resource shortfall\n slots total 22 in-use 20 reserved 0 requested 5\n plan 2 hosts 2 tasks per position [3, 2]\n host pc13v rejected (per-host check): resource shortfall\n slots total 22 in-use 20 reserved 0 requested 3\n plan 3 hosts 3 tasks per position [2, 2, 1]\n host pc13v accepted\n ...\n plans: 5 generated, 5 tried, 0 discarded, 0 not tried; no plan could be placed\n outcome: PEND\n host pc13v: resource slots: NO_RESOURCE\n```\n**Expected:** a job with `minTasks=2, maxTasks=5` is placed with any task count in 2..5 that fits, preferring the largest; here 2 tasks on pc13v (plan `[2]`). If even `minTasks` tasks cannot be placed, the job pends.\n**Impact:** every parallel job submitted with a task range (`minTasks < maxTasks`) is scheduled as if `minTasks == maxTasks`: it pends until `maxTasks` tasks fit, wasting free capacity and delaying the job indefinitely on a busy cluster. Wrong scheduling result, no crash.\n**Found by:** user review of the sim decision trace (2026-10-09).", "file": "src/scheduler/src/plugins/parjob_allocator.c, src/scheduler/src/alloc_runner.c", - "root_cause": "TODO: required for fixed/partial", - "fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why", + "root_cause": "`parjob_plan_iter_init()`/`parjob_plan_iter_next()` (`src/scheduler/src/plugins/parjob_allocator.c`) partition `resreq->maxTasks` only, and the runner (`sched_allocator_run()`, `src/scheduler/src/alloc_runner.c`) searched those plans once. Nothing in the allocation path read `minTasks` (it reached the primary resreq with bug-1770, but only the trace header and the bug-1932 cache fingerprint used it). The job-level prepare pass in `parjob_prepare_allocation()` also scaled per-task shared wants and global limits to `maxTasks`. So a `minTasks < maxTasks` job behaved as a fixed `maxTasks` job: it pended until all `maxTasks` tasks fit, even when `minTasks` did. Present since the parjob allocator was written (scheduler-allocation spec); affects every job submitted with a task range.", + "fix": "The runner now handles a task range (`minTasks > 0 && minTasks < maxTasks`) in `sched_alloc_range_()`; fixed counts keep the old path unchanged:\n- **Minimum first:** probe `minTasks` (job-level pass, cluster-wide check, plan search; a fitting plan's claims are released). If it does not fit, the job pends after that one search, with only the minimum's reasons; no larger count is tried.\n- **Then the maximum**, placed directly when it fits.\n- **Else a binary search** between them for the largest count that fits, which is then placed. Correct because fitting is monotone in the count for balanced plans: each part of balanced(N-1, k) is no larger than balanced(N, k)'s, a host that fits a part fits any smaller one, first-fit largest-part-first is complete for balanced plans, and per-task shared wants only grow with N.\n- **Plumbing:** `xce_sched_ctx_t.ntasks` (new, reset by `sched_ctx_set_primary()`) carries the count; `parjob_prepare_allocation()`'s job-level pass scales to it, so per-task shared resources and global slot limits are checked at the count being tried; the plan generator gets the request with its task count fixed to that count (no iface change). The placed count's shared wants are debited before its search and given back if it places nothing.\n- **Decision trace:** new `SCHED_TRACE_E_TASK_COUNT` entry (`sched_trace_task_count()`), rendered `task count 2 (minimum)`, `task count 5 (maximum)`, `task count 3`, `placing 3 tasks`; a probe that fits shows `plan N succeeded` and `released hosts: ...`; one `plans:` line covers the attempt.\n- Plans are still today's balanced partitions, so the placed count is the largest whose balanced plan fits; capacity-driven shapes (e.g. `[4,1]`) and span options belong to the `parjob-placement` spec.\n- Investigation: the controller launches per host the slots in the allocation (`mk_launchinfo()`/`get_nslots_in_alloc()`, `src/controller/job.c`), never `maxtasks`, so a smaller placed count launches correctly; no downstream bug. The bug-1932 cache key includes min/max tasks, and an \"unsatisfiable\" verdict now means the minimum did not fit, which by monotonicity holds for the whole session.\n\nVerification:\n- Test: `src/scheduler/testsuite/test_parjob_task_range.c` (new; real runner + real parjob plan generator and prepare hook). RED before the fix: 5/7 failed (`test_range_places_minimum_when_only_minimum_fits` \"Expected 1 Was 0. a 2..5 job must start when 2 slots are free\"; `test_range_minimum_unfit_pends_without_trying_larger_counts` \"Expected 2 Was 5\"; largest-count, minimum-one and per-task-shared tests pended). GREEN after: 8/8 (adds `test_range_unplaced_maximum_leaves_no_shared_debit`). `test_trace_render.c` `test_task_range_counts` covers the new trace lines.\n- Suites: scheduler suite 119/119 OK (asan, `src/xce.build`, `meson test --suite scheduler`), ASAN clean. Test fixtures that used `minTasks = 1` as filler for \"an N-task job\" (test_alloc_runner_search/result, test_alloc_equivalence_parjob, test_parjob_combinatorial, test_trace_hooks_runner, test_trace_bounds) were made fixed-count (`minTasks = maxTasks`), which is what they meant; `test_trace_model.c` emits the new entry kind; `test_repl_trace_e2e.c` scenario 1 now expects job 101's (`tasks 2..4`) minimum probe before its unchanged 4-task placement.\n- Live: `xce_scheduler --sim --interactive` on `fixtures/repl_trace_baseline/contention.replay` shows the new trace for the range job 101 (minimum probe, maximum placed as [2, 2]); `--sim --sim-verify` on the two real recordings in `data/` gives the same mismatch counts as before the fix (35 and 4848, pre-existing), exit 0.", "tags": [ "scheduler", "parjob", @@ -17688,7 +17688,9 @@ "mintasks" ], "related_bugs": [], - "fix_commits": [], + "fix_commits": [ + "e3b87d28" + ], "record": ".wolf/bugs/bug-1931.md", "gitea_issue": 356, "gitea_synced_status": "open" diff --git a/.wolf/bugs/bug-1931.md b/.wolf/bugs/bug-1931.md index ace9d42d..ccc4a792 100644 --- a/.wolf/bugs/bug-1931.md +++ b/.wolf/bugs/bug-1931.md @@ -1,8 +1,8 @@ --- id: "bug-1931" -status: "open" +status: "fixed" status_updated: "2026-10-09" -status_note: "newly logged" +status_note: "" component: "scheduler" severity: "wrong-result" timestamp: "2026-10-09T07:23:49Z" @@ -11,6 +11,7 @@ occurrences: 1 files: ["src/scheduler/src/plugins/parjob_allocator.c", "src/scheduler/src/alloc_runner.c"] tags: ["scheduler", "parjob", "allocation", "mintasks"] related: [] +fix_commits: ["e3b87d28"] gitea_issue: 356 gitea_synced_status: "open" gitea_synced_hash: "cc521a2d8caa" @@ -55,16 +56,24 @@ gitea_synced_hash: "cc521a2d8caa" - Frequency: deterministic - Evidence: decision trace quoted above (from the user's sim session); session not saved ## Root cause -TODO: required for fixed/partial +`parjob_plan_iter_init()`/`parjob_plan_iter_next()` (`src/scheduler/src/plugins/parjob_allocator.c`) partition `resreq->maxTasks` only, and the runner (`sched_allocator_run()`, `src/scheduler/src/alloc_runner.c`) searched those plans once. Nothing in the allocation path read `minTasks` (it reached the primary resreq with bug-1770, but only the trace header and the bug-1932 cache fingerprint used it). The job-level prepare pass in `parjob_prepare_allocation()` also scaled per-task shared wants and global limits to `maxTasks`. So a `minTasks < maxTasks` job behaved as a fixed `maxTasks` job: it pended until all `maxTasks` tasks fit, even when `minTasks` did. Present since the parjob allocator was written (scheduler-allocation spec); affects every job submitted with a task range. ## Fix -TODO: required for fixed/partial +The runner now handles a task range (`minTasks > 0 && minTasks < maxTasks`) in `sched_alloc_range_()`; fixed counts keep the old path unchanged: +- **Minimum first:** probe `minTasks` (job-level pass, cluster-wide check, plan search; a fitting plan's claims are released). If it does not fit, the job pends after that one search, with only the minimum's reasons; no larger count is tried. +- **Then the maximum**, placed directly when it fits. +- **Else a binary search** between them for the largest count that fits, which is then placed. Correct because fitting is monotone in the count for balanced plans: each part of balanced(N-1, k) is no larger than balanced(N, k)'s, a host that fits a part fits any smaller one, first-fit largest-part-first is complete for balanced plans, and per-task shared wants only grow with N. +- **Plumbing:** `xce_sched_ctx_t.ntasks` (new, reset by `sched_ctx_set_primary()`) carries the count; `parjob_prepare_allocation()`'s job-level pass scales to it, so per-task shared resources and global slot limits are checked at the count being tried; the plan generator gets the request with its task count fixed to that count (no iface change). The placed count's shared wants are debited before its search and given back if it places nothing. +- **Decision trace:** new `SCHED_TRACE_E_TASK_COUNT` entry (`sched_trace_task_count()`), rendered `task count 2 (minimum)`, `task count 5 (maximum)`, `task count 3`, `placing 3 tasks`; a probe that fits shows `plan N succeeded` and `released hosts: ...`; one `plans:` line covers the attempt. +- Plans are still today's balanced partitions, so the placed count is the largest whose balanced plan fits; capacity-driven shapes (e.g. `[4,1]`) and span options belong to the `parjob-placement` spec. +- Investigation: the controller launches per host the slots in the allocation (`mk_launchinfo()`/`get_nslots_in_alloc()`, `src/controller/job.c`), never `maxtasks`, so a smaller placed count launches correctly; no downstream bug. The bug-1932 cache key includes min/max tasks, and an "unsatisfiable" verdict now means the minimum did not fit, which by monotonicity holds for the whole session. ## Verification -- Test: TODO: name + file; RED before the fix, GREEN after -- Suites: TODO: suite results and build profile -- Live: TODO: soak/deploy run, or n/a and why +- Test: `src/scheduler/testsuite/test_parjob_task_range.c` (new; real runner + real parjob plan generator and prepare hook). RED before the fix: 5/7 failed (`test_range_places_minimum_when_only_minimum_fits` "Expected 1 Was 0. a 2..5 job must start when 2 slots are free"; `test_range_minimum_unfit_pends_without_trying_larger_counts` "Expected 2 Was 5"; largest-count, minimum-one and per-task-shared tests pended). GREEN after: 8/8 (adds `test_range_unplaced_maximum_leaves_no_shared_debit`). `test_trace_render.c` `test_task_range_counts` covers the new trace lines. +- Suites: scheduler suite 119/119 OK (asan, `src/xce.build`, `meson test --suite scheduler`), ASAN clean. Test fixtures that used `minTasks = 1` as filler for "an N-task job" (test_alloc_runner_search/result, test_alloc_equivalence_parjob, test_parjob_combinatorial, test_trace_hooks_runner, test_trace_bounds) were made fixed-count (`minTasks = maxTasks`), which is what they meant; `test_trace_model.c` emits the new entry kind; `test_repl_trace_e2e.c` scenario 1 now expects job 101's (`tasks 2..4`) minimum probe before its unchanged 4-task placement. +- Live: `xce_scheduler --sim --interactive` on `fixtures/repl_trace_baseline/contention.replay` shows the new trace for the range job 101 (minimum probe, maximum placed as [2, 2]); `--sim --sim-verify` on the two real recordings in `data/` gives the same mismatch counts as before the fix (35 and 4848, pre-existing), exit 0. ## References +- Fix commit: see `fix_commits` - Related: bug-1770 (wired real minTasks/maxTasks into the primary resreq; the allocator never consumed minTasks), bug-1932 (same trace; unsatisfiable cache not used in sim) - Docs: .kiro/specs/scheduler-allocation/requirements.md (Req 4 AC 1), .kiro/specs/resreq-maxtasks-wiring/, .kiro/specs/parjob-placement/ (owns the full task-range behaviour together with capacity-driven partitioning and span options; this bug fix covers only the min-first + upward search on today's balanced plans)