myxce/.wolf/bugs/bug-1931.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits	gitea_issue	gitea_synced_status	gitea_synced_hash
bug-1931	fixed	2026-10-09		scheduler	wrong-result	2026-10-09T07:23:49Z	2026-10-09T07:23:49Z	1	src/scheduler/src/plugins/parjob_allocator.c src/scheduler/src/alloc_runner.c	scheduler parjob allocation mintasks		e3b87d28	356	fixed	b35f6255449b

bug-1931: xce_scheduler parjob allocator ignores minTasks: a tasks 2..5 job pends though 2 slots are free

Description

Observed: in a xce_scheduler --sim session, element [5] of an array job requesting tasks 2..5 pends although its only candidate host, pc13v, has 2 free slots (22 total, 20 in use by elements [1]-[4], 5 tasks each). The decision trace shows that only 5-task partitions are tried:

[trace] job 0-5-1791529946-0[5]  session 23  position 5  level full
  requirement single  tasks 2..5  requests mem=409600.000000 (400.00 KB)
    plan 1  hosts 1  tasks per position [5]
        host pc13v  rejected (per-host check): resource shortfall
          slots total 22 in-use 20 reserved 0 requested 5
    plan 2  hosts 2  tasks per position [3, 2]
        host pc13v  rejected (per-host check): resource shortfall
          slots total 22 in-use 20 reserved 0 requested 3
    plan 3  hosts 3  tasks per position [2, 2, 1]
        host pc13v  accepted
        ...
    plans: 5 generated, 5 tried, 0 discarded, 0 not tried; no plan could be placed
  outcome: PEND
    host pc13v: resource slots: NO_RESOURCE

Expected: a job with minTasks=2, maxTasks=5 is placed with any task count in 2..5 that fits, preferring the largest; here 2 tasks on pc13v (plan [2]). If even minTasks tasks cannot be placed, the job pends. Impact: every parallel job submitted with a task range (minTasks < maxTasks) is scheduled as if minTasks == maxTasks: it pends until maxTasks tasks fit, wasting free capacity and delaying the job indefinitely on a busy cluster. Wrong scheduling result, no crash. Found by: user review of the sim decision trace (2026-10-09).

Environment

• OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
• libc: not recorded: the failing xce_scheduler binary is not in this checkout (no build dir); host ldd reports glibc 2.40. The defect is in allocator logic and does not depend on libc
• Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
• Build: profile not recorded (asan, debug or release unknown; user's sim session); the defect is profile-independent allocator logic
• Code: scheduler-framework @ a64d0846-dirty
• Deployment: xce_scheduler --sim (sim session 23), one simulated worker pc13v (22 slots), queue xce, policy builtin_fcfs

Reproduction

1. In xce_scheduler --sim, set up one worker with 22 slots and submit an array job of at least 5 elements, each with minTasks=2, maxTasks=5 (any small mem request, e.g. 400 KB).
2. trace on all, then run a scheduling session: elements [1]-[4] start with 5 tasks each (20 slots used).
3. Element [5] pends; its trace lists only partitions of 5 tasks, never [2], [3] or [4].
4. Code-level check: parjob_plan_iter_init()/parjob_plan_iter_next() (src/scheduler/src/plugins/parjob_allocator.c:202-288) read only resreq->maxTasks; no allocator reads minTasks (grep -rn minTasks src/scheduler/src shows only the trace and cache-fingerprint fields).

• Frequency: deterministic
• Evidence: decision trace quoted above (from the user's sim session); session not saved

Root cause

parjob_plan_iter_init()/parjob_plan_iter_next() (src/scheduler/src/plugins/parjob_allocator.c) partition resreq->maxTasks only, and the runner (sched_allocator_run(), src/scheduler/src/alloc_runner.c) searched those plans once. Nothing in the allocation path read minTasks (it reached the primary resreq with bug-1770, but only the trace header and the bug-1932 cache fingerprint used it). The job-level prepare pass in parjob_prepare_allocation() also scaled per-task shared wants and global limits to maxTasks. So a minTasks < maxTasks job behaved as a fixed maxTasks job: it pended until all maxTasks tasks fit, even when minTasks did. Present since the parjob allocator was written (scheduler-allocation spec); affects every job submitted with a task range.

Fix

The runner now handles a task range (minTasks > 0 && minTasks < maxTasks) in sched_alloc_range_(); fixed counts keep the old path unchanged:

• Minimum first: probe minTasks (job-level pass, cluster-wide check, plan search; a fitting plan's claims are released). If it does not fit, the job pends after that one search, with only the minimum's reasons; no larger count is tried.
• Then the maximum, placed directly when it fits.
• Else a binary search between them for the largest count that fits, which is then placed. Correct because fitting is monotone in the count for balanced plans: each part of balanced(N-1, k) is no larger than balanced(N, k)'s, a host that fits a part fits any smaller one, first-fit largest-part-first is complete for balanced plans, and per-task shared wants only grow with N.
• Plumbing: xce_sched_ctx_t.ntasks (new, reset by sched_ctx_set_primary()) carries the count; parjob_prepare_allocation()'s job-level pass scales to it, so per-task shared resources and global slot limits are checked at the count being tried; the plan generator gets the request with its task count fixed to that count (no iface change). The placed count's shared wants are debited before its search and given back if it places nothing.
• Decision trace: new SCHED_TRACE_E_TASK_COUNT entry (sched_trace_task_count()), rendered task count 2 (minimum), task count 5 (maximum), task count 3, placing 3 tasks; a probe that fits shows plan N succeeded and released hosts: ...; one plans: line covers the attempt.
• Plans are still today's balanced partitions, so the placed count is the largest whose balanced plan fits; capacity-driven shapes (e.g. [4,1]) and span options belong to the parjob-placement spec.
• Investigation: the controller launches per host the slots in the allocation (mk_launchinfo()/get_nslots_in_alloc(), src/controller/job.c), never maxtasks, so a smaller placed count launches correctly; no downstream bug. The bug-1932 cache key includes min/max tasks, and an "unsatisfiable" verdict now means the minimum did not fit, which by monotonicity holds for the whole session.

Verification

• Test: src/scheduler/testsuite/test_parjob_task_range.c (new; real runner + real parjob plan generator and prepare hook). RED before the fix: 5/7 failed (test_range_places_minimum_when_only_minimum_fits "Expected 1 Was 0. a 2..5 job must start when 2 slots are free"; test_range_minimum_unfit_pends_without_trying_larger_counts "Expected 2 Was 5"; largest-count, minimum-one and per-task-shared tests pended). GREEN after: 8/8 (adds test_range_unplaced_maximum_leaves_no_shared_debit). test_trace_render.c test_task_range_counts covers the new trace lines.
• Suites: scheduler suite 119/119 OK (asan, src/xce.build, meson test --suite scheduler), ASAN clean. Test fixtures that used minTasks = 1 as filler for "an N-task job" (test_alloc_runner_search/result, test_alloc_equivalence_parjob, test_parjob_combinatorial, test_trace_hooks_runner, test_trace_bounds) were made fixed-count (minTasks = maxTasks), which is what they meant; test_trace_model.c emits the new entry kind; test_repl_trace_e2e.c scenario 1 now expects job 101's (tasks 2..4) minimum probe before its unchanged 4-task placement.
• Live: xce_scheduler --sim --interactive on fixtures/repl_trace_baseline/contention.replay shows the new trace for the range job 101 (minimum probe, maximum placed as [2, 2]); --sim --sim-verify on the two real recordings in data/ gives the same mismatch counts as before the fix (35 and 4848, pre-existing), exit 0.

References

• Fix commit: see fix_commits
• Related: bug-1770 (wired real minTasks/maxTasks into the primary resreq; the allocator never consumed minTasks), bug-1932 (same trace; unsatisfiable cache not used in sim)
• Docs: .kiro/specs/scheduler-allocation/requirements.md (Req 4 AC 1), .kiro/specs/resreq-maxtasks-wiring/, .kiro/specs/parjob-placement/ (owns the full task-range behaviour together with capacity-driven partitioning and span options; this bug fix covers only the min-first + upward search on today's balanced plans)