3.6 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | gitea_issue | gitea_synced_status | gitea_synced_hash | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1927 | open | 2026-10-08 | newly logged | scheduler | noise | 2026-10-08T17:12:08Z | 2026-10-08T17:12:08Z | 1 |
|
|
|
352 | open | 2a0d743e5de9 |
bug-1927: scheduler pend reasons repeat one NO_RESOURCE entry per host per plan (no dedup in add_res_preason)
Description
Observed: a pending parallel job's pend-reason set holds one identical NO_RESOURCE entry per host per placement plan tried, with no de-duplication. In the scheduler-sim-decision-trace bounds test (1000 workers, maxTasks=64, the job pends after 64 plans), the traced pend outcome shows:
host bh0000: resource slots: NO_RESOURCE, resource slots: NO_RESOURCE, ... (63 times, 1842 chars)
...
omitted hosts' pend reasons: NO_RESOURCE x87232
... 950 hosts omitted
That is about 93k heap-allocated xce_res_reason_t entries (each with strdup'd host and resource strings) for what is semantically about 1000 distinct (host, resource, code) reasons.
Expected: each distinct (host, resource, code) reason appears once in a job's pend-reason set.
Impact: noise / payload inflation; no wrong scheduling decision. The set is rebuilt for each attempt (scheduler.c:328 and :419 free the old obj->reason before installing the new one), so it does not accumulate across sessions. But each pend carries plans × hosts entries: the allocation cost on the allocator thread, the sched_publish_pend_reason payload sent to the controller, and anything that displays pend reasons (the decision trace's pend outcome; the trace caps it at 50 hosts) all scale with that product.
Found by: spec task scheduler-sim-decision-trace 6.4 (bounds test) and its review.
Suspected: add_res_preason() (src/lib/preason.c ~93-143) always appends, and resource.c (~601) calls add_job_preason(..., PREASON_NO_RESOURCE, w->name, r->name) for every unmet resource on every per-host evaluation. The runner re-evaluates the same hosts for each plan, so the same entry is appended once per plan. Related code, but not this defect: bug-1692, bug-1845.
Environment
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
- Build: asan (xce.build)
- Code: scheduler-framework @ 1b78fa96-dirty
- Deployment: none (unit test
sched_trace_bounds_test: the real scheduler allocator thread over a 1000-worker fixture core; the same code runs in live and sim mode)
Reproduction
- Build the scheduler:
bash src/tools/xce-build.sh scheduler(ASAN,src/xce.build). XCE_TRACE_BOUNDS_DUMP=1 ASAN_OPTIONS=detect_leaks=0 src/xce.build/scheduler/testsuite/sched_trace_bounds_test(test in src/scheduler/testsuite/test_trace_bounds.c, commit1b78fa96).- In the dumped pend outcome, each listed host line repeats
resource slots: NO_RESOURCE63 times, and the omitted-host summary readsNO_RESOURCE x87232for 950 hosts.
- Frequency: deterministic (1/1, 2026-10-08)
- Evidence:
/home/claude/tmp/bug-1927/bounds-dump.out
Root cause
TODO: required for fixed/partial
Fix
TODO: required for fixed/partial
Verification
- Test: TODO: name + file; RED before the fix, GREEN after
- Suites: TODO: suite results and build profile
- Live: TODO: soak/deploy run, or n/a and why
References
- Related: none