63 lines
3.6 KiB
Markdown
63 lines
3.6 KiB
Markdown
---
|
||
id: "bug-1927"
|
||
status: "open"
|
||
status_updated: "2026-10-08"
|
||
status_note: "newly logged"
|
||
component: "scheduler"
|
||
severity: "noise"
|
||
timestamp: "2026-10-08T17:12:08Z"
|
||
last_seen: "2026-10-08T17:12:08Z"
|
||
occurrences: 1
|
||
files: ["src/lib/preason.c", "src/scheduler/src/resource.c", "src/scheduler/src/scheduler.c"]
|
||
tags: ["preason", "parallel-job", "pend-reason", "payload"]
|
||
related: ["bug-1692", "bug-1845"]
|
||
---
|
||
|
||
# bug-1927: scheduler pend reasons repeat one NO_RESOURCE entry per host per plan (no dedup in add_res_preason)
|
||
|
||
## Description
|
||
**Observed:** a pending parallel job's pend-reason set holds one identical `NO_RESOURCE` entry per host per placement plan tried, with no de-duplication. In the scheduler-sim-decision-trace bounds test (1000 workers, maxTasks=64, the job pends after 64 plans), the traced pend outcome shows:
|
||
```
|
||
host bh0000: resource slots: NO_RESOURCE, resource slots: NO_RESOURCE, ... (63 times, 1842 chars)
|
||
...
|
||
omitted hosts' pend reasons: NO_RESOURCE x87232
|
||
... 950 hosts omitted
|
||
```
|
||
That is about 93k heap-allocated `xce_res_reason_t` entries (each with strdup'd host and resource strings) for what is semantically about 1000 distinct (host, resource, code) reasons.
|
||
|
||
**Expected:** each distinct (host, resource, code) reason appears once in a job's pend-reason set.
|
||
|
||
**Impact:** noise / payload inflation; no wrong scheduling decision. The set is rebuilt for each attempt (scheduler.c:328 and :419 free the old `obj->reason` before installing the new one), so it does not accumulate across sessions. But each pend carries plans × hosts entries: the allocation cost on the allocator thread, the `sched_publish_pend_reason` payload sent to the controller, and anything that displays pend reasons (the decision trace's pend outcome; the trace caps it at 50 hosts) all scale with that product.
|
||
|
||
**Found by:** spec task scheduler-sim-decision-trace 6.4 (bounds test) and its review.
|
||
|
||
Suspected: `add_res_preason()` (src/lib/preason.c ~93-143) always appends, and `resource.c` (~601) calls `add_job_preason(..., PREASON_NO_RESOURCE, w->name, r->name)` for every unmet resource on every per-host evaluation. The runner re-evaluates the same hosts for each plan, so the same entry is appended once per plan. Related code, but not this defect: bug-1692, bug-1845.
|
||
|
||
## Environment
|
||
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
||
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
||
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
||
- Build: asan (xce.build)
|
||
- Code: scheduler-framework @ 1b78fa96-dirty
|
||
- Deployment: none (unit test `sched_trace_bounds_test`: the real scheduler allocator thread over a 1000-worker fixture core; the same code runs in live and sim mode)
|
||
|
||
## Reproduction
|
||
1. Build the scheduler: `bash src/tools/xce-build.sh scheduler` (ASAN, `src/xce.build`).
|
||
2. `XCE_TRACE_BOUNDS_DUMP=1 ASAN_OPTIONS=detect_leaks=0 src/xce.build/scheduler/testsuite/sched_trace_bounds_test` (test in src/scheduler/testsuite/test_trace_bounds.c, commit 1b78fa96).
|
||
3. In the dumped pend outcome, each listed host line repeats `resource slots: NO_RESOURCE` 63 times, and the omitted-host summary reads `NO_RESOURCE x87232` for 950 hosts.
|
||
- Frequency: deterministic (1/1, 2026-10-08)
|
||
- Evidence: `/home/claude/tmp/bug-1927/bounds-dump.out`
|
||
|
||
## Root cause
|
||
TODO: required for fixed/partial
|
||
|
||
## Fix
|
||
TODO: required for fixed/partial
|
||
|
||
## Verification
|
||
- Test: TODO: name + file; RED before the fix, GREEN after
|
||
- Suites: TODO: suite results and build profile
|
||
- Live: TODO: soak/deploy run, or n/a and why
|
||
|
||
## References
|
||
- Related: none
|