docs(buglog): bug-1927 pend reasons repeat one NO_RESOURCE entry per host per plan
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
1b78fa9638
commit
dc624bfca5
GPG key ID:
80533536B0EB651C
2 changed files with 90 additions and 0 deletions
|
|
@ -17533,6 +17533,33 @@
|
|||
"related_bugs": [],
|
||||
"fix_commits": [],
|
||||
"record": ".wolf/bugs/bug-1926.md"
|
||||
},
|
||||
{
|
||||
"id": "bug-1927",
|
||||
"status": "open",
|
||||
"status_updated": "2026-10-08",
|
||||
"status_note": "newly logged",
|
||||
"component": "scheduler",
|
||||
"severity": "noise",
|
||||
"timestamp": "2026-10-08T17:12:08Z",
|
||||
"last_seen": "2026-10-08T17:12:08Z",
|
||||
"occurrences": 1,
|
||||
"error_message": "scheduler pend reasons repeat one NO_RESOURCE entry per host per plan (no dedup in add_res_preason)\n\n**Observed:** a pending parallel job's pend-reason set holds one identical `NO_RESOURCE` entry per host per placement plan tried, with no de-duplication. In the scheduler-sim-decision-trace bounds test (1000 workers, maxTasks=64, the job pends after 64 plans), the traced pend outcome shows:\n```\n host bh0000: resource slots: NO_RESOURCE, resource slots: NO_RESOURCE, ... (63 times, 1842 chars)\n ...\n omitted hosts' pend reasons: NO_RESOURCE x87232\n ... 950 hosts omitted\n```\nThat is about 93k heap-allocated `xce_res_reason_t` entries (each with strdup'd host and resource strings) for what is semantically about 1000 distinct (host, resource, code) reasons.\n\n**Expected:** each distinct (host, resource, code) reason appears once in a job's pend-reason set.\n\n**Impact:** noise / payload inflation; no wrong scheduling decision. The set is rebuilt for each attempt (scheduler.c:328 and :419 free the old `obj->reason` before installing the new one), so it does not accumulate across sessions. But each pend carries plans × hosts entries: the allocation cost on the allocator thread, the `sched_publish_pend_reason` payload sent to the controller, and anything that displays pend reasons (the decision trace's pend outcome; the trace caps it at 50 hosts) all scale with that product.\n\n**Found by:** spec task scheduler-sim-decision-trace 6.4 (bounds test) and its review.\n\nSuspected: `add_res_preason()` (src/lib/preason.c ~93-143) always appends, and `resource.c` (~601) calls `add_job_preason(..., PREASON_NO_RESOURCE, w->name, r->name)` for every unmet resource on every per-host evaluation. The runner re-evaluates the same hosts for each plan, so the same entry is appended once per plan. Related code, but not this defect: bug-1692, bug-1845.",
|
||||
"file": "src/lib/preason.c, src/scheduler/src/resource.c, src/scheduler/src/scheduler.c",
|
||||
"root_cause": "TODO: required for fixed/partial",
|
||||
"fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why",
|
||||
"tags": [
|
||||
"preason",
|
||||
"parallel-job",
|
||||
"pend-reason",
|
||||
"payload"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1692",
|
||||
"bug-1845"
|
||||
],
|
||||
"fix_commits": [],
|
||||
"record": ".wolf/bugs/bug-1927.md"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
63
.wolf/bugs/bug-1927.md
Normal file
63
.wolf/bugs/bug-1927.md
Normal file
|
|
@ -0,0 +1,63 @@
|
|||
---
|
||||
id: "bug-1927"
|
||||
status: "open"
|
||||
status_updated: "2026-10-08"
|
||||
status_note: "newly logged"
|
||||
component: "scheduler"
|
||||
severity: "noise"
|
||||
timestamp: "2026-10-08T17:12:08Z"
|
||||
last_seen: "2026-10-08T17:12:08Z"
|
||||
occurrences: 1
|
||||
files: ["src/lib/preason.c", "src/scheduler/src/resource.c", "src/scheduler/src/scheduler.c"]
|
||||
tags: ["preason", "parallel-job", "pend-reason", "payload"]
|
||||
related: ["bug-1692", "bug-1845"]
|
||||
---
|
||||
|
||||
# bug-1927: scheduler pend reasons repeat one NO_RESOURCE entry per host per plan (no dedup in add_res_preason)
|
||||
|
||||
## Description
|
||||
**Observed:** a pending parallel job's pend-reason set holds one identical `NO_RESOURCE` entry per host per placement plan tried, with no de-duplication. In the scheduler-sim-decision-trace bounds test (1000 workers, maxTasks=64, the job pends after 64 plans), the traced pend outcome shows:
|
||||
```
|
||||
host bh0000: resource slots: NO_RESOURCE, resource slots: NO_RESOURCE, ... (63 times, 1842 chars)
|
||||
...
|
||||
omitted hosts' pend reasons: NO_RESOURCE x87232
|
||||
... 950 hosts omitted
|
||||
```
|
||||
That is about 93k heap-allocated `xce_res_reason_t` entries (each with strdup'd host and resource strings) for what is semantically about 1000 distinct (host, resource, code) reasons.
|
||||
|
||||
**Expected:** each distinct (host, resource, code) reason appears once in a job's pend-reason set.
|
||||
|
||||
**Impact:** noise / payload inflation; no wrong scheduling decision. The set is rebuilt for each attempt (scheduler.c:328 and :419 free the old `obj->reason` before installing the new one), so it does not accumulate across sessions. But each pend carries plans × hosts entries: the allocation cost on the allocator thread, the `sched_publish_pend_reason` payload sent to the controller, and anything that displays pend reasons (the decision trace's pend outcome; the trace caps it at 50 hosts) all scale with that product.
|
||||
|
||||
**Found by:** spec task scheduler-sim-decision-trace 6.4 (bounds test) and its review.
|
||||
|
||||
Suspected: `add_res_preason()` (src/lib/preason.c ~93-143) always appends, and `resource.c` (~601) calls `add_job_preason(..., PREASON_NO_RESOURCE, w->name, r->name)` for every unmet resource on every per-host evaluation. The runner re-evaluates the same hosts for each plan, so the same entry is appended once per plan. Related code, but not this defect: bug-1692, bug-1845.
|
||||
|
||||
## Environment
|
||||
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
||||
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
||||
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
||||
- Build: asan (xce.build)
|
||||
- Code: scheduler-framework @ 1b78fa96-dirty
|
||||
- Deployment: none (unit test `sched_trace_bounds_test`: the real scheduler allocator thread over a 1000-worker fixture core; the same code runs in live and sim mode)
|
||||
|
||||
## Reproduction
|
||||
1. Build the scheduler: `bash src/tools/xce-build.sh scheduler` (ASAN, `src/xce.build`).
|
||||
2. `XCE_TRACE_BOUNDS_DUMP=1 ASAN_OPTIONS=detect_leaks=0 src/xce.build/scheduler/testsuite/sched_trace_bounds_test` (test in src/scheduler/testsuite/test_trace_bounds.c, commit 1b78fa96).
|
||||
3. In the dumped pend outcome, each listed host line repeats `resource slots: NO_RESOURCE` 63 times, and the omitted-host summary reads `NO_RESOURCE x87232` for 950 hosts.
|
||||
- Frequency: deterministic (1/1, 2026-10-08)
|
||||
- Evidence: `/home/claude/tmp/bug-1927/bounds-dump.out`
|
||||
|
||||
## Root cause
|
||||
TODO: required for fixed/partial
|
||||
|
||||
## Fix
|
||||
TODO: required for fixed/partial
|
||||
|
||||
## Verification
|
||||
- Test: TODO: name + file; RED before the fix, GREEN after
|
||||
- Suites: TODO: suite results and build profile
|
||||
- Live: TODO: soak/deploy run, or n/a and why
|
||||
|
||||
## References
|
||||
- Related: none
|
||||
Loading…
Add table
Add a link
Reference in a new issue