From dc624bfca52f16a050f9a4d0367cb5d468eacf72 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Thu, 8 Oct 2026 17:13:06 +0000 Subject: [PATCH] docs(buglog): bug-1927 pend reasons repeat one NO_RESOURCE entry per host per plan Co-Authored-By: Claude Opus 5.5 (1M context) --- .wolf/buglog.json | 27 ++++++++++++++++++ .wolf/bugs/bug-1927.md | 63 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 90 insertions(+) create mode 100644 .wolf/bugs/bug-1927.md diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 22d904f2..d8642b12 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -17533,6 +17533,33 @@ "related_bugs": [], "fix_commits": [], "record": ".wolf/bugs/bug-1926.md" + }, + { + "id": "bug-1927", + "status": "open", + "status_updated": "2026-10-08", + "status_note": "newly logged", + "component": "scheduler", + "severity": "noise", + "timestamp": "2026-10-08T17:12:08Z", + "last_seen": "2026-10-08T17:12:08Z", + "occurrences": 1, + "error_message": "scheduler pend reasons repeat one NO_RESOURCE entry per host per plan (no dedup in add_res_preason)\n\n**Observed:** a pending parallel job's pend-reason set holds one identical `NO_RESOURCE` entry per host per placement plan tried, with no de-duplication. In the scheduler-sim-decision-trace bounds test (1000 workers, maxTasks=64, the job pends after 64 plans), the traced pend outcome shows:\n```\n host bh0000: resource slots: NO_RESOURCE, resource slots: NO_RESOURCE, ... (63 times, 1842 chars)\n ...\n omitted hosts' pend reasons: NO_RESOURCE x87232\n ... 950 hosts omitted\n```\nThat is about 93k heap-allocated `xce_res_reason_t` entries (each with strdup'd host and resource strings) for what is semantically about 1000 distinct (host, resource, code) reasons.\n\n**Expected:** each distinct (host, resource, code) reason appears once in a job's pend-reason set.\n\n**Impact:** noise / payload inflation; no wrong scheduling decision. The set is rebuilt for each attempt (scheduler.c:328 and :419 free the old `obj->reason` before installing the new one), so it does not accumulate across sessions. But each pend carries plans × hosts entries: the allocation cost on the allocator thread, the `sched_publish_pend_reason` payload sent to the controller, and anything that displays pend reasons (the decision trace's pend outcome; the trace caps it at 50 hosts) all scale with that product.\n\n**Found by:** spec task scheduler-sim-decision-trace 6.4 (bounds test) and its review.\n\nSuspected: `add_res_preason()` (src/lib/preason.c ~93-143) always appends, and `resource.c` (~601) calls `add_job_preason(..., PREASON_NO_RESOURCE, w->name, r->name)` for every unmet resource on every per-host evaluation. The runner re-evaluates the same hosts for each plan, so the same entry is appended once per plan. Related code, but not this defect: bug-1692, bug-1845.", + "file": "src/lib/preason.c, src/scheduler/src/resource.c, src/scheduler/src/scheduler.c", + "root_cause": "TODO: required for fixed/partial", + "fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why", + "tags": [ + "preason", + "parallel-job", + "pend-reason", + "payload" + ], + "related_bugs": [ + "bug-1692", + "bug-1845" + ], + "fix_commits": [], + "record": ".wolf/bugs/bug-1927.md" } ] } diff --git a/.wolf/bugs/bug-1927.md b/.wolf/bugs/bug-1927.md new file mode 100644 index 00000000..c589faf3 --- /dev/null +++ b/.wolf/bugs/bug-1927.md @@ -0,0 +1,63 @@ +--- +id: "bug-1927" +status: "open" +status_updated: "2026-10-08" +status_note: "newly logged" +component: "scheduler" +severity: "noise" +timestamp: "2026-10-08T17:12:08Z" +last_seen: "2026-10-08T17:12:08Z" +occurrences: 1 +files: ["src/lib/preason.c", "src/scheduler/src/resource.c", "src/scheduler/src/scheduler.c"] +tags: ["preason", "parallel-job", "pend-reason", "payload"] +related: ["bug-1692", "bug-1845"] +--- + +# bug-1927: scheduler pend reasons repeat one NO_RESOURCE entry per host per plan (no dedup in add_res_preason) + +## Description +**Observed:** a pending parallel job's pend-reason set holds one identical `NO_RESOURCE` entry per host per placement plan tried, with no de-duplication. In the scheduler-sim-decision-trace bounds test (1000 workers, maxTasks=64, the job pends after 64 plans), the traced pend outcome shows: +``` + host bh0000: resource slots: NO_RESOURCE, resource slots: NO_RESOURCE, ... (63 times, 1842 chars) + ... + omitted hosts' pend reasons: NO_RESOURCE x87232 + ... 950 hosts omitted +``` +That is about 93k heap-allocated `xce_res_reason_t` entries (each with strdup'd host and resource strings) for what is semantically about 1000 distinct (host, resource, code) reasons. + +**Expected:** each distinct (host, resource, code) reason appears once in a job's pend-reason set. + +**Impact:** noise / payload inflation; no wrong scheduling decision. The set is rebuilt for each attempt (scheduler.c:328 and :419 free the old `obj->reason` before installing the new one), so it does not accumulate across sessions. But each pend carries plans × hosts entries: the allocation cost on the allocator thread, the `sched_publish_pend_reason` payload sent to the controller, and anything that displays pend reasons (the decision trace's pend outcome; the trace caps it at 50 hosts) all scale with that product. + +**Found by:** spec task scheduler-sim-decision-trace 6.4 (bounds test) and its review. + +Suspected: `add_res_preason()` (src/lib/preason.c ~93-143) always appends, and `resource.c` (~601) calls `add_job_preason(..., PREASON_NO_RESOURCE, w->name, r->name)` for every unmet resource on every per-host evaluation. The runner re-evaluates the same hosts for each plan, so the same entry is appended once per plan. Related code, but not this defect: bug-1692, bug-1845. + +## Environment +- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64 +- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2) +- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 +- Build: asan (xce.build) +- Code: scheduler-framework @ 1b78fa96-dirty +- Deployment: none (unit test `sched_trace_bounds_test`: the real scheduler allocator thread over a 1000-worker fixture core; the same code runs in live and sim mode) + +## Reproduction +1. Build the scheduler: `bash src/tools/xce-build.sh scheduler` (ASAN, `src/xce.build`). +2. `XCE_TRACE_BOUNDS_DUMP=1 ASAN_OPTIONS=detect_leaks=0 src/xce.build/scheduler/testsuite/sched_trace_bounds_test` (test in src/scheduler/testsuite/test_trace_bounds.c, commit 1b78fa96). +3. In the dumped pend outcome, each listed host line repeats `resource slots: NO_RESOURCE` 63 times, and the omitted-host summary reads `NO_RESOURCE x87232` for 950 hosts. +- Frequency: deterministic (1/1, 2026-10-08) +- Evidence: `/home/claude/tmp/bug-1927/bounds-dump.out` + +## Root cause +TODO: required for fixed/partial + +## Fix +TODO: required for fixed/partial + +## Verification +- Test: TODO: name + file; RED before the fix, GREEN after +- Suites: TODO: suite results and build profile +- Live: TODO: soak/deploy run, or n/a and why + +## References +- Related: none