myxce/.wolf/bugs/bug-1927.md
Rongsong Shen 5ca59dc9a8
docs(buglog): record Gitea sync for bug-1926 and bug-1927
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-09 03:08:49 +00:00

3.6 KiB
Raw Blame History

id status status_updated status_note component severity timestamp last_seen occurrences files tags related gitea_issue gitea_synced_status gitea_synced_hash
bug-1927 open 2026-10-08 newly logged scheduler noise 2026-10-08T17:12:08Z 2026-10-08T17:12:08Z 1
src/lib/preason.c
src/scheduler/src/resource.c
src/scheduler/src/scheduler.c
preason
parallel-job
pend-reason
payload
bug-1692
bug-1845
352 open 2a0d743e5de9

bug-1927: scheduler pend reasons repeat one NO_RESOURCE entry per host per plan (no dedup in add_res_preason)

Description

Observed: a pending parallel job's pend-reason set holds one identical NO_RESOURCE entry per host per placement plan tried, with no de-duplication. In the scheduler-sim-decision-trace bounds test (1000 workers, maxTasks=64, the job pends after 64 plans), the traced pend outcome shows:

    host bh0000: resource slots: NO_RESOURCE, resource slots: NO_RESOURCE, ... (63 times, 1842 chars)
    ...
    omitted hosts' pend reasons: NO_RESOURCE x87232
    ... 950 hosts omitted

That is about 93k heap-allocated xce_res_reason_t entries (each with strdup'd host and resource strings) for what is semantically about 1000 distinct (host, resource, code) reasons.

Expected: each distinct (host, resource, code) reason appears once in a job's pend-reason set.

Impact: noise / payload inflation; no wrong scheduling decision. The set is rebuilt for each attempt (scheduler.c:328 and :419 free the old obj->reason before installing the new one), so it does not accumulate across sessions. But each pend carries plans × hosts entries: the allocation cost on the allocator thread, the sched_publish_pend_reason payload sent to the controller, and anything that displays pend reasons (the decision trace's pend outcome; the trace caps it at 50 hosts) all scale with that product.

Found by: spec task scheduler-sim-decision-trace 6.4 (bounds test) and its review.

Suspected: add_res_preason() (src/lib/preason.c ~93-143) always appends, and resource.c (~601) calls add_job_preason(..., PREASON_NO_RESOURCE, w->name, r->name) for every unmet resource on every per-host evaluation. The runner re-evaluates the same hosts for each plan, so the same entry is appended once per plan. Related code, but not this defect: bug-1692, bug-1845.

Environment

  • OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
  • libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
  • Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
  • Build: asan (xce.build)
  • Code: scheduler-framework @ 1b78fa96-dirty
  • Deployment: none (unit test sched_trace_bounds_test: the real scheduler allocator thread over a 1000-worker fixture core; the same code runs in live and sim mode)

Reproduction

  1. Build the scheduler: bash src/tools/xce-build.sh scheduler (ASAN, src/xce.build).
  2. XCE_TRACE_BOUNDS_DUMP=1 ASAN_OPTIONS=detect_leaks=0 src/xce.build/scheduler/testsuite/sched_trace_bounds_test (test in src/scheduler/testsuite/test_trace_bounds.c, commit 1b78fa96).
  3. In the dumped pend outcome, each listed host line repeats resource slots: NO_RESOURCE 63 times, and the omitted-host summary reads NO_RESOURCE x87232 for 950 hosts.
  • Frequency: deterministic (1/1, 2026-10-08)
  • Evidence: /home/claude/tmp/bug-1927/bounds-dump.out

Root cause

TODO: required for fixed/partial

Fix

TODO: required for fixed/partial

Verification

  • Test: TODO: name + file; RED before the fix, GREEN after
  • Suites: TODO: suite results and build profile
  • Live: TODO: soak/deploy run, or n/a and why

References

  • Related: none