Rongsong Shen
84a93a0199
- job-array-one-based-index spec.json phase -> implementation-complete - roadmap: job-array-one-based-index entry updated (validated, bugs fixed); 7 implemented specs ticked (component-failover-adoption, config-upgrade-snapshot, ha-term-safety, job-array-single-wal-record, raft-sim-conformance, scheduler-sim-diagnostics, worker-rejoin-resync) - Gitea #89 (bug-1856, wontfix), #331 (bug-1880), #335 (bug-1884) closed; records carry the synced status Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
116 lines
7 KiB
Markdown
116 lines
7 KiB
Markdown
---
|
|
id: "bug-1884"
|
|
status: "fixed"
|
|
status_updated: "2026-09-29"
|
|
status_note: ""
|
|
component: "scheduler"
|
|
severity: "leak"
|
|
timestamp: "2026-09-29T08:20:03Z"
|
|
last_seen: "2026-09-29T08:20:03Z"
|
|
occurrences: 1
|
|
files: ["src/scheduler/src/sim_loop.c", "src/scheduler/src/sim.c", "src/scheduler/src/plugins/limit.c", "src/scheduler/src/main.c"]
|
|
tags: ["scheduler", "sim", "asan", "lsan", "leak"]
|
|
related: ["bug-1632", "bug-1763"]
|
|
fix_commits: ["9b61fd99", "c89e4e45", "8134b75b", "ebfe4794", "c1cd8f91", "90b99d91"]
|
|
gitea_issue: 335
|
|
gitea_synced_status: "fixed"
|
|
gitea_synced_hash: "bfbff0444e9c"
|
|
---
|
|
|
|
# bug-1884: standalone ASAN xce_scheduler --sim exits 1: LeakSanitizer reports 17 MB still allocated at exit
|
|
|
|
## Description
|
|
**Observed:** an ASAN build of `xce_scheduler --sim=<file> --sim-verify`
|
|
replays the whole recording (0 MISMATCH lines) and then exits 1 because
|
|
LeakSanitizer reports at exit:
|
|
```
|
|
==2234882==ERROR: LeakSanitizer: detected memory leaks
|
|
...
|
|
SUMMARY: AddressSanitizer: 17041405 byte(s) leaked in 1187 allocation(s).
|
|
```
|
|
Of the 17,041,405 bytes, 17,036,261 are indirect and 5,144 direct.
|
|
Grouped by first scheduler/lib frame (bytes, leak groups):
|
|
- `kvtab_init` (lib/kv.c:106): 16,941,432 in 15 -- core kv tables, reachable only from leaked roots
|
|
- `sim_process_replay_event_message` (sim_loop.c:403): 49,688 in 54 -- decoded replay events (e.g. `decode_JobStateChangeEvent_ptr`)
|
|
- `xkvmgr_add` (lib/kvmgr.c:114): 21,376 in 9
|
|
- `sim_locate_and_load_snapshot` (sim.c:224): 14,347 in 106 -- snapshot decode (`decode_Conf4Sched` / `decode_QueueConf_list`)
|
|
- `put_resvals` (resource.c:99) via limit.c `update_limit_res_value` and `sched_update_res_accounts`: 2,048 in 2
|
|
- smaller: `parse_resreq`, `core_build_job`, `sched_ctx_init`, `match_job_host_limit_`
|
|
|
|
The indirect bulk fits the simulator never tearing down its core at exit
|
|
(its picker/allocator threads are never joined, see bug-1632/bug-1763);
|
|
the direct groups may include real per-event leaks.
|
|
|
|
**Expected:** a clean replay exits 0 under ASAN with LeakSanitizer enabled,
|
|
or the exit-time teardown frees what the driver owns.
|
|
**Impact:** leak; for scripts, any ASAN `--sim` replay must set
|
|
`ASAN_OPTIONS=detect_leaks=0` or its exit code is always 1, which masks real
|
|
failures and hides any genuine per-event leak among the exit-time noise.
|
|
Release builds are unaffected at exit; a per-event leak would still grow a
|
|
long replay's memory.
|
|
**Found by:** spec task (libxce-alloc-preason-equality 5.3, live
|
|
default-settings replay, 2026-09-29)
|
|
|
|
## Environment
|
|
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
|
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
|
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
|
- Build: asan (src/xce.build.discrete, debug + b_sanitize=address)
|
|
- Code: scheduler-framework @ 4500ec46-dirty
|
|
- Deployment: n/a (standalone simulation mode, no cluster); the replay file was recorded by a local 3-node HA soak (run-ha-soak.sh, 2 sequential rounds, xcecd_virtual worker) at /home/claude/tmp/xce-alloceq-53
|
|
|
|
## Reproduction
|
|
1. `bash src/tools/xce-build.sh scheduler asan` (or any ASAN build dir)
|
|
2. `src/xce.build/scheduler/xce_scheduler --sim=/home/claude/tmp/bug-1884/scheduler-replay-20260929-064412.replay --sim-verify --logfile /tmp/sim.log; echo rc=$?`
|
|
3. Observe `rc=1` and the LeakSanitizer SUMMARY on stderr (before the fix). With `ASAN_OPTIONS=detect_leaks=0` the same run exits 0.
|
|
- Frequency: deterministic (2/2 runs, identical totals, 2026-09-29); also seen on every recording of /home/claude/tmp/xce-ha-dc-soak
|
|
- Evidence: /home/claude/tmp/bug-1884/lsan-report.txt (full report), /home/claude/tmp/bug-1884/scheduler-replay-20260929-064412.replay
|
|
|
|
## Root cause
|
|
Nothing tore the simulator down (threads never joined, core never
|
|
finalized), and teardown then exposed a chain of ownership gaps, several of
|
|
which also leak at runtime in normal mode:
|
|
- xce_sched_core_fini() never destroyed the plugin/job/worker/queue/user/
|
|
project registries, the shared-value arrays, the config or the
|
|
pend_updates queues; kill_sched_job/queue/worker are shallow by design
|
|
(a failed key_add() runs them and callers keep using the value).
|
|
- sim_alloc.c's allocator stand-ins ran a full xce_sched_core_init(),
|
|
re-initializing the process-wide string and selector tables.
|
|
- plugin_unregister() key_del()'d from the registry being destroyed; the
|
|
limit plugin's usage-instance table used kill_limit on xce_vres_t;
|
|
add_vres() put a bare calloc() into the resobj registry.
|
|
- Replay-event payloads were never released after the push ("accepted
|
|
leak"); the start snapshot's non-config remainder was never freed.
|
|
- kill_sched_resval() never freed the value struct (free_Value() frees only
|
|
a string's buffer) -- every replaced value leaked, WORKERS_PUSH updates
|
|
included; put_resvals() created a second value for a repeated index.
|
|
- dup_val() tested which == XCE_VAL_STRING (0 = Value_u_ivalue32): real
|
|
strings were shallow-copied, a nonzero int32 went to makeString().
|
|
- The satisfy checks re-initialized the ctx's live unmeet vector per
|
|
candidate host; replaced pend reasons, old parsed resreqs (JOB_UPDATE,
|
|
duplicate JOBS_PUSH), the parser's error message and each written
|
|
snapshot were never freed; the simulator never freed decision structs.
|
|
|
|
## Fix
|
|
Six commits: 9b61fd99 (sim teardown, core registries, allocator stand-ins,
|
|
plugin_unregister, kill_vres), c89e4e45 (replay-event payload release),
|
|
8134b75b (core owns/frees its config; start snapshot freed), ebfe4794 (sim
|
|
decisions, fcfs plugin, pend_updates), c1cd8f91 (collect-then-free
|
|
teardown for job/worker/queue/resource tables with shallow killers kept;
|
|
dup_val; unmeet; add_vres via makeResource; plugin_register_core rejects
|
|
duplicates up front), 90b99d91 (kill_sched_resval frees the value;
|
|
put_resvals repeated index; resreq/schedobj/pend-reason/errmsg frees;
|
|
write_snapshot_entry frees its snapshot, save side strdup()s).
|
|
Exit-time leak on the reproduction replay: 17,041,405 bytes -> 0.
|
|
Not fixed here (separate): normal mode never frees decision structs;
|
|
queues rejected on config re-apply and a config replaced by CONFIG_PUSH
|
|
are not freed; select parse errors inside the resreq grammar are ignored.
|
|
|
|
## Verification
|
|
- Test: test_cli_batch_replay_exits_clean_under_leak_detection in src/scheduler/testsuite/test_sim_discrete_clock.c (real binary, two-session fixture, --sim-verify --sim-output, detect_leaks=1): RED before (exit 1), GREEN after (exit 0)
|
|
- Suites: ASAN debug build (src/xce.build.discrete), `meson test --no-suite xceadmin`: 117/117 OK, 0 sanitizer reports (2026-09-29)
|
|
- Live: 3-round sequential HA soak on rebuilt ASAN binaries (/home/claude/tmp/xce-bug1884-soak): PASS, zero job loss, SIMVERIFY 4 files 0 mismatches, no sanitizer report in the live scheduler's stderr; all 13 recordings (this soak + xce-alloceq-53 + xce-ha-dc-soak) replay with detect_leaks=1: exit 0, 0 mismatches, no leak report
|
|
|
|
## References
|
|
- Related: bug-1632, bug-1763 (sim picker/allocator threads never joined)
|
|
- run-ha-soak.sh replay gate already runs `--sim` with `ASAN_OPTIONS=detect_leaks=0`
|