lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

chore(buglog): bug-1884 fixed; log bug-1885 (decision leak), bug-1886 (queue/config leak), bug-1887 (resreq parse errors ignored)

Browse source 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-29 09:49:59 +00:00
parent 90b99d91e9
commit 5b4d61180b
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
5 changed files with 387 additions and 41 deletions
Download patch file Download diff file Expand all files Collapse all files

178
.wolf/buglog.json
View file

@ -16112,7 +16112,9 @@
"fix_commits": [
"aa7fc65a"
],
"record": ".wolf/bugs/bug-1878.md"
"record": ".wolf/bugs/bug-1878.md",
"gitea_issue": 329,
"gitea_synced_status": "fixed"
},
{
"id": "bug-1879",
@ -16141,7 +16143,9 @@
"fix_commits": [
"aa7fc65a"
],
"record": ".wolf/bugs/bug-1879.md"
"record": ".wolf/bugs/bug-1879.md",
"gitea_issue": 330,
"gitea_synced_status": "fixed"
},
{
"id": "bug-1880",
@ -16170,13 +16174,15 @@
"bug-1845"
],
"fix_commits": [],
"record": ".wolf/bugs/bug-1880.md"
"record": ".wolf/bugs/bug-1880.md",
"gitea_issue": 331,
"gitea_synced_status": "investigating"
},
{
"id": "bug-1881",
"status": "partial",
"status_updated": "2026-09-28",
"status_note": "mismatch lines now carry a first-difference diff (24b05718); live default-speed replay proof waits on spec scheduler-sim-discrete-clock (task 5.3)",
"status": "fixed",
"status_updated": "2026-09-29",
"status_note": "",
"component": "scheduler",
"severity": "wrong-result",
"timestamp": "2026-09-28T07:26:09Z",
@ -16185,7 +16191,7 @@
"error_message": "sim_log_mismatch prints identical expected/actual, hiding the field that differs\n\n**Observed:**\n```\n... MISMATCH ... -- expected: DISPATCH jobid=0-12-1790578108-0[49] nhosts=1 -- actual: DISPATCH jobid=0-12-1790578108-0[49] nhosts=1\n```\n**Expected:** the message names the field(s) that differ, with both values.\n**Impact:** a mismatch cannot be diagnosed from the log (bug-1880: 43 such lines, none showing the difference).\n**Found by:** user reading the bug-1880 replay log, 2026-09-28\n\nUPDATE (2026-09-28): Fix planned in spec libxce-alloc-preason-equality (Requirement 3, Requirement 5.3, task 5): the libxce equality functions gain an optional why/whylen buffer describing the first difference, and sim_log_mismatch appends it to the MISMATCH line.",
"file": "src/scheduler/src/sim_verify.c",
"root_cause": "sim_log_mismatch() (src/scheduler/src/sim_verify.c) summarizes each decision as kind + jobid + nhosts, while sim_decision_eq() also compares every host's allocations (resname, value, duration, reserve), so a difference there prints two identical summaries.",
"fix": "Spec libxce-alloc-preason-equality (commits 8f25c1c3..dcf3acf3). sim_decision_eq() now compares DISPATCH decisions with libxce's public job_alloc_eq() and returns a caller-owned first-difference description (`char** why`) naming the field path and both values; sim_log_mismatch() appends it as ` -- diff: <why>`, and the replay loop in sim_loop.c passes and frees it (fixed text \"no live decision\" for leftover recorded decisions) — 24b05718. job_alloc_eq() also compares `from`, so the scheduler now stamps every allocation's `from` with the deciding session's start (scheduler.c sched_make_primary_alloc_, c1960457) and the normal-mode recorder takes beginRound from the same reading (driver.c, 2d7ba660), making `from` reproducible on replay.\n\nRemaining: the simulator's own session-begin clock read (sim_loop.c) still drifts at --sim-speed >= 1, so a live default-settings replay cannot yet be shown to report zero `from` mismatches. That is task 5.3, blocked on spec scheduler-sim-discrete-clock.\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why",
"fix": "Spec libxce-alloc-preason-equality (commits 8f25c1c3..dcf3acf3). sim_decision_eq() now compares DISPATCH decisions with libxce's public job_alloc_eq() and returns a caller-owned first-difference description (`char** why`) naming the field path and both values; sim_log_mismatch() appends it as ` -- diff: <why>`, and the replay loop in sim_loop.c passes and frees it (fixed text \"no live decision\" for leftover recorded decisions) — 24b05718. job_alloc_eq() also compares `from`, so the scheduler now stamps every allocation's `from` with the deciding session's start (scheduler.c sched_make_primary_alloc_, c1960457) and the normal-mode recorder takes beginRound from the same reading (driver.c, 2d7ba660), making `from` reproducible on replay.\n\nThe remaining gap (the simulator's own session-begin clock read drifting at --sim-speed >= 1) was closed by spec scheduler-sim-discrete-clock (4f653d8f..d9bc36cd: the simulator clock is pinned to recorded timestamps, --sim-speed is a no-op), so a replayed session starts at exactly the recorded beginRound and every replayed `from` matches; task 5.3 adds the proof (4e57bde6).\n\nVerification:\n- Test: test_log_mismatch_appends_provided_description and test_verify_against_altered_copy_reports_exact_mismatch (asserts ` -- diff: main_reason: 99 != 0`) plus test_verify_leftover_recorded_decision_names_missing_side in src/scheduler/testsuite/test_sim_verify.c: RED before 24b05718 (no diff text on the MISMATCH line), GREEN after. test_sim_replay_reproduces_recorded_alloc_from in src/scheduler/testsuite/test_sim_live_allocation.c (4e57bde6): GREEN at default settings; a +0.5 s clock mutant at sim_loop.c's sim_advance_virtual_clock(sim, begin_round) makes it FAIL\n- Suites: ASAN debug build (src/xce.build.discrete), `meson test --no-suite xceadmin`: 117/117 OK, 0 Unity :FAIL lines, 0 sanitizer reports (2026-09-29)\n- Live: 2-round sequential HA soak (/home/claude/tmp/xce-alloceq-53, PASS, zero job loss) recorded 3 clean .replay files with 11 DISPATCH decisions; each replayed with `xce_scheduler --sim --sim-verify` and no --sim-speed (ASAN_OPTIONS=detect_leaks=0): rc=0, 0 MISMATCH lines",
"tags": [
"scheduler",
"sim",
@ -16193,8 +16199,15 @@
"logging"
],
"related_bugs": [],
"fix_commits": [],
"record": ".wolf/bugs/bug-1881.md"
"fix_commits": [
"24b05718",
"c1960457",
"2d7ba660",
"4e57bde6"
],
"record": ".wolf/bugs/bug-1881.md",
"gitea_issue": 332,
"gitea_synced_status": "fixed"
},
{
"id": "bug-1882",
@ -16223,7 +16236,152 @@
"fix_commits": [
"f9146864"
],
"record": ".wolf/bugs/bug-1882.md"
"record": ".wolf/bugs/bug-1882.md",
"gitea_issue": 333,
"gitea_synced_status": "fixed"
},
{
"id": "bug-1883",
"status": "fixed",
"status_updated": "2026-09-29",
"status_note": "",
"component": "scheduler",
"severity": "wrong-result",
"timestamp": "2026-09-29T04:07:19Z",
"last_seen": "2026-09-29T04:07:19Z",
"occurrences": 1,
"error_message": "xce_scheduler seqjob reuses a stale per-host want from an earlier COMPOUND/ALTER primary\n\n**Observed:** (by code reading; no failing run captured yet)\nWhen a COMPOUND or ALTER job has a `maxTasks == 1` primary after another\nprimary in the same attempt, seqjob's host search for the later primary\nstarts from the per-host want (`ctx->alloc_req.host`) left behind by the\nearlier primary, with `slots:1` appended rather than replacing it:\n\n```\n.host on entry to seqjob host loop (2nd+ primary, shared ctx)\n = <previous primary's last per-host want, e.g. slots:N + rusage> + slots:1\n```\n\n- `plugin_prepare_allocation()` resets only `alloc_req.shared` in the\n global pass (`src/scheduler/src/plugin.c:617-619`), never `.host`.\n- `seqjob_prepare_allocation()` appends `slots=1`\n (`src/scheduler/src/plugins/seqjob_allocator.c:118`), and\n `xvec_idval_store` appends without dedup (`src/lib/xvec.c:247-270`).\n- The host iterator pre-filters candidates against this stale vector\n (`src/scheduler/src/host_iter.c:136,158`), silently dropping hosts that\n would satisfy the later primary's real `{slots:1}` want.\n- A global limit on `slots` reads the first match via `get_resval_by_id`\n (`src/scheduler/src/resource.c:951-968`), so it sees the stale N, not 1.\n\nparjob does not have this problem because it clears `.host` per bucket\n(`src/scheduler/src/plugins/parjob_allocator.c:478-479`).\n\n**Expected:** every primary's per-host want is computed from that primary's\nown resource request only (the bug-1768 principle), so the later seqjob\nprimary is pre-filtered and limit-checked against `{slots:1}` plus its own\nrusage.\n**Impact:** wrong result. A later single-task primary of a COMPOUND/ALTER job\ncan pend (or pick a later host) although a host that fits it is free; pend\nreasons can list hosts filtered on resources the primary never asked for.\nPlain single-primary jobs are unaffected (fresh ctx).\n**Found by:** review. Discovery/requirements research for spec\n`scheduler-allocator-runner` (2026-09-28). That spec deliberately keeps this\nbehaviour (Requirement 3 AC 6) so the refactor stays behaviour-preserving;\nfix here, separately, with its own test.",
"file": "src/scheduler/src/plugins/seqjob_allocator.c, src/scheduler/src/plugin.c, src/scheduler/src/schedobj.c, src/scheduler/src/host_iter.c",
"root_cause": "`plugin_prepare_allocation()` (src/scheduler/src/plugin.c) reset only\n`ctx->alloc_req.shared` in its job-level pass (`ctx->worker == NULL`), while\nseqjob's and parjob's prepare hooks append this primary's job-level slots\nwant into `ctx->alloc_req.host` with the append-only `xvec_idval_store()`.\nA COMPOUND/ALTER job's primaries share one ctx (`sched_ctx_set_primary()`\nruns the job-level pass per primary), so `.host` still held the previous\nprimary's last per-host want. The host iterator pre-filtered on it\n(`host_iter.c`), and a global limit on slots read its first `slots` entry.",
"fix": "`plugin_prepare_allocation()`'s job-level pass now resets `.host` as well\nas `.shared`, so each primary's pass starts from its own want only\n(`c10f32cc`). The per-host pass already reset `.host`; parjob's per-bucket\nreset is unchanged.\n\nVerification:\n- Test: test_compound_later_seqjob_primary_ignores_previous_want in src/scheduler/testsuite/test_parjob_e2e_multislot.c (hosts A=4 slots, B=1 slot; COMPOUND `4*{..} + 1*{..}` through the real scheduler dispatch): RED before the fix (PEND, \"Expected 1 Was 2\"), GREEN after (DISPATCH, 4 slots on A + 1 on B)\n- Suites: ASAN debug build (src/xce.build.discrete), `meson test --no-suite xceadmin`: 117/117 OK, 0 Unity :FAIL lines, 0 sanitizer reports (2026-09-29)\n- Live: n/a -- no soak or deploy script submits COMPOUND/ALTER jobs; the test drives the real picker/allocator dispatch path end to end",
"tags": [
"scheduler",
"allocation",
"compound",
"seqjob"
],
"related_bugs": [
"bug-1768",
"bug-1766"
],
"fix_commits": [
"c10f32cc"
],
"record": ".wolf/bugs/bug-1883.md",
"gitea_issue": 334,
"gitea_synced_status": "fixed"
},
{
"id": "bug-1884",
"status": "fixed",
"status_updated": "2026-09-29",
"status_note": "",
"component": "scheduler",
"severity": "leak",
"timestamp": "2026-09-29T08:20:03Z",
"last_seen": "2026-09-29T08:20:03Z",
"occurrences": 1,
"error_message": "standalone ASAN xce_scheduler --sim exits 1: LeakSanitizer reports 17 MB still allocated at exit\n\n**Observed:** an ASAN build of `xce_scheduler --sim=<file> --sim-verify`\nreplays the whole recording (0 MISMATCH lines) and then exits 1 because\nLeakSanitizer reports at exit:\n```\n==2234882==ERROR: LeakSanitizer: detected memory leaks\n...\nSUMMARY: AddressSanitizer: 17041405 byte(s) leaked in 1187 allocation(s).\n```\nOf the 17,041,405 bytes, 17,036,261 are indirect and 5,144 direct.\nGrouped by first scheduler/lib frame (bytes, leak groups):\n- `kvtab_init` (lib/kv.c:106): 16,941,432 in 15 -- core kv tables, reachable only from leaked roots\n- `sim_process_replay_event_message` (sim_loop.c:403): 49,688 in 54 -- decoded replay events (e.g. `decode_JobStateChangeEvent_ptr`)\n- `xkvmgr_add` (lib/kvmgr.c:114): 21,376 in 9\n- `sim_locate_and_load_snapshot` (sim.c:224): 14,347 in 106 -- snapshot decode (`decode_Conf4Sched` / `decode_QueueConf_list`)\n- `put_resvals` (resource.c:99) via limit.c `update_limit_res_value` and `sched_update_res_accounts`: 2,048 in 2\n- smaller: `parse_resreq`, `core_build_job`, `sched_ctx_init`, `match_job_host_limit_`\n\nThe indirect bulk fits the simulator never tearing down its core at exit\n(its picker/allocator threads are never joined, see bug-1632/bug-1763);\nthe direct groups may include real per-event leaks.\n\n**Expected:** a clean replay exits 0 under ASAN with LeakSanitizer enabled,\nor the exit-time teardown frees what the driver owns.\n**Impact:** leak; for scripts, any ASAN `--sim` replay must set\n`ASAN_OPTIONS=detect_leaks=0` or its exit code is always 1, which masks real\nfailures and hides any genuine per-event leak among the exit-time noise.\nRelease builds are unaffected at exit; a per-event leak would still grow a\nlong replay's memory.\n**Found by:** spec task (libxce-alloc-preason-equality 5.3, live\ndefault-settings replay, 2026-09-29)",
"file": "src/scheduler/src/sim_loop.c, src/scheduler/src/sim.c, src/scheduler/src/plugins/limit.c, src/scheduler/src/main.c",
"root_cause": "Nothing tore the simulator down (threads never joined, core never\nfinalized), and teardown then exposed a chain of ownership gaps, several of\nwhich also leak at runtime in normal mode:\n- xce_sched_core_fini() never destroyed the plugin/job/worker/queue/user/\n project registries, the shared-value arrays, the config or the\n pend_updates queues; kill_sched_job/queue/worker are shallow by design\n (a failed key_add() runs them and callers keep using the value).\n- sim_alloc.c's allocator stand-ins ran a full xce_sched_core_init(),\n re-initializing the process-wide string and selector tables.\n- plugin_unregister() key_del()'d from the registry being destroyed; the\n limit plugin's usage-instance table used kill_limit on xce_vres_t;\n add_vres() put a bare calloc() into the resobj registry.\n- Replay-event payloads were never released after the push (\"accepted\n leak\"); the start snapshot's non-config remainder was never freed.\n- kill_sched_resval() never freed the value struct (free_Value() frees only\n a string's buffer) -- every replaced value leaked, WORKERS_PUSH updates\n included; put_resvals() created a second value for a repeated index.\n- dup_val() tested which == XCE_VAL_STRING (0 = Value_u_ivalue32): real\n strings were shallow-copied, a nonzero int32 went to makeString().\n- The satisfy checks re-initialized the ctx's live unmeet vector per\n candidate host; replaced pend reasons, old parsed resreqs (JOB_UPDATE,\n duplicate JOBS_PUSH), the parser's error message and each written\n snapshot were never freed; the simulator never freed decision structs.",
"fix": "Six commits: 9b61fd99 (sim teardown, core registries, allocator stand-ins,\nplugin_unregister, kill_vres), c89e4e45 (replay-event payload release),\n8134b75b (core owns/frees its config; start snapshot freed), ebfe4794 (sim\ndecisions, fcfs plugin, pend_updates), c1cd8f91 (collect-then-free\nteardown for job/worker/queue/resource tables with shallow killers kept;\ndup_val; unmeet; add_vres via makeResource; plugin_register_core rejects\nduplicates up front), 90b99d91 (kill_sched_resval frees the value;\nput_resvals repeated index; resreq/schedobj/pend-reason/errmsg frees;\nwrite_snapshot_entry frees its snapshot, save side strdup()s).\nExit-time leak on the reproduction replay: 17,041,405 bytes -> 0.\nNot fixed here (separate): normal mode never frees decision structs;\nqueues rejected on config re-apply and a config replaced by CONFIG_PUSH\nare not freed; select parse errors inside the resreq grammar are ignored.\n\nVerification:\n- Test: test_cli_batch_replay_exits_clean_under_leak_detection in src/scheduler/testsuite/test_sim_discrete_clock.c (real binary, two-session fixture, --sim-verify --sim-output, detect_leaks=1): RED before (exit 1), GREEN after (exit 0)\n- Suites: ASAN debug build (src/xce.build.discrete), `meson test --no-suite xceadmin`: 117/117 OK, 0 sanitizer reports (2026-09-29)\n- Live: 3-round sequential HA soak on rebuilt ASAN binaries (/home/claude/tmp/xce-bug1884-soak): PASS, zero job loss, SIMVERIFY 4 files 0 mismatches, no sanitizer report in the live scheduler's stderr; all 13 recordings (this soak + xce-alloceq-53 + xce-ha-dc-soak) replay with detect_leaks=1: exit 0, 0 mismatches, no leak report",
"tags": [
"scheduler",
"sim",
"asan",
"lsan",
"leak"
],
"related_bugs": [
"bug-1632",
"bug-1763"
],
"fix_commits": [
"9b61fd99",
"c89e4e45",
"8134b75b",
"ebfe4794",
"c1cd8f91",
"90b99d91"
],
"record": ".wolf/bugs/bug-1884.md",
"gitea_issue": 335,
"gitea_synced_status": "open"
},
{
"id": "bug-1885",
"status": "open",
"status_updated": "2026-09-29",
"status_note": "newly logged",
"component": "scheduler",
"severity": "leak",
"timestamp": "2026-09-29T09:48:44Z",
"last_seen": "2026-09-29T09:48:44Z",
"occurrences": 1,
"error_message": "scheduler normal mode never frees decision structs: 64 bytes leaked per published decision\n\n**Observed:** (code reading) the normal-mode round loop in\n`xce_sched_driver_run_normal_mode_round()` (src/scheduler/src/driver.c)\ndrains each `xce_decision_t*` from `core->decisions`, records it and passes\nit to `on_decision`, but never frees the struct `make_*_decision()`\n(decision.c) calloc()s:\n```\ncbuf_get(&(sched->core.decisions), &decision);\n... sched->core.on_decision(sched->core.driver_ctx, decision);\n/* decision never freed */\n```\n\n**Expected:** the round loop frees the decision struct (not its alloc/pend reason, which the job owns) once published.\n**Impact:** leak of 64 bytes per dispatch/pend decision for the life of a normal-mode scheduler. Simulation mode was fixed in bug-1884 (ebfe4794). Three tests (test_core_full_cycle.c, test_core_session.c, test_snapshot_save_load.c) keep decision pointers from their callback, so the fix changes the callback contract (\"valid only during the call\") and those tests.\n**Found by:** review (while fixing bug-1884, 2026-09-29)",
"file": "src/scheduler/src/driver.c, src/scheduler/src/decision.c",
"root_cause": "TODO: required for fixed/partial",
"fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why",
"tags": [
"scheduler",
"leak",
"decision"
],
"related_bugs": [
"bug-1884"
],
"fix_commits": [],
"record": ".wolf/bugs/bug-1885.md"
},
{
"id": "bug-1886",
"status": "open",
"status_updated": "2026-09-29",
"status_note": "newly logged",
"component": "scheduler",
"severity": "leak",
"timestamp": "2026-09-29T09:48:47Z",
"last_seen": "2026-09-29T09:48:47Z",
"occurrences": 1,
"error_message": "scheduler leaks the rejected queue on each config re-apply and the old config on each CONFIG_PUSH\n\n**Observed:** (code reading)\n- `sched_init_queues()` (src/scheduler/src/scheduler.c) builds a new\n `xce_sched_queue_t` for every configured queue on each config apply and\n `sched_add_queue()`s it; for an existing name the add fails and the new\n queue (with its interned name/resreq) is still handed to the picker but\n owned by nothing.\n- `CONFIG_PUSH` (core_events.c) sets `core->config = config` without freeing\n the previous config.\n\n**Expected:** re-applied queues update the existing queue in place (as `core_upsert_queue()` does for QUEUE_UPDATE), and a replaced config is freed once nothing points into it.\n**Impact:** leak per config push/re-apply (one queue per configured queue plus one whole `xce_conf4sched_t`). Freeing the old config needs care: the picker and limit plugin may keep pointers into it. The queue table's datakiller must stay a no-op for as long as `sched_init_queues()` keeps using a rejected queue (bug-1884 found a use-after-free when it freed).\n**Found by:** review (while fixing bug-1884, 2026-09-29)",
"file": "src/scheduler/src/scheduler.c, src/scheduler/src/core_events.c",
"root_cause": "TODO: required for fixed/partial",
"fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why",
"tags": [
"scheduler",
"leak",
"config"
],
"related_bugs": [
"bug-1884"
],
"fix_commits": [],
"record": ".wolf/bugs/bug-1886.md"
},
{
"id": "bug-1887",
"status": "open",
"status_updated": "2026-09-29",
"status_note": "newly logged",
"component": "lib",
"severity": "wrong-result",
"timestamp": "2026-09-29T09:48:49Z",
"last_seen": "2026-09-29T09:48:49Z",
"occurrences": 1,
"error_message": "resreq parser ignores a primary's select/rusage parse error and accepts the job without that clause\n\n**Observed:** (code reading) the `primary_resreq` action in\nsrc/lib/xce_resreq.peg calls\n`parse_resreq_primary(primary, $0, auxil->resmgr, &(auxil->errmsg))`\nand ignores its return value; nothing sets `has_error`, so\n`parse_resreq()` returns success. A primary whose select or rusage clause\nfails to parse is kept without that clause (select NULL / rusages empty),\ni.e. the job is scheduled as if it had no such constraint. While fixing\nbug-1884 a LeakSanitizer report showed such an error message being\nproduced during a normal replay fixture (its job's resreq parsed \"OK\").\n\n**Expected:** a primary sub-parse error makes `parse_resreq()` fail with that message (the job is rejected/pended with a parse error), not silently dropped.\n**Impact:** wrong result: a job with a malformed select (e.g. a typo) may dispatch onto hosts its select would have excluded.\n**Found by:** review (while fixing bug-1884, 2026-09-29)",
"file": "src/lib/xce_resreq.peg, src/lib/resreq.c",
"root_cause": "TODO: required for fixed/partial",
"fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why",
"tags": [
"lib",
"resreq",
"parser"
],
"related_bugs": [
"bug-1884"
],
"fix_commits": [],
"record": ".wolf/bugs/bug-1887.md"
}
]
}

73
.wolf/bugs/bug-1884.md
View file

@ -1,8 +1,8 @@
---
id: "bug-1884"
status: "partial"
status: "fixed"
status_updated: "2026-09-29"
status_note: "part 1 (9b61fd99): sim teardown + core registries, 17.2 MB -> 75 KB at exit; part 2: replay-event payload release, core->config, job/resource destructors, snapshot-load data"
status_note: ""
component: "scheduler"
severity: "leak"
timestamp: "2026-09-29T08:20:03Z"
@ -11,6 +11,7 @@ occurrences: 1
files: ["src/scheduler/src/sim_loop.c", "src/scheduler/src/sim.c", "src/scheduler/src/plugins/limit.c", "src/scheduler/src/main.c"]
tags: ["scheduler", "sim", "asan", "lsan", "leak"]
related: ["bug-1632", "bug-1763"]
fix_commits: ["9b61fd99", "c89e4e45", "8134b75b", "ebfe4794", "c1cd8f91", "90b99d91"]
gitea_issue: 335
gitea_synced_status: "open"
gitea_synced_hash: "097b7ddb90dd"
@ -61,44 +62,54 @@ default-settings replay, 2026-09-29)
## Reproduction
1. `bash src/tools/xce-build.sh scheduler asan` (or any ASAN build dir)
2. `src/xce.build/scheduler/xce_scheduler --sim=/home/claude/tmp/bug-1884/scheduler-replay-20260929-064412.replay --sim-verify --logfile /tmp/sim.log; echo rc=$?`
3. Observe `rc=1` and the LeakSanitizer SUMMARY on stderr. With `ASAN_OPTIONS=detect_leaks=0` the same run exits 0.
3. Observe `rc=1` and the LeakSanitizer SUMMARY on stderr (before the fix). With `ASAN_OPTIONS=detect_leaks=0` the same run exits 0.
- Frequency: deterministic (2/2 runs, identical totals, 2026-09-29); also seen on every recording of /home/claude/tmp/xce-ha-dc-soak
- Evidence: /home/claude/tmp/bug-1884/lsan-report.txt (full report), /home/claude/tmp/bug-1884/scheduler-replay-20260929-064412.replay
## Root cause
Nothing tears the simulator down: sched_sim_driver_run() never joined its
picker/allocator threads and main() exited without finalizing the core. On
top of that, several ownership gaps only show once teardown runs:
Nothing tore the simulator down (threads never joined, core never
finalized), and teardown then exposed a chain of ownership gaps, several of
which also leak at runtime in normal mode:
- xce_sched_core_fini() never destroyed the plugin/job/worker/queue/user/
project registries or the shared-resource arrays that init creates.
- sim_alloc.c's two sched_make_allocs() stand-ins each ran a full
xce_sched_core_init(): a whole unused core, plus a re-init of the
process-wide string-intern and selector tables (two 8 MB tables leaked).
- plugin_unregister() (the plugin registry's datakiller, never run before)
key_del()'d from the registry being destroyed and skipped fini_fn for
plugins registered with plugin_register_core().
- The limit plugin's usage-instance table had kill_limit as datakiller for
xce_vres_t records (bad free on destroy); kill_vres() was an empty stub.
- Remaining (part 2): replay-event payloads are never released after
xce_core_push_events() (sim_loop.c: documented "accepted leak"), unlike
driver.c's release_translated_payload(); the adopted core->config is never
freed (and a CONFIG_PUSH replaces it without freeing the old one);
kill_sched_job() is shallow; resmgr has no datakiller; snapshot-load data,
the builtin_fcfs plugin struct and some allocator scratch are not freed.
project registries, the shared-value arrays, the config or the
pend_updates queues; kill_sched_job/queue/worker are shallow by design
(a failed key_add() runs them and callers keep using the value).
- sim_alloc.c's allocator stand-ins ran a full xce_sched_core_init(),
re-initializing the process-wide string and selector tables.
- plugin_unregister() key_del()'d from the registry being destroyed; the
limit plugin's usage-instance table used kill_limit on xce_vres_t;
add_vres() put a bare calloc() into the resobj registry.
- Replay-event payloads were never released after the push ("accepted
leak"); the start snapshot's non-config remainder was never freed.
- kill_sched_resval() never freed the value struct (free_Value() frees only
a string's buffer) -- every replaced value leaked, WORKERS_PUSH updates
included; put_resvals() created a second value for a repeated index.
- dup_val() tested which == XCE_VAL_STRING (0 = Value_u_ivalue32): real
strings were shallow-copied, a nonzero int32 went to makeString().
- The satisfy checks re-initialized the ctx's live unmeet vector per
candidate host; replaced pend reasons, old parsed resreqs (JOB_UPDATE,
duplicate JOBS_PUSH), the parser's error message and each written
snapshot were never freed; the simulator never freed decision structs.
## Fix
Part 1 (9b61fd99): sched_sim_driver_fini() -- cancel/join the threads the
way sched_fini() does (thread ids now in core.jobpicker/allocator), close
the files, finalize the core; main() calls it after a clean run. Core fini
destroys every registry init makes. The allocator stand-ins set up only
lock, nextid and the aliased resmgr. plugin_unregister() only runs fini_fn.
Usage instances use kill_vres(), which frees the record. Leak at exit on the
reproduction replay: 17,041,405 -> 75,495 bytes.
Six commits: 9b61fd99 (sim teardown, core registries, allocator stand-ins,
plugin_unregister, kill_vres), c89e4e45 (replay-event payload release),
8134b75b (core owns/frees its config; start snapshot freed), ebfe4794 (sim
decisions, fcfs plugin, pend_updates), c1cd8f91 (collect-then-free
teardown for job/worker/queue/resource tables with shallow killers kept;
dup_val; unmeet; add_vres via makeResource; plugin_register_core rejects
duplicates up front), 90b99d91 (kill_sched_resval frees the value;
put_resvals repeated index; resreq/schedobj/pend-reason/errmsg frees;
write_snapshot_entry frees its snapshot, save side strdup()s).
Exit-time leak on the reproduction replay: 17,041,405 bytes -> 0.
Not fixed here (separate): normal mode never frees decision structs;
queues rejected on config re-apply and a config replaced by CONFIG_PUSH
are not freed; select parse errors inside the resreq grammar are ignored.
## Verification
- Test: TODO: name + file; RED before the fix, GREEN after
- Suites: TODO: suite results and build profile
- Live: TODO: soak/deploy run, or n/a and why
- Test: test_cli_batch_replay_exits_clean_under_leak_detection in src/scheduler/testsuite/test_sim_discrete_clock.c (real binary, two-session fixture, --sim-verify --sim-output, detect_leaks=1): RED before (exit 1), GREEN after (exit 0)
- Suites: ASAN debug build (src/xce.build.discrete), `meson test --no-suite xceadmin`: 117/117 OK, 0 sanitizer reports (2026-09-29)
- Live: 3-round sequential HA soak on rebuilt ASAN binaries (/home/claude/tmp/xce-bug1884-soak): PASS, zero job loss, SIMVERIFY 4 files 0 mismatches, no sanitizer report in the live scheduler's stderr; all 13 recordings (this soak + xce-alloceq-53 + xce-ha-dc-soak) replay with detect_leaks=1: exit 0, 0 mismatches, no leak report
## References
- Related: bug-1632, bug-1763 (sim picker/allocator threads never joined)

60
.wolf/bugs/bug-1885.md Normal file
View file

@ -0,0 +1,60 @@
---
id: "bug-1885"
status: "open"
status_updated: "2026-09-29"
status_note: "newly logged"
component: "scheduler"
severity: "leak"
timestamp: "2026-09-29T09:48:44Z"
last_seen: "2026-09-29T09:48:44Z"
occurrences: 1
files: ["src/scheduler/src/driver.c", "src/scheduler/src/decision.c"]
tags: ["scheduler", "leak", "decision"]
related: ["bug-1884"]
---
# bug-1885: scheduler normal mode never frees decision structs: 64 bytes leaked per published decision
## Description
**Observed:** (code reading) the normal-mode round loop in
`xce_sched_driver_run_normal_mode_round()` (src/scheduler/src/driver.c)
drains each `xce_decision_t*` from `core->decisions`, records it and passes
it to `on_decision`, but never frees the struct `make_*_decision()`
(decision.c) calloc()s:
```
cbuf_get(&(sched->core.decisions), &decision);
... sched->core.on_decision(sched->core.driver_ctx, decision);
/* decision never freed */
```
**Expected:** the round loop frees the decision struct (not its alloc/pend reason, which the job owns) once published.
**Impact:** leak of 64 bytes per dispatch/pend decision for the life of a normal-mode scheduler. Simulation mode was fixed in bug-1884 (ebfe4794). Three tests (test_core_full_cycle.c, test_core_session.c, test_snapshot_save_load.c) keep decision pointers from their callback, so the fix changes the callback contract ("valid only during the call") and those tests.
**Found by:** review (while fixing bug-1884, 2026-09-29)
## Environment
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
- Build: asan (src/xce.build.discrete)
- Code: scheduler-framework @ 90b99d91-dirty
- Deployment: n/a (found by code reading while fixing bug-1884; not observed on a deployment)
## Reproduction
1. Read `src/scheduler/src/driver.c` round loop (the `cbuf_get(&(sched->core.decisions), ...)` loop) and `src/scheduler/src/decision.c` `make_dispatch_decision()`/`make_pend_decision()`.
2. A live check: run a normal-mode scheduler under ASAN with detect_leaks=1 through a few rounds and stop it cleanly; expect `make_*_decision` direct leaks.
- Frequency: deterministic by code reading (one struct per decision)
- Evidence: not kept; code reading
## Root cause
TODO: required for fixed/partial
## Fix
TODO: required for fixed/partial
## Verification
- Test: TODO: name + file; RED before the fix, GREEN after
- Suites: TODO: suite results and build profile
- Live: TODO: soak/deploy run, or n/a and why
## References
- Related: bug-1884

58
.wolf/bugs/bug-1886.md Normal file
View file

@ -0,0 +1,58 @@
---
id: "bug-1886"
status: "open"
status_updated: "2026-09-29"
status_note: "newly logged"
component: "scheduler"
severity: "leak"
timestamp: "2026-09-29T09:48:47Z"
last_seen: "2026-09-29T09:48:47Z"
occurrences: 1
files: ["src/scheduler/src/scheduler.c", "src/scheduler/src/core_events.c"]
tags: ["scheduler", "leak", "config"]
related: ["bug-1884"]
---
# bug-1886: scheduler leaks the rejected queue on each config re-apply and the old config on each CONFIG_PUSH
## Description
**Observed:** (code reading)
- `sched_init_queues()` (src/scheduler/src/scheduler.c) builds a new
`xce_sched_queue_t` for every configured queue on each config apply and
`sched_add_queue()`s it; for an existing name the add fails and the new
queue (with its interned name/resreq) is still handed to the picker but
owned by nothing.
- `CONFIG_PUSH` (core_events.c) sets `core->config = config` without freeing
the previous config.
**Expected:** re-applied queues update the existing queue in place (as `core_upsert_queue()` does for QUEUE_UPDATE), and a replaced config is freed once nothing points into it.
**Impact:** leak per config push/re-apply (one queue per configured queue plus one whole `xce_conf4sched_t`). Freeing the old config needs care: the picker and limit plugin may keep pointers into it. The queue table's datakiller must stay a no-op for as long as `sched_init_queues()` keeps using a rejected queue (bug-1884 found a use-after-free when it freed).
**Found by:** review (while fixing bug-1884, 2026-09-29)
## Environment
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
- Build: asan (src/xce.build.discrete)
- Code: scheduler-framework @ 90b99d91-dirty
- Deployment: n/a (found by code reading while fixing bug-1884; not observed on a deployment)
## Reproduction
1. Read `sched_init_queues()` in src/scheduler/src/scheduler.c and the `XCE_CORE_EVENT_CONFIG_PUSH` case in src/scheduler/src/core_events.c.
2. Test-level check: `test_config_push_deferred.c` applies a config twice; with a freeing queue killer ASAN reports a heap-use-after-free in `builtin_fcfs_on_queue_update` (seen 2026-09-29 while fixing bug-1884).
- Frequency: deterministic on every config re-apply
- Evidence: not kept; code reading plus the test_config_push_deferred UAF seen with a freeing killer
## Root cause
TODO: required for fixed/partial
## Fix
TODO: required for fixed/partial
## Verification
- Test: TODO: name + file; RED before the fix, GREEN after
- Suites: TODO: suite results and build profile
- Live: TODO: soak/deploy run, or n/a and why
## References
- Related: bug-1884

59
.wolf/bugs/bug-1887.md Normal file
View file

@ -0,0 +1,59 @@
---
id: "bug-1887"
status: "open"
status_updated: "2026-09-29"
status_note: "newly logged"
component: "lib"
severity: "wrong-result"
timestamp: "2026-09-29T09:48:49Z"
last_seen: "2026-09-29T09:48:49Z"
occurrences: 1
files: ["src/lib/xce_resreq.peg", "src/lib/resreq.c"]
tags: ["lib", "resreq", "parser"]
related: ["bug-1884"]
---
# bug-1887: resreq parser ignores a primary's select/rusage parse error and accepts the job without that clause
## Description
**Observed:** (code reading) the `primary_resreq` action in
src/lib/xce_resreq.peg calls
`parse_resreq_primary(primary, $0, auxil->resmgr, &(auxil->errmsg))`
and ignores its return value; nothing sets `has_error`, so
`parse_resreq()` returns success. A primary whose select or rusage clause
fails to parse is kept without that clause (select NULL / rusages empty),
i.e. the job is scheduled as if it had no such constraint. While fixing
bug-1884 a LeakSanitizer report showed such an error message being
produced during a normal replay fixture (its job's resreq parsed "OK").
**Expected:** a primary sub-parse error makes `parse_resreq()` fail with that message (the job is rejected/pended with a parse error), not silently dropped.
**Impact:** wrong result: a job with a malformed select (e.g. a typo) may dispatch onto hosts its select would have excluded.
**Found by:** review (while fixing bug-1884, 2026-09-29)
## Environment
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
- Build: asan (src/xce.build.discrete)
- Code: scheduler-framework @ 90b99d91-dirty
- Deployment: n/a (found by code reading while fixing bug-1884; not observed on a deployment)
## Reproduction
1. Read the `primary_resreq` action in src/lib/xce_resreq.peg and `parse_resreq_primary()` in src/lib/resreq.c.
2. Unit check: `parse_resreq(&r, "select[type==] rusage[mem=1]", resmgr, &err)` -- expect rc < 0; observe the return value.
- Frequency: deterministic for any malformed select/rusage inside a primary
- Evidence: not kept; LeakSanitizer stack from the bug-1884 fixture (strdup in parse_select error path under core_build_job)
## Root cause
TODO: required for fixed/partial
## Fix
TODO: required for fixed/partial
## Verification
- Test: TODO: name + file; RED before the fix, GREEN after
- Suites: TODO: suite results and build profile
- Live: TODO: soak/deploy run, or n/a and why
## References
- Related: bug-1884