chore(buglog): bug-1884 fixed; log bug-1885 (decision leak), bug-1886 (queue/config leak), bug-1887 (resreq parse errors ignored)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-29 09:49:59 +00:00
parent 90b99d91e9
commit 5b4d61180b
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
5 changed files with 387 additions and 41 deletions

View file

@ -16112,7 +16112,9 @@
"fix_commits": [
"aa7fc65a"
],
"record": ".wolf/bugs/bug-1878.md"
"record": ".wolf/bugs/bug-1878.md",
"gitea_issue": 329,
"gitea_synced_status": "fixed"
},
{
"id": "bug-1879",
@ -16141,7 +16143,9 @@
"fix_commits": [
"aa7fc65a"
],
"record": ".wolf/bugs/bug-1879.md"
"record": ".wolf/bugs/bug-1879.md",
"gitea_issue": 330,
"gitea_synced_status": "fixed"
},
{
"id": "bug-1880",
@ -16170,13 +16174,15 @@
"bug-1845"
],
"fix_commits": [],
"record": ".wolf/bugs/bug-1880.md"
"record": ".wolf/bugs/bug-1880.md",
"gitea_issue": 331,
"gitea_synced_status": "investigating"
},
{
"id": "bug-1881",
"status": "partial",
"status_updated": "2026-09-28",
"status_note": "mismatch lines now carry a first-difference diff (24b05718); live default-speed replay proof waits on spec scheduler-sim-discrete-clock (task 5.3)",
"status": "fixed",
"status_updated": "2026-09-29",
"status_note": "",
"component": "scheduler",
"severity": "wrong-result",
"timestamp": "2026-09-28T07:26:09Z",
@ -16185,7 +16191,7 @@
"error_message": "sim_log_mismatch prints identical expected/actual, hiding the field that differs\n\n**Observed:**\n```\n... MISMATCH ... -- expected: DISPATCH jobid=0-12-1790578108-0[49] nhosts=1 -- actual: DISPATCH jobid=0-12-1790578108-0[49] nhosts=1\n```\n**Expected:** the message names the field(s) that differ, with both values.\n**Impact:** a mismatch cannot be diagnosed from the log (bug-1880: 43 such lines, none showing the difference).\n**Found by:** user reading the bug-1880 replay log, 2026-09-28\n\nUPDATE (2026-09-28): Fix planned in spec libxce-alloc-preason-equality (Requirement 3, Requirement 5.3, task 5): the libxce equality functions gain an optional why/whylen buffer describing the first difference, and sim_log_mismatch appends it to the MISMATCH line.",
"file": "src/scheduler/src/sim_verify.c",
"root_cause": "sim_log_mismatch() (src/scheduler/src/sim_verify.c) summarizes each decision as kind + jobid + nhosts, while sim_decision_eq() also compares every host's allocations (resname, value, duration, reserve), so a difference there prints two identical summaries.",
"fix": "Spec libxce-alloc-preason-equality (commits 8f25c1c3..dcf3acf3). sim_decision_eq() now compares DISPATCH decisions with libxce's public job_alloc_eq() and returns a caller-owned first-difference description (`char** why`) naming the field path and both values; sim_log_mismatch() appends it as ` -- diff: <why>`, and the replay loop in sim_loop.c passes and frees it (fixed text \"no live decision\" for leftover recorded decisions) — 24b05718. job_alloc_eq() also compares `from`, so the scheduler now stamps every allocation's `from` with the deciding session's start (scheduler.c sched_make_primary_alloc_, c1960457) and the normal-mode recorder takes beginRound from the same reading (driver.c, 2d7ba660), making `from` reproducible on replay.\n\nRemaining: the simulator's own session-begin clock read (sim_loop.c) still drifts at --sim-speed >= 1, so a live default-settings replay cannot yet be shown to report zero `from` mismatches. That is task 5.3, blocked on spec scheduler-sim-discrete-clock.\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why",
"fix": "Spec libxce-alloc-preason-equality (commits 8f25c1c3..dcf3acf3). sim_decision_eq() now compares DISPATCH decisions with libxce's public job_alloc_eq() and returns a caller-owned first-difference description (`char** why`) naming the field path and both values; sim_log_mismatch() appends it as ` -- diff: <why>`, and the replay loop in sim_loop.c passes and frees it (fixed text \"no live decision\" for leftover recorded decisions) — 24b05718. job_alloc_eq() also compares `from`, so the scheduler now stamps every allocation's `from` with the deciding session's start (scheduler.c sched_make_primary_alloc_, c1960457) and the normal-mode recorder takes beginRound from the same reading (driver.c, 2d7ba660), making `from` reproducible on replay.\n\nThe remaining gap (the simulator's own session-begin clock read drifting at --sim-speed >= 1) was closed by spec scheduler-sim-discrete-clock (4f653d8f..d9bc36cd: the simulator clock is pinned to recorded timestamps, --sim-speed is a no-op), so a replayed session starts at exactly the recorded beginRound and every replayed `from` matches; task 5.3 adds the proof (4e57bde6).\n\nVerification:\n- Test: test_log_mismatch_appends_provided_description and test_verify_against_altered_copy_reports_exact_mismatch (asserts ` -- diff: main_reason: 99 != 0`) plus test_verify_leftover_recorded_decision_names_missing_side in src/scheduler/testsuite/test_sim_verify.c: RED before 24b05718 (no diff text on the MISMATCH line), GREEN after. test_sim_replay_reproduces_recorded_alloc_from in src/scheduler/testsuite/test_sim_live_allocation.c (4e57bde6): GREEN at default settings; a +0.5 s clock mutant at sim_loop.c's sim_advance_virtual_clock(sim, begin_round) makes it FAIL\n- Suites: ASAN debug build (src/xce.build.discrete), `meson test --no-suite xceadmin`: 117/117 OK, 0 Unity :FAIL lines, 0 sanitizer reports (2026-09-29)\n- Live: 2-round sequential HA soak (/home/claude/tmp/xce-alloceq-53, PASS, zero job loss) recorded 3 clean .replay files with 11 DISPATCH decisions; each replayed with `xce_scheduler --sim --sim-verify` and no --sim-speed (ASAN_OPTIONS=detect_leaks=0): rc=0, 0 MISMATCH lines",
"tags": [
"scheduler",
"sim",
@ -16193,8 +16199,15 @@
"logging"
],
"related_bugs": [],
"fix_commits": [],
"record": ".wolf/bugs/bug-1881.md"
"fix_commits": [
"24b05718",
"c1960457",
"2d7ba660",
"4e57bde6"
],
"record": ".wolf/bugs/bug-1881.md",
"gitea_issue": 332,
"gitea_synced_status": "fixed"
},
{
"id": "bug-1882",
@ -16223,7 +16236,152 @@
"fix_commits": [
"f9146864"
],
"record": ".wolf/bugs/bug-1882.md"
"record": ".wolf/bugs/bug-1882.md",
"gitea_issue": 333,
"gitea_synced_status": "fixed"
},
{
"id": "bug-1883",
"status": "fixed",
"status_updated": "2026-09-29",
"status_note": "",
"component": "scheduler",
"severity": "wrong-result",
"timestamp": "2026-09-29T04:07:19Z",
"last_seen": "2026-09-29T04:07:19Z",
"occurrences": 1,
"error_message": "xce_scheduler seqjob reuses a stale per-host want from an earlier COMPOUND/ALTER primary\n\n**Observed:** (by code reading; no failing run captured yet)\nWhen a COMPOUND or ALTER job has a `maxTasks == 1` primary after another\nprimary in the same attempt, seqjob's host search for the later primary\nstarts from the per-host want (`ctx->alloc_req.host`) left behind by the\nearlier primary, with `slots:1` appended rather than replacing it:\n\n```\n.host on entry to seqjob host loop (2nd+ primary, shared ctx)\n = <previous primary's last per-host want, e.g. slots:N + rusage> + slots:1\n```\n\n- `plugin_prepare_allocation()` resets only `alloc_req.shared` in the\n global pass (`src/scheduler/src/plugin.c:617-619`), never `.host`.\n- `seqjob_prepare_allocation()` appends `slots=1`\n (`src/scheduler/src/plugins/seqjob_allocator.c:118`), and\n `xvec_idval_store` appends without dedup (`src/lib/xvec.c:247-270`).\n- The host iterator pre-filters candidates against this stale vector\n (`src/scheduler/src/host_iter.c:136,158`), silently dropping hosts that\n would satisfy the later primary's real `{slots:1}` want.\n- A global limit on `slots` reads the first match via `get_resval_by_id`\n (`src/scheduler/src/resource.c:951-968`), so it sees the stale N, not 1.\n\nparjob does not have this problem because it clears `.host` per bucket\n(`src/scheduler/src/plugins/parjob_allocator.c:478-479`).\n\n**Expected:** every primary's per-host want is computed from that primary's\nown resource request only (the bug-1768 principle), so the later seqjob\nprimary is pre-filtered and limit-checked against `{slots:1}` plus its own\nrusage.\n**Impact:** wrong result. A later single-task primary of a COMPOUND/ALTER job\ncan pend (or pick a later host) although a host that fits it is free; pend\nreasons can list hosts filtered on resources the primary never asked for.\nPlain single-primary jobs are unaffected (fresh ctx).\n**Found by:** review. Discovery/requirements research for spec\n`scheduler-allocator-runner` (2026-09-28). That spec deliberately keeps this\nbehaviour (Requirement 3 AC 6) so the refactor stays behaviour-preserving;\nfix here, separately, with its own test.",
"file": "src/scheduler/src/plugins/seqjob_allocator.c, src/scheduler/src/plugin.c, src/scheduler/src/schedobj.c, src/scheduler/src/host_iter.c",
"root_cause": "`plugin_prepare_allocation()` (src/scheduler/src/plugin.c) reset only\n`ctx->alloc_req.shared` in its job-level pass (`ctx->worker == NULL`), while\nseqjob's and parjob's prepare hooks append this primary's job-level slots\nwant into `ctx->alloc_req.host` with the append-only `xvec_idval_store()`.\nA COMPOUND/ALTER job's primaries share one ctx (`sched_ctx_set_primary()`\nruns the job-level pass per primary), so `.host` still held the previous\nprimary's last per-host want. The host iterator pre-filtered on it\n(`host_iter.c`), and a global limit on slots read its first `slots` entry.",
"fix": "`plugin_prepare_allocation()`'s job-level pass now resets `.host` as well\nas `.shared`, so each primary's pass starts from its own want only\n(`c10f32cc`). The per-host pass already reset `.host`; parjob's per-bucket\nreset is unchanged.\n\nVerification:\n- Test: test_compound_later_seqjob_primary_ignores_previous_want in src/scheduler/testsuite/test_parjob_e2e_multislot.c (hosts A=4 slots, B=1 slot; COMPOUND `4*{..} + 1*{..}` through the real scheduler dispatch): RED before the fix (PEND, \"Expected 1 Was 2\"), GREEN after (DISPATCH, 4 slots on A + 1 on B)\n- Suites: ASAN debug build (src/xce.build.discrete), `meson test --no-suite xceadmin`: 117/117 OK, 0 Unity :FAIL lines, 0 sanitizer reports (2026-09-29)\n- Live: n/a -- no soak or deploy script submits COMPOUND/ALTER jobs; the test drives the real picker/allocator dispatch path end to end",
"tags": [
"scheduler",
"allocation",
"compound",
"seqjob"
],
"related_bugs": [
"bug-1768",
"bug-1766"
],
"fix_commits": [
"c10f32cc"
],
"record": ".wolf/bugs/bug-1883.md",
"gitea_issue": 334,
"gitea_synced_status": "fixed"
},
{
"id": "bug-1884",
"status": "fixed",
"status_updated": "2026-09-29",
"status_note": "",
"component": "scheduler",
"severity": "leak",
"timestamp": "2026-09-29T08:20:03Z",
"last_seen": "2026-09-29T08:20:03Z",
"occurrences": 1,
"error_message": "standalone ASAN xce_scheduler --sim exits 1: LeakSanitizer reports 17 MB still allocated at exit\n\n**Observed:** an ASAN build of `xce_scheduler --sim=<file> --sim-verify`\nreplays the whole recording (0 MISMATCH lines) and then exits 1 because\nLeakSanitizer reports at exit:\n```\n==2234882==ERROR: LeakSanitizer: detected memory leaks\n...\nSUMMARY: AddressSanitizer: 17041405 byte(s) leaked in 1187 allocation(s).\n```\nOf the 17,041,405 bytes, 17,036,261 are indirect and 5,144 direct.\nGrouped by first scheduler/lib frame (bytes, leak groups):\n- `kvtab_init` (lib/kv.c:106): 16,941,432 in 15 -- core kv tables, reachable only from leaked roots\n- `sim_process_replay_event_message` (sim_loop.c:403): 49,688 in 54 -- decoded replay events (e.g. `decode_JobStateChangeEvent_ptr`)\n- `xkvmgr_add` (lib/kvmgr.c:114): 21,376 in 9\n- `sim_locate_and_load_snapshot` (sim.c:224): 14,347 in 106 -- snapshot decode (`decode_Conf4Sched` / `decode_QueueConf_list`)\n- `put_resvals` (resource.c:99) via limit.c `update_limit_res_value` and `sched_update_res_accounts`: 2,048 in 2\n- smaller: `parse_resreq`, `core_build_job`, `sched_ctx_init`, `match_job_host_limit_`\n\nThe indirect bulk fits the simulator never tearing down its core at exit\n(its picker/allocator threads are never joined, see bug-1632/bug-1763);\nthe direct groups may include real per-event leaks.\n\n**Expected:** a clean replay exits 0 under ASAN with LeakSanitizer enabled,\nor the exit-time teardown frees what the driver owns.\n**Impact:** leak; for scripts, any ASAN `--sim` replay must set\n`ASAN_OPTIONS=detect_leaks=0` or its exit code is always 1, which masks real\nfailures and hides any genuine per-event leak among the exit-time noise.\nRelease builds are unaffected at exit; a per-event leak would still grow a\nlong replay's memory.\n**Found by:** spec task (libxce-alloc-preason-equality 5.3, live\ndefault-settings replay, 2026-09-29)",
"file": "src/scheduler/src/sim_loop.c, src/scheduler/src/sim.c, src/scheduler/src/plugins/limit.c, src/scheduler/src/main.c",
"root_cause": "Nothing tore the simulator down (threads never joined, core never\nfinalized), and teardown then exposed a chain of ownership gaps, several of\nwhich also leak at runtime in normal mode:\n- xce_sched_core_fini() never destroyed the plugin/job/worker/queue/user/\n project registries, the shared-value arrays, the config or the\n pend_updates queues; kill_sched_job/queue/worker are shallow by design\n (a failed key_add() runs them and callers keep using the value).\n- sim_alloc.c's allocator stand-ins ran a full xce_sched_core_init(),\n re-initializing the process-wide string and selector tables.\n- plugin_unregister() key_del()'d from the registry being destroyed; the\n limit plugin's usage-instance table used kill_limit on xce_vres_t;\n add_vres() put a bare calloc() into the resobj registry.\n- Replay-event payloads were never released after the push (\"accepted\n leak\"); the start snapshot's non-config remainder was never freed.\n- kill_sched_resval() never freed the value struct (free_Value() frees only\n a string's buffer) -- every replaced value leaked, WORKERS_PUSH updates\n included; put_resvals() created a second value for a repeated index.\n- dup_val() tested which == XCE_VAL_STRING (0 = Value_u_ivalue32): real\n strings were shallow-copied, a nonzero int32 went to makeString().\n- The satisfy checks re-initialized the ctx's live unmeet vector per\n candidate host; replaced pend reasons, old parsed resreqs (JOB_UPDATE,\n duplicate JOBS_PUSH), the parser's error message and each written\n snapshot were never freed; the simulator never freed decision structs.",
"fix": "Six commits: 9b61fd99 (sim teardown, core registries, allocator stand-ins,\nplugin_unregister, kill_vres), c89e4e45 (replay-event payload release),\n8134b75b (core owns/frees its config; start snapshot freed), ebfe4794 (sim\ndecisions, fcfs plugin, pend_updates), c1cd8f91 (collect-then-free\nteardown for job/worker/queue/resource tables with shallow killers kept;\ndup_val; unmeet; add_vres via makeResource; plugin_register_core rejects\nduplicates up front), 90b99d91 (kill_sched_resval frees the value;\nput_resvals repeated index; resreq/schedobj/pend-reason/errmsg frees;\nwrite_snapshot_entry frees its snapshot, save side strdup()s).\nExit-time leak on the reproduction replay: 17,041,405 bytes -> 0.\nNot fixed here (separate): normal mode never frees decision structs;\nqueues rejected on config re-apply and a config replaced by CONFIG_PUSH\nare not freed; select parse errors inside the resreq grammar are ignored.\n\nVerification:\n- Test: test_cli_batch_replay_exits_clean_under_leak_detection in src/scheduler/testsuite/test_sim_discrete_clock.c (real binary, two-session fixture, --sim-verify --sim-output, detect_leaks=1): RED before (exit 1), GREEN after (exit 0)\n- Suites: ASAN debug build (src/xce.build.discrete), `meson test --no-suite xceadmin`: 117/117 OK, 0 sanitizer reports (2026-09-29)\n- Live: 3-round sequential HA soak on rebuilt ASAN binaries (/home/claude/tmp/xce-bug1884-soak): PASS, zero job loss, SIMVERIFY 4 files 0 mismatches, no sanitizer report in the live scheduler's stderr; all 13 recordings (this soak + xce-alloceq-53 + xce-ha-dc-soak) replay with detect_leaks=1: exit 0, 0 mismatches, no leak report",
"tags": [
"scheduler",
"sim",
"asan",
"lsan",
"leak"
],
"related_bugs": [
"bug-1632",
"bug-1763"
],
"fix_commits": [
"9b61fd99",
"c89e4e45",
"8134b75b",
"ebfe4794",
"c1cd8f91",
"90b99d91"
],
"record": ".wolf/bugs/bug-1884.md",
"gitea_issue": 335,
"gitea_synced_status": "open"
},
{
"id": "bug-1885",
"status": "open",
"status_updated": "2026-09-29",
"status_note": "newly logged",
"component": "scheduler",
"severity": "leak",
"timestamp": "2026-09-29T09:48:44Z",
"last_seen": "2026-09-29T09:48:44Z",
"occurrences": 1,
"error_message": "scheduler normal mode never frees decision structs: 64 bytes leaked per published decision\n\n**Observed:** (code reading) the normal-mode round loop in\n`xce_sched_driver_run_normal_mode_round()` (src/scheduler/src/driver.c)\ndrains each `xce_decision_t*` from `core->decisions`, records it and passes\nit to `on_decision`, but never frees the struct `make_*_decision()`\n(decision.c) calloc()s:\n```\ncbuf_get(&(sched->core.decisions), &decision);\n... sched->core.on_decision(sched->core.driver_ctx, decision);\n/* decision never freed */\n```\n\n**Expected:** the round loop frees the decision struct (not its alloc/pend reason, which the job owns) once published.\n**Impact:** leak of 64 bytes per dispatch/pend decision for the life of a normal-mode scheduler. Simulation mode was fixed in bug-1884 (ebfe4794). Three tests (test_core_full_cycle.c, test_core_session.c, test_snapshot_save_load.c) keep decision pointers from their callback, so the fix changes the callback contract (\"valid only during the call\") and those tests.\n**Found by:** review (while fixing bug-1884, 2026-09-29)",
"file": "src/scheduler/src/driver.c, src/scheduler/src/decision.c",
"root_cause": "TODO: required for fixed/partial",
"fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why",
"tags": [
"scheduler",
"leak",
"decision"
],
"related_bugs": [
"bug-1884"
],
"fix_commits": [],
"record": ".wolf/bugs/bug-1885.md"
},
{
"id": "bug-1886",
"status": "open",
"status_updated": "2026-09-29",
"status_note": "newly logged",
"component": "scheduler",
"severity": "leak",
"timestamp": "2026-09-29T09:48:47Z",
"last_seen": "2026-09-29T09:48:47Z",
"occurrences": 1,
"error_message": "scheduler leaks the rejected queue on each config re-apply and the old config on each CONFIG_PUSH\n\n**Observed:** (code reading)\n- `sched_init_queues()` (src/scheduler/src/scheduler.c) builds a new\n `xce_sched_queue_t` for every configured queue on each config apply and\n `sched_add_queue()`s it; for an existing name the add fails and the new\n queue (with its interned name/resreq) is still handed to the picker but\n owned by nothing.\n- `CONFIG_PUSH` (core_events.c) sets `core->config = config` without freeing\n the previous config.\n\n**Expected:** re-applied queues update the existing queue in place (as `core_upsert_queue()` does for QUEUE_UPDATE), and a replaced config is freed once nothing points into it.\n**Impact:** leak per config push/re-apply (one queue per configured queue plus one whole `xce_conf4sched_t`). Freeing the old config needs care: the picker and limit plugin may keep pointers into it. The queue table's datakiller must stay a no-op for as long as `sched_init_queues()` keeps using a rejected queue (bug-1884 found a use-after-free when it freed).\n**Found by:** review (while fixing bug-1884, 2026-09-29)",
"file": "src/scheduler/src/scheduler.c, src/scheduler/src/core_events.c",
"root_cause": "TODO: required for fixed/partial",
"fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why",
"tags": [
"scheduler",
"leak",
"config"
],
"related_bugs": [
"bug-1884"
],
"fix_commits": [],
"record": ".wolf/bugs/bug-1886.md"
},
{
"id": "bug-1887",
"status": "open",
"status_updated": "2026-09-29",
"status_note": "newly logged",
"component": "lib",
"severity": "wrong-result",
"timestamp": "2026-09-29T09:48:49Z",
"last_seen": "2026-09-29T09:48:49Z",
"occurrences": 1,
"error_message": "resreq parser ignores a primary's select/rusage parse error and accepts the job without that clause\n\n**Observed:** (code reading) the `primary_resreq` action in\nsrc/lib/xce_resreq.peg calls\n`parse_resreq_primary(primary, $0, auxil->resmgr, &(auxil->errmsg))`\nand ignores its return value; nothing sets `has_error`, so\n`parse_resreq()` returns success. A primary whose select or rusage clause\nfails to parse is kept without that clause (select NULL / rusages empty),\ni.e. the job is scheduled as if it had no such constraint. While fixing\nbug-1884 a LeakSanitizer report showed such an error message being\nproduced during a normal replay fixture (its job's resreq parsed \"OK\").\n\n**Expected:** a primary sub-parse error makes `parse_resreq()` fail with that message (the job is rejected/pended with a parse error), not silently dropped.\n**Impact:** wrong result: a job with a malformed select (e.g. a typo) may dispatch onto hosts its select would have excluded.\n**Found by:** review (while fixing bug-1884, 2026-09-29)",
"file": "src/lib/xce_resreq.peg, src/lib/resreq.c",
"root_cause": "TODO: required for fixed/partial",
"fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why",
"tags": [
"lib",
"resreq",
"parser"
],
"related_bugs": [
"bug-1884"
],
"fix_commits": [],
"record": ".wolf/bugs/bug-1887.md"
}
]
}