diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 9ab1f002..55d888ee 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -16112,7 +16112,9 @@ "fix_commits": [ "aa7fc65a" ], - "record": ".wolf/bugs/bug-1878.md" + "record": ".wolf/bugs/bug-1878.md", + "gitea_issue": 329, + "gitea_synced_status": "fixed" }, { "id": "bug-1879", @@ -16141,7 +16143,9 @@ "fix_commits": [ "aa7fc65a" ], - "record": ".wolf/bugs/bug-1879.md" + "record": ".wolf/bugs/bug-1879.md", + "gitea_issue": 330, + "gitea_synced_status": "fixed" }, { "id": "bug-1880", @@ -16170,13 +16174,15 @@ "bug-1845" ], "fix_commits": [], - "record": ".wolf/bugs/bug-1880.md" + "record": ".wolf/bugs/bug-1880.md", + "gitea_issue": 331, + "gitea_synced_status": "investigating" }, { "id": "bug-1881", - "status": "partial", - "status_updated": "2026-09-28", - "status_note": "mismatch lines now carry a first-difference diff (24b05718); live default-speed replay proof waits on spec scheduler-sim-discrete-clock (task 5.3)", + "status": "fixed", + "status_updated": "2026-09-29", + "status_note": "", "component": "scheduler", "severity": "wrong-result", "timestamp": "2026-09-28T07:26:09Z", @@ -16185,7 +16191,7 @@ "error_message": "sim_log_mismatch prints identical expected/actual, hiding the field that differs\n\n**Observed:**\n```\n... MISMATCH ... -- expected: DISPATCH jobid=0-12-1790578108-0[49] nhosts=1 -- actual: DISPATCH jobid=0-12-1790578108-0[49] nhosts=1\n```\n**Expected:** the message names the field(s) that differ, with both values.\n**Impact:** a mismatch cannot be diagnosed from the log (bug-1880: 43 such lines, none showing the difference).\n**Found by:** user reading the bug-1880 replay log, 2026-09-28\n\nUPDATE (2026-09-28): Fix planned in spec libxce-alloc-preason-equality (Requirement 3, Requirement 5.3, task 5): the libxce equality functions gain an optional why/whylen buffer describing the first difference, and sim_log_mismatch appends it to the MISMATCH line.", "file": "src/scheduler/src/sim_verify.c", "root_cause": "sim_log_mismatch() (src/scheduler/src/sim_verify.c) summarizes each decision as kind + jobid + nhosts, while sim_decision_eq() also compares every host's allocations (resname, value, duration, reserve), so a difference there prints two identical summaries.", - "fix": "Spec libxce-alloc-preason-equality (commits 8f25c1c3..dcf3acf3). sim_decision_eq() now compares DISPATCH decisions with libxce's public job_alloc_eq() and returns a caller-owned first-difference description (`char** why`) naming the field path and both values; sim_log_mismatch() appends it as ` -- diff: `, and the replay loop in sim_loop.c passes and frees it (fixed text \"no live decision\" for leftover recorded decisions) — 24b05718. job_alloc_eq() also compares `from`, so the scheduler now stamps every allocation's `from` with the deciding session's start (scheduler.c sched_make_primary_alloc_, c1960457) and the normal-mode recorder takes beginRound from the same reading (driver.c, 2d7ba660), making `from` reproducible on replay.\n\nRemaining: the simulator's own session-begin clock read (sim_loop.c) still drifts at --sim-speed >= 1, so a live default-settings replay cannot yet be shown to report zero `from` mismatches. That is task 5.3, blocked on spec scheduler-sim-discrete-clock.\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why", + "fix": "Spec libxce-alloc-preason-equality (commits 8f25c1c3..dcf3acf3). sim_decision_eq() now compares DISPATCH decisions with libxce's public job_alloc_eq() and returns a caller-owned first-difference description (`char** why`) naming the field path and both values; sim_log_mismatch() appends it as ` -- diff: `, and the replay loop in sim_loop.c passes and frees it (fixed text \"no live decision\" for leftover recorded decisions) — 24b05718. job_alloc_eq() also compares `from`, so the scheduler now stamps every allocation's `from` with the deciding session's start (scheduler.c sched_make_primary_alloc_, c1960457) and the normal-mode recorder takes beginRound from the same reading (driver.c, 2d7ba660), making `from` reproducible on replay.\n\nThe remaining gap (the simulator's own session-begin clock read drifting at --sim-speed >= 1) was closed by spec scheduler-sim-discrete-clock (4f653d8f..d9bc36cd: the simulator clock is pinned to recorded timestamps, --sim-speed is a no-op), so a replayed session starts at exactly the recorded beginRound and every replayed `from` matches; task 5.3 adds the proof (4e57bde6).\n\nVerification:\n- Test: test_log_mismatch_appends_provided_description and test_verify_against_altered_copy_reports_exact_mismatch (asserts ` -- diff: main_reason: 99 != 0`) plus test_verify_leftover_recorded_decision_names_missing_side in src/scheduler/testsuite/test_sim_verify.c: RED before 24b05718 (no diff text on the MISMATCH line), GREEN after. test_sim_replay_reproduces_recorded_alloc_from in src/scheduler/testsuite/test_sim_live_allocation.c (4e57bde6): GREEN at default settings; a +0.5 s clock mutant at sim_loop.c's sim_advance_virtual_clock(sim, begin_round) makes it FAIL\n- Suites: ASAN debug build (src/xce.build.discrete), `meson test --no-suite xceadmin`: 117/117 OK, 0 Unity :FAIL lines, 0 sanitizer reports (2026-09-29)\n- Live: 2-round sequential HA soak (/home/claude/tmp/xce-alloceq-53, PASS, zero job loss) recorded 3 clean .replay files with 11 DISPATCH decisions; each replayed with `xce_scheduler --sim --sim-verify` and no --sim-speed (ASAN_OPTIONS=detect_leaks=0): rc=0, 0 MISMATCH lines", "tags": [ "scheduler", "sim", @@ -16193,8 +16199,15 @@ "logging" ], "related_bugs": [], - "fix_commits": [], - "record": ".wolf/bugs/bug-1881.md" + "fix_commits": [ + "24b05718", + "c1960457", + "2d7ba660", + "4e57bde6" + ], + "record": ".wolf/bugs/bug-1881.md", + "gitea_issue": 332, + "gitea_synced_status": "fixed" }, { "id": "bug-1882", @@ -16223,7 +16236,152 @@ "fix_commits": [ "f9146864" ], - "record": ".wolf/bugs/bug-1882.md" + "record": ".wolf/bugs/bug-1882.md", + "gitea_issue": 333, + "gitea_synced_status": "fixed" + }, + { + "id": "bug-1883", + "status": "fixed", + "status_updated": "2026-09-29", + "status_note": "", + "component": "scheduler", + "severity": "wrong-result", + "timestamp": "2026-09-29T04:07:19Z", + "last_seen": "2026-09-29T04:07:19Z", + "occurrences": 1, + "error_message": "xce_scheduler seqjob reuses a stale per-host want from an earlier COMPOUND/ALTER primary\n\n**Observed:** (by code reading; no failing run captured yet)\nWhen a COMPOUND or ALTER job has a `maxTasks == 1` primary after another\nprimary in the same attempt, seqjob's host search for the later primary\nstarts from the per-host want (`ctx->alloc_req.host`) left behind by the\nearlier primary, with `slots:1` appended rather than replacing it:\n\n```\n.host on entry to seqjob host loop (2nd+ primary, shared ctx)\n = + slots:1\n```\n\n- `plugin_prepare_allocation()` resets only `alloc_req.shared` in the\n global pass (`src/scheduler/src/plugin.c:617-619`), never `.host`.\n- `seqjob_prepare_allocation()` appends `slots=1`\n (`src/scheduler/src/plugins/seqjob_allocator.c:118`), and\n `xvec_idval_store` appends without dedup (`src/lib/xvec.c:247-270`).\n- The host iterator pre-filters candidates against this stale vector\n (`src/scheduler/src/host_iter.c:136,158`), silently dropping hosts that\n would satisfy the later primary's real `{slots:1}` want.\n- A global limit on `slots` reads the first match via `get_resval_by_id`\n (`src/scheduler/src/resource.c:951-968`), so it sees the stale N, not 1.\n\nparjob does not have this problem because it clears `.host` per bucket\n(`src/scheduler/src/plugins/parjob_allocator.c:478-479`).\n\n**Expected:** every primary's per-host want is computed from that primary's\nown resource request only (the bug-1768 principle), so the later seqjob\nprimary is pre-filtered and limit-checked against `{slots:1}` plus its own\nrusage.\n**Impact:** wrong result. A later single-task primary of a COMPOUND/ALTER job\ncan pend (or pick a later host) although a host that fits it is free; pend\nreasons can list hosts filtered on resources the primary never asked for.\nPlain single-primary jobs are unaffected (fresh ctx).\n**Found by:** review. Discovery/requirements research for spec\n`scheduler-allocator-runner` (2026-09-28). That spec deliberately keeps this\nbehaviour (Requirement 3 AC 6) so the refactor stays behaviour-preserving;\nfix here, separately, with its own test.", + "file": "src/scheduler/src/plugins/seqjob_allocator.c, src/scheduler/src/plugin.c, src/scheduler/src/schedobj.c, src/scheduler/src/host_iter.c", + "root_cause": "`plugin_prepare_allocation()` (src/scheduler/src/plugin.c) reset only\n`ctx->alloc_req.shared` in its job-level pass (`ctx->worker == NULL`), while\nseqjob's and parjob's prepare hooks append this primary's job-level slots\nwant into `ctx->alloc_req.host` with the append-only `xvec_idval_store()`.\nA COMPOUND/ALTER job's primaries share one ctx (`sched_ctx_set_primary()`\nruns the job-level pass per primary), so `.host` still held the previous\nprimary's last per-host want. The host iterator pre-filtered on it\n(`host_iter.c`), and a global limit on slots read its first `slots` entry.", + "fix": "`plugin_prepare_allocation()`'s job-level pass now resets `.host` as well\nas `.shared`, so each primary's pass starts from its own want only\n(`c10f32cc`). The per-host pass already reset `.host`; parjob's per-bucket\nreset is unchanged.\n\nVerification:\n- Test: test_compound_later_seqjob_primary_ignores_previous_want in src/scheduler/testsuite/test_parjob_e2e_multislot.c (hosts A=4 slots, B=1 slot; COMPOUND `4*{..} + 1*{..}` through the real scheduler dispatch): RED before the fix (PEND, \"Expected 1 Was 2\"), GREEN after (DISPATCH, 4 slots on A + 1 on B)\n- Suites: ASAN debug build (src/xce.build.discrete), `meson test --no-suite xceadmin`: 117/117 OK, 0 Unity :FAIL lines, 0 sanitizer reports (2026-09-29)\n- Live: n/a -- no soak or deploy script submits COMPOUND/ALTER jobs; the test drives the real picker/allocator dispatch path end to end", + "tags": [ + "scheduler", + "allocation", + "compound", + "seqjob" + ], + "related_bugs": [ + "bug-1768", + "bug-1766" + ], + "fix_commits": [ + "c10f32cc" + ], + "record": ".wolf/bugs/bug-1883.md", + "gitea_issue": 334, + "gitea_synced_status": "fixed" + }, + { + "id": "bug-1884", + "status": "fixed", + "status_updated": "2026-09-29", + "status_note": "", + "component": "scheduler", + "severity": "leak", + "timestamp": "2026-09-29T08:20:03Z", + "last_seen": "2026-09-29T08:20:03Z", + "occurrences": 1, + "error_message": "standalone ASAN xce_scheduler --sim exits 1: LeakSanitizer reports 17 MB still allocated at exit\n\n**Observed:** an ASAN build of `xce_scheduler --sim= --sim-verify`\nreplays the whole recording (0 MISMATCH lines) and then exits 1 because\nLeakSanitizer reports at exit:\n```\n==2234882==ERROR: LeakSanitizer: detected memory leaks\n...\nSUMMARY: AddressSanitizer: 17041405 byte(s) leaked in 1187 allocation(s).\n```\nOf the 17,041,405 bytes, 17,036,261 are indirect and 5,144 direct.\nGrouped by first scheduler/lib frame (bytes, leak groups):\n- `kvtab_init` (lib/kv.c:106): 16,941,432 in 15 -- core kv tables, reachable only from leaked roots\n- `sim_process_replay_event_message` (sim_loop.c:403): 49,688 in 54 -- decoded replay events (e.g. `decode_JobStateChangeEvent_ptr`)\n- `xkvmgr_add` (lib/kvmgr.c:114): 21,376 in 9\n- `sim_locate_and_load_snapshot` (sim.c:224): 14,347 in 106 -- snapshot decode (`decode_Conf4Sched` / `decode_QueueConf_list`)\n- `put_resvals` (resource.c:99) via limit.c `update_limit_res_value` and `sched_update_res_accounts`: 2,048 in 2\n- smaller: `parse_resreq`, `core_build_job`, `sched_ctx_init`, `match_job_host_limit_`\n\nThe indirect bulk fits the simulator never tearing down its core at exit\n(its picker/allocator threads are never joined, see bug-1632/bug-1763);\nthe direct groups may include real per-event leaks.\n\n**Expected:** a clean replay exits 0 under ASAN with LeakSanitizer enabled,\nor the exit-time teardown frees what the driver owns.\n**Impact:** leak; for scripts, any ASAN `--sim` replay must set\n`ASAN_OPTIONS=detect_leaks=0` or its exit code is always 1, which masks real\nfailures and hides any genuine per-event leak among the exit-time noise.\nRelease builds are unaffected at exit; a per-event leak would still grow a\nlong replay's memory.\n**Found by:** spec task (libxce-alloc-preason-equality 5.3, live\ndefault-settings replay, 2026-09-29)", + "file": "src/scheduler/src/sim_loop.c, src/scheduler/src/sim.c, src/scheduler/src/plugins/limit.c, src/scheduler/src/main.c", + "root_cause": "Nothing tore the simulator down (threads never joined, core never\nfinalized), and teardown then exposed a chain of ownership gaps, several of\nwhich also leak at runtime in normal mode:\n- xce_sched_core_fini() never destroyed the plugin/job/worker/queue/user/\n project registries, the shared-value arrays, the config or the\n pend_updates queues; kill_sched_job/queue/worker are shallow by design\n (a failed key_add() runs them and callers keep using the value).\n- sim_alloc.c's allocator stand-ins ran a full xce_sched_core_init(),\n re-initializing the process-wide string and selector tables.\n- plugin_unregister() key_del()'d from the registry being destroyed; the\n limit plugin's usage-instance table used kill_limit on xce_vres_t;\n add_vres() put a bare calloc() into the resobj registry.\n- Replay-event payloads were never released after the push (\"accepted\n leak\"); the start snapshot's non-config remainder was never freed.\n- kill_sched_resval() never freed the value struct (free_Value() frees only\n a string's buffer) -- every replaced value leaked, WORKERS_PUSH updates\n included; put_resvals() created a second value for a repeated index.\n- dup_val() tested which == XCE_VAL_STRING (0 = Value_u_ivalue32): real\n strings were shallow-copied, a nonzero int32 went to makeString().\n- The satisfy checks re-initialized the ctx's live unmeet vector per\n candidate host; replaced pend reasons, old parsed resreqs (JOB_UPDATE,\n duplicate JOBS_PUSH), the parser's error message and each written\n snapshot were never freed; the simulator never freed decision structs.", + "fix": "Six commits: 9b61fd99 (sim teardown, core registries, allocator stand-ins,\nplugin_unregister, kill_vres), c89e4e45 (replay-event payload release),\n8134b75b (core owns/frees its config; start snapshot freed), ebfe4794 (sim\ndecisions, fcfs plugin, pend_updates), c1cd8f91 (collect-then-free\nteardown for job/worker/queue/resource tables with shallow killers kept;\ndup_val; unmeet; add_vres via makeResource; plugin_register_core rejects\nduplicates up front), 90b99d91 (kill_sched_resval frees the value;\nput_resvals repeated index; resreq/schedobj/pend-reason/errmsg frees;\nwrite_snapshot_entry frees its snapshot, save side strdup()s).\nExit-time leak on the reproduction replay: 17,041,405 bytes -> 0.\nNot fixed here (separate): normal mode never frees decision structs;\nqueues rejected on config re-apply and a config replaced by CONFIG_PUSH\nare not freed; select parse errors inside the resreq grammar are ignored.\n\nVerification:\n- Test: test_cli_batch_replay_exits_clean_under_leak_detection in src/scheduler/testsuite/test_sim_discrete_clock.c (real binary, two-session fixture, --sim-verify --sim-output, detect_leaks=1): RED before (exit 1), GREEN after (exit 0)\n- Suites: ASAN debug build (src/xce.build.discrete), `meson test --no-suite xceadmin`: 117/117 OK, 0 sanitizer reports (2026-09-29)\n- Live: 3-round sequential HA soak on rebuilt ASAN binaries (/home/claude/tmp/xce-bug1884-soak): PASS, zero job loss, SIMVERIFY 4 files 0 mismatches, no sanitizer report in the live scheduler's stderr; all 13 recordings (this soak + xce-alloceq-53 + xce-ha-dc-soak) replay with detect_leaks=1: exit 0, 0 mismatches, no leak report", + "tags": [ + "scheduler", + "sim", + "asan", + "lsan", + "leak" + ], + "related_bugs": [ + "bug-1632", + "bug-1763" + ], + "fix_commits": [ + "9b61fd99", + "c89e4e45", + "8134b75b", + "ebfe4794", + "c1cd8f91", + "90b99d91" + ], + "record": ".wolf/bugs/bug-1884.md", + "gitea_issue": 335, + "gitea_synced_status": "open" + }, + { + "id": "bug-1885", + "status": "open", + "status_updated": "2026-09-29", + "status_note": "newly logged", + "component": "scheduler", + "severity": "leak", + "timestamp": "2026-09-29T09:48:44Z", + "last_seen": "2026-09-29T09:48:44Z", + "occurrences": 1, + "error_message": "scheduler normal mode never frees decision structs: 64 bytes leaked per published decision\n\n**Observed:** (code reading) the normal-mode round loop in\n`xce_sched_driver_run_normal_mode_round()` (src/scheduler/src/driver.c)\ndrains each `xce_decision_t*` from `core->decisions`, records it and passes\nit to `on_decision`, but never frees the struct `make_*_decision()`\n(decision.c) calloc()s:\n```\ncbuf_get(&(sched->core.decisions), &decision);\n... sched->core.on_decision(sched->core.driver_ctx, decision);\n/* decision never freed */\n```\n\n**Expected:** the round loop frees the decision struct (not its alloc/pend reason, which the job owns) once published.\n**Impact:** leak of 64 bytes per dispatch/pend decision for the life of a normal-mode scheduler. Simulation mode was fixed in bug-1884 (ebfe4794). Three tests (test_core_full_cycle.c, test_core_session.c, test_snapshot_save_load.c) keep decision pointers from their callback, so the fix changes the callback contract (\"valid only during the call\") and those tests.\n**Found by:** review (while fixing bug-1884, 2026-09-29)", + "file": "src/scheduler/src/driver.c, src/scheduler/src/decision.c", + "root_cause": "TODO: required for fixed/partial", + "fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why", + "tags": [ + "scheduler", + "leak", + "decision" + ], + "related_bugs": [ + "bug-1884" + ], + "fix_commits": [], + "record": ".wolf/bugs/bug-1885.md" + }, + { + "id": "bug-1886", + "status": "open", + "status_updated": "2026-09-29", + "status_note": "newly logged", + "component": "scheduler", + "severity": "leak", + "timestamp": "2026-09-29T09:48:47Z", + "last_seen": "2026-09-29T09:48:47Z", + "occurrences": 1, + "error_message": "scheduler leaks the rejected queue on each config re-apply and the old config on each CONFIG_PUSH\n\n**Observed:** (code reading)\n- `sched_init_queues()` (src/scheduler/src/scheduler.c) builds a new\n `xce_sched_queue_t` for every configured queue on each config apply and\n `sched_add_queue()`s it; for an existing name the add fails and the new\n queue (with its interned name/resreq) is still handed to the picker but\n owned by nothing.\n- `CONFIG_PUSH` (core_events.c) sets `core->config = config` without freeing\n the previous config.\n\n**Expected:** re-applied queues update the existing queue in place (as `core_upsert_queue()` does for QUEUE_UPDATE), and a replaced config is freed once nothing points into it.\n**Impact:** leak per config push/re-apply (one queue per configured queue plus one whole `xce_conf4sched_t`). Freeing the old config needs care: the picker and limit plugin may keep pointers into it. The queue table's datakiller must stay a no-op for as long as `sched_init_queues()` keeps using a rejected queue (bug-1884 found a use-after-free when it freed).\n**Found by:** review (while fixing bug-1884, 2026-09-29)", + "file": "src/scheduler/src/scheduler.c, src/scheduler/src/core_events.c", + "root_cause": "TODO: required for fixed/partial", + "fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why", + "tags": [ + "scheduler", + "leak", + "config" + ], + "related_bugs": [ + "bug-1884" + ], + "fix_commits": [], + "record": ".wolf/bugs/bug-1886.md" + }, + { + "id": "bug-1887", + "status": "open", + "status_updated": "2026-09-29", + "status_note": "newly logged", + "component": "lib", + "severity": "wrong-result", + "timestamp": "2026-09-29T09:48:49Z", + "last_seen": "2026-09-29T09:48:49Z", + "occurrences": 1, + "error_message": "resreq parser ignores a primary's select/rusage parse error and accepts the job without that clause\n\n**Observed:** (code reading) the `primary_resreq` action in\nsrc/lib/xce_resreq.peg calls\n`parse_resreq_primary(primary, $0, auxil->resmgr, &(auxil->errmsg))`\nand ignores its return value; nothing sets `has_error`, so\n`parse_resreq()` returns success. A primary whose select or rusage clause\nfails to parse is kept without that clause (select NULL / rusages empty),\ni.e. the job is scheduled as if it had no such constraint. While fixing\nbug-1884 a LeakSanitizer report showed such an error message being\nproduced during a normal replay fixture (its job's resreq parsed \"OK\").\n\n**Expected:** a primary sub-parse error makes `parse_resreq()` fail with that message (the job is rejected/pended with a parse error), not silently dropped.\n**Impact:** wrong result: a job with a malformed select (e.g. a typo) may dispatch onto hosts its select would have excluded.\n**Found by:** review (while fixing bug-1884, 2026-09-29)", + "file": "src/lib/xce_resreq.peg, src/lib/resreq.c", + "root_cause": "TODO: required for fixed/partial", + "fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why", + "tags": [ + "lib", + "resreq", + "parser" + ], + "related_bugs": [ + "bug-1884" + ], + "fix_commits": [], + "record": ".wolf/bugs/bug-1887.md" } ] } diff --git a/.wolf/bugs/bug-1884.md b/.wolf/bugs/bug-1884.md index 55e8e2a4..4ca2d07a 100644 --- a/.wolf/bugs/bug-1884.md +++ b/.wolf/bugs/bug-1884.md @@ -1,8 +1,8 @@ --- id: "bug-1884" -status: "partial" +status: "fixed" status_updated: "2026-09-29" -status_note: "part 1 (9b61fd99): sim teardown + core registries, 17.2 MB -> 75 KB at exit; part 2: replay-event payload release, core->config, job/resource destructors, snapshot-load data" +status_note: "" component: "scheduler" severity: "leak" timestamp: "2026-09-29T08:20:03Z" @@ -11,6 +11,7 @@ occurrences: 1 files: ["src/scheduler/src/sim_loop.c", "src/scheduler/src/sim.c", "src/scheduler/src/plugins/limit.c", "src/scheduler/src/main.c"] tags: ["scheduler", "sim", "asan", "lsan", "leak"] related: ["bug-1632", "bug-1763"] +fix_commits: ["9b61fd99", "c89e4e45", "8134b75b", "ebfe4794", "c1cd8f91", "90b99d91"] gitea_issue: 335 gitea_synced_status: "open" gitea_synced_hash: "097b7ddb90dd" @@ -61,44 +62,54 @@ default-settings replay, 2026-09-29) ## Reproduction 1. `bash src/tools/xce-build.sh scheduler asan` (or any ASAN build dir) 2. `src/xce.build/scheduler/xce_scheduler --sim=/home/claude/tmp/bug-1884/scheduler-replay-20260929-064412.replay --sim-verify --logfile /tmp/sim.log; echo rc=$?` -3. Observe `rc=1` and the LeakSanitizer SUMMARY on stderr. With `ASAN_OPTIONS=detect_leaks=0` the same run exits 0. +3. Observe `rc=1` and the LeakSanitizer SUMMARY on stderr (before the fix). With `ASAN_OPTIONS=detect_leaks=0` the same run exits 0. - Frequency: deterministic (2/2 runs, identical totals, 2026-09-29); also seen on every recording of /home/claude/tmp/xce-ha-dc-soak - Evidence: /home/claude/tmp/bug-1884/lsan-report.txt (full report), /home/claude/tmp/bug-1884/scheduler-replay-20260929-064412.replay ## Root cause -Nothing tears the simulator down: sched_sim_driver_run() never joined its -picker/allocator threads and main() exited without finalizing the core. On -top of that, several ownership gaps only show once teardown runs: +Nothing tore the simulator down (threads never joined, core never +finalized), and teardown then exposed a chain of ownership gaps, several of +which also leak at runtime in normal mode: - xce_sched_core_fini() never destroyed the plugin/job/worker/queue/user/ - project registries or the shared-resource arrays that init creates. -- sim_alloc.c's two sched_make_allocs() stand-ins each ran a full - xce_sched_core_init(): a whole unused core, plus a re-init of the - process-wide string-intern and selector tables (two 8 MB tables leaked). -- plugin_unregister() (the plugin registry's datakiller, never run before) - key_del()'d from the registry being destroyed and skipped fini_fn for - plugins registered with plugin_register_core(). -- The limit plugin's usage-instance table had kill_limit as datakiller for - xce_vres_t records (bad free on destroy); kill_vres() was an empty stub. -- Remaining (part 2): replay-event payloads are never released after - xce_core_push_events() (sim_loop.c: documented "accepted leak"), unlike - driver.c's release_translated_payload(); the adopted core->config is never - freed (and a CONFIG_PUSH replaces it without freeing the old one); - kill_sched_job() is shallow; resmgr has no datakiller; snapshot-load data, - the builtin_fcfs plugin struct and some allocator scratch are not freed. + project registries, the shared-value arrays, the config or the + pend_updates queues; kill_sched_job/queue/worker are shallow by design + (a failed key_add() runs them and callers keep using the value). +- sim_alloc.c's allocator stand-ins ran a full xce_sched_core_init(), + re-initializing the process-wide string and selector tables. +- plugin_unregister() key_del()'d from the registry being destroyed; the + limit plugin's usage-instance table used kill_limit on xce_vres_t; + add_vres() put a bare calloc() into the resobj registry. +- Replay-event payloads were never released after the push ("accepted + leak"); the start snapshot's non-config remainder was never freed. +- kill_sched_resval() never freed the value struct (free_Value() frees only + a string's buffer) -- every replaced value leaked, WORKERS_PUSH updates + included; put_resvals() created a second value for a repeated index. +- dup_val() tested which == XCE_VAL_STRING (0 = Value_u_ivalue32): real + strings were shallow-copied, a nonzero int32 went to makeString(). +- The satisfy checks re-initialized the ctx's live unmeet vector per + candidate host; replaced pend reasons, old parsed resreqs (JOB_UPDATE, + duplicate JOBS_PUSH), the parser's error message and each written + snapshot were never freed; the simulator never freed decision structs. ## Fix -Part 1 (9b61fd99): sched_sim_driver_fini() -- cancel/join the threads the -way sched_fini() does (thread ids now in core.jobpicker/allocator), close -the files, finalize the core; main() calls it after a clean run. Core fini -destroys every registry init makes. The allocator stand-ins set up only -lock, nextid and the aliased resmgr. plugin_unregister() only runs fini_fn. -Usage instances use kill_vres(), which frees the record. Leak at exit on the -reproduction replay: 17,041,405 -> 75,495 bytes. +Six commits: 9b61fd99 (sim teardown, core registries, allocator stand-ins, +plugin_unregister, kill_vres), c89e4e45 (replay-event payload release), +8134b75b (core owns/frees its config; start snapshot freed), ebfe4794 (sim +decisions, fcfs plugin, pend_updates), c1cd8f91 (collect-then-free +teardown for job/worker/queue/resource tables with shallow killers kept; +dup_val; unmeet; add_vres via makeResource; plugin_register_core rejects +duplicates up front), 90b99d91 (kill_sched_resval frees the value; +put_resvals repeated index; resreq/schedobj/pend-reason/errmsg frees; +write_snapshot_entry frees its snapshot, save side strdup()s). +Exit-time leak on the reproduction replay: 17,041,405 bytes -> 0. +Not fixed here (separate): normal mode never frees decision structs; +queues rejected on config re-apply and a config replaced by CONFIG_PUSH +are not freed; select parse errors inside the resreq grammar are ignored. ## Verification -- Test: TODO: name + file; RED before the fix, GREEN after -- Suites: TODO: suite results and build profile -- Live: TODO: soak/deploy run, or n/a and why +- Test: test_cli_batch_replay_exits_clean_under_leak_detection in src/scheduler/testsuite/test_sim_discrete_clock.c (real binary, two-session fixture, --sim-verify --sim-output, detect_leaks=1): RED before (exit 1), GREEN after (exit 0) +- Suites: ASAN debug build (src/xce.build.discrete), `meson test --no-suite xceadmin`: 117/117 OK, 0 sanitizer reports (2026-09-29) +- Live: 3-round sequential HA soak on rebuilt ASAN binaries (/home/claude/tmp/xce-bug1884-soak): PASS, zero job loss, SIMVERIFY 4 files 0 mismatches, no sanitizer report in the live scheduler's stderr; all 13 recordings (this soak + xce-alloceq-53 + xce-ha-dc-soak) replay with detect_leaks=1: exit 0, 0 mismatches, no leak report ## References - Related: bug-1632, bug-1763 (sim picker/allocator threads never joined) diff --git a/.wolf/bugs/bug-1885.md b/.wolf/bugs/bug-1885.md new file mode 100644 index 00000000..b8c32c9b --- /dev/null +++ b/.wolf/bugs/bug-1885.md @@ -0,0 +1,60 @@ +--- +id: "bug-1885" +status: "open" +status_updated: "2026-09-29" +status_note: "newly logged" +component: "scheduler" +severity: "leak" +timestamp: "2026-09-29T09:48:44Z" +last_seen: "2026-09-29T09:48:44Z" +occurrences: 1 +files: ["src/scheduler/src/driver.c", "src/scheduler/src/decision.c"] +tags: ["scheduler", "leak", "decision"] +related: ["bug-1884"] +--- + +# bug-1885: scheduler normal mode never frees decision structs: 64 bytes leaked per published decision + +## Description +**Observed:** (code reading) the normal-mode round loop in +`xce_sched_driver_run_normal_mode_round()` (src/scheduler/src/driver.c) +drains each `xce_decision_t*` from `core->decisions`, records it and passes +it to `on_decision`, but never frees the struct `make_*_decision()` +(decision.c) calloc()s: +``` +cbuf_get(&(sched->core.decisions), &decision); +... sched->core.on_decision(sched->core.driver_ctx, decision); +/* decision never freed */ +``` + +**Expected:** the round loop frees the decision struct (not its alloc/pend reason, which the job owns) once published. +**Impact:** leak of 64 bytes per dispatch/pend decision for the life of a normal-mode scheduler. Simulation mode was fixed in bug-1884 (ebfe4794). Three tests (test_core_full_cycle.c, test_core_session.c, test_snapshot_save_load.c) keep decision pointers from their callback, so the fix changes the callback contract ("valid only during the call") and those tests. +**Found by:** review (while fixing bug-1884, 2026-09-29) + +## Environment +- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64 +- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2) +- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 +- Build: asan (src/xce.build.discrete) +- Code: scheduler-framework @ 90b99d91-dirty +- Deployment: n/a (found by code reading while fixing bug-1884; not observed on a deployment) + +## Reproduction +1. Read `src/scheduler/src/driver.c` round loop (the `cbuf_get(&(sched->core.decisions), ...)` loop) and `src/scheduler/src/decision.c` `make_dispatch_decision()`/`make_pend_decision()`. +2. A live check: run a normal-mode scheduler under ASAN with detect_leaks=1 through a few rounds and stop it cleanly; expect `make_*_decision` direct leaks. +- Frequency: deterministic by code reading (one struct per decision) +- Evidence: not kept; code reading + +## Root cause +TODO: required for fixed/partial + +## Fix +TODO: required for fixed/partial + +## Verification +- Test: TODO: name + file; RED before the fix, GREEN after +- Suites: TODO: suite results and build profile +- Live: TODO: soak/deploy run, or n/a and why + +## References +- Related: bug-1884 diff --git a/.wolf/bugs/bug-1886.md b/.wolf/bugs/bug-1886.md new file mode 100644 index 00000000..efd6618f --- /dev/null +++ b/.wolf/bugs/bug-1886.md @@ -0,0 +1,58 @@ +--- +id: "bug-1886" +status: "open" +status_updated: "2026-09-29" +status_note: "newly logged" +component: "scheduler" +severity: "leak" +timestamp: "2026-09-29T09:48:47Z" +last_seen: "2026-09-29T09:48:47Z" +occurrences: 1 +files: ["src/scheduler/src/scheduler.c", "src/scheduler/src/core_events.c"] +tags: ["scheduler", "leak", "config"] +related: ["bug-1884"] +--- + +# bug-1886: scheduler leaks the rejected queue on each config re-apply and the old config on each CONFIG_PUSH + +## Description +**Observed:** (code reading) +- `sched_init_queues()` (src/scheduler/src/scheduler.c) builds a new + `xce_sched_queue_t` for every configured queue on each config apply and + `sched_add_queue()`s it; for an existing name the add fails and the new + queue (with its interned name/resreq) is still handed to the picker but + owned by nothing. +- `CONFIG_PUSH` (core_events.c) sets `core->config = config` without freeing + the previous config. + +**Expected:** re-applied queues update the existing queue in place (as `core_upsert_queue()` does for QUEUE_UPDATE), and a replaced config is freed once nothing points into it. +**Impact:** leak per config push/re-apply (one queue per configured queue plus one whole `xce_conf4sched_t`). Freeing the old config needs care: the picker and limit plugin may keep pointers into it. The queue table's datakiller must stay a no-op for as long as `sched_init_queues()` keeps using a rejected queue (bug-1884 found a use-after-free when it freed). +**Found by:** review (while fixing bug-1884, 2026-09-29) + +## Environment +- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64 +- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2) +- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 +- Build: asan (src/xce.build.discrete) +- Code: scheduler-framework @ 90b99d91-dirty +- Deployment: n/a (found by code reading while fixing bug-1884; not observed on a deployment) + +## Reproduction +1. Read `sched_init_queues()` in src/scheduler/src/scheduler.c and the `XCE_CORE_EVENT_CONFIG_PUSH` case in src/scheduler/src/core_events.c. +2. Test-level check: `test_config_push_deferred.c` applies a config twice; with a freeing queue killer ASAN reports a heap-use-after-free in `builtin_fcfs_on_queue_update` (seen 2026-09-29 while fixing bug-1884). +- Frequency: deterministic on every config re-apply +- Evidence: not kept; code reading plus the test_config_push_deferred UAF seen with a freeing killer + +## Root cause +TODO: required for fixed/partial + +## Fix +TODO: required for fixed/partial + +## Verification +- Test: TODO: name + file; RED before the fix, GREEN after +- Suites: TODO: suite results and build profile +- Live: TODO: soak/deploy run, or n/a and why + +## References +- Related: bug-1884 diff --git a/.wolf/bugs/bug-1887.md b/.wolf/bugs/bug-1887.md new file mode 100644 index 00000000..e08a381b --- /dev/null +++ b/.wolf/bugs/bug-1887.md @@ -0,0 +1,59 @@ +--- +id: "bug-1887" +status: "open" +status_updated: "2026-09-29" +status_note: "newly logged" +component: "lib" +severity: "wrong-result" +timestamp: "2026-09-29T09:48:49Z" +last_seen: "2026-09-29T09:48:49Z" +occurrences: 1 +files: ["src/lib/xce_resreq.peg", "src/lib/resreq.c"] +tags: ["lib", "resreq", "parser"] +related: ["bug-1884"] +--- + +# bug-1887: resreq parser ignores a primary's select/rusage parse error and accepts the job without that clause + +## Description +**Observed:** (code reading) the `primary_resreq` action in +src/lib/xce_resreq.peg calls +`parse_resreq_primary(primary, $0, auxil->resmgr, &(auxil->errmsg))` +and ignores its return value; nothing sets `has_error`, so +`parse_resreq()` returns success. A primary whose select or rusage clause +fails to parse is kept without that clause (select NULL / rusages empty), +i.e. the job is scheduled as if it had no such constraint. While fixing +bug-1884 a LeakSanitizer report showed such an error message being +produced during a normal replay fixture (its job's resreq parsed "OK"). + +**Expected:** a primary sub-parse error makes `parse_resreq()` fail with that message (the job is rejected/pended with a parse error), not silently dropped. +**Impact:** wrong result: a job with a malformed select (e.g. a typo) may dispatch onto hosts its select would have excluded. +**Found by:** review (while fixing bug-1884, 2026-09-29) + +## Environment +- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64 +- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2) +- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 +- Build: asan (src/xce.build.discrete) +- Code: scheduler-framework @ 90b99d91-dirty +- Deployment: n/a (found by code reading while fixing bug-1884; not observed on a deployment) + +## Reproduction +1. Read the `primary_resreq` action in src/lib/xce_resreq.peg and `parse_resreq_primary()` in src/lib/resreq.c. +2. Unit check: `parse_resreq(&r, "select[type==] rusage[mem=1]", resmgr, &err)` -- expect rc < 0; observe the return value. +- Frequency: deterministic for any malformed select/rusage inside a primary +- Evidence: not kept; LeakSanitizer stack from the bug-1884 fixture (strdup in parse_select error path under core_build_job) + +## Root cause +TODO: required for fixed/partial + +## Fix +TODO: required for fixed/partial + +## Verification +- Test: TODO: name + file; RED before the fix, GREEN after +- Suites: TODO: suite results and build profile +- Live: TODO: soak/deploy run, or n/a and why + +## References +- Related: bug-1884