From c3cdf2f515b58d86d19b7aa50c10f9e533ea5b4a Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Fri, 9 Oct 2026 03:37:04 +0000 Subject: [PATCH] docs(buglog): bug-1928, bug-1929, bug-1930 fixed (ba6f42cd) Co-Authored-By: Claude Opus 5.5 (1M context) --- .wolf/buglog.json | 87 ++++++++++++++++++++++++++++++++++++++++++ .wolf/bugs/bug-1928.md | 77 +++++++++++++++++++++++++++++++++++++ .wolf/bugs/bug-1929.md | 73 +++++++++++++++++++++++++++++++++++ .wolf/bugs/bug-1930.md | 66 ++++++++++++++++++++++++++++++++ 4 files changed, 303 insertions(+) create mode 100644 .wolf/bugs/bug-1928.md create mode 100644 .wolf/bugs/bug-1929.md create mode 100644 .wolf/bugs/bug-1930.md diff --git a/.wolf/buglog.json b/.wolf/buglog.json index c14b6030..cd7d1f92 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -17566,6 +17566,93 @@ "record": ".wolf/bugs/bug-1927.md", "gitea_issue": 352, "gitea_synced_status": "open" + }, + { + "id": "bug-1928", + "status": "fixed", + "status_updated": "2026-10-09", + "status_note": "", + "component": "scheduler", + "severity": "leak", + "timestamp": "2026-10-09T03:14:08Z", + "last_seen": "2026-10-09T03:14:08Z", + "occurrences": 1, + "error_message": "xce_scheduler --sim REPL leaks a job-table iterator every continue round (LeakSanitizer at exit)\n\n**Observed:** a normal interactive simulator session (`continue`, then `quit`) on the ASAN build exits 1 from LeakSanitizer, and the buffered console output is lost (0 bytes on stdout):\n```\n==4149978==ERROR: LeakSanitizer: detected memory leaks\n\nDirect leak of 144 byte(s) in 9 object(s) allocated from:\n #0 0x79e7779215a3 in calloc (.../libasan.so.8+0x1215a3)\n #1 0x5907bfd480ae in xkvmgr_make_iter ../lib/kvmgr.c:159\n #2 0x5907bfde1eca in iter_make (.../xce_scheduler+0x652eca)\n #3 0x5907bfcf1004 in repl_count_jobs ../scheduler/src/repl.c:189\n #4 0x5907bfcf5464 in repl_query_pend_jobids ../scheduler/src/repl_rollup.c:94\n #5 0x5907bfcf64bc in repl_round_rollup_process ../scheduler/src/repl_rollup.c:295\n #6 0x5907bfcf3238 in xce_repl_post_session_check ../scheduler/src/repl.c:626\n...\nSUMMARY: AddressSanitizer: 160 byte(s) leaked in 10 allocation(s).\n```\n**Expected:** the session exits 0 with leak detection on, and its console output reaches stdout.\n**Impact:** leak of 16 bytes (one kvmgr iterator) per call of `repl_count_jobs()`: once at the load banner and once per completed round in `continue` mode (Round Rollup), so it grows linearly with the replay length; `info jobs` leaks one more per call. Small, but it makes every standalone ASAN REPL run fail with exit 1 and drop its stdout, and forced several REPL tests to run with `detect_leaks=0` (the trace wiring test's meson comment named this leak as the reason).\n**Found by:** ASAN run of the simulator during scheduler-sim-decision-trace validation; the second site (`repl_info_jobs()`) by audit. A grep of every `iter_make()` in src/scheduler/src for a matching `iter_destroy()` on every path found no others.", + "file": "src/scheduler/src/repl.c, src/scheduler/src/repl_info.c, src/scheduler/src/repl_rollup.c, src/scheduler/testsuite/meson.build", + "root_cause": "`repl_count_jobs()` (src/scheduler/src/repl.c:189) creates an iterator over `core->jobs` with `iter_make()`, walks it, and returns the count without calling `iter_destroy()`. For a kvmgr container `iter_make()` heap-allocates the inner iterator (`xkvmgr_make_iter`, lib/kvmgr.c:159) and only `xkvmgr_iter_destroy()` frees it, so each call leaks 16 bytes. It is called from `repl_print_banner()` once and from `repl_query_pend_jobids()` (repl_rollup.c) on every completed round in `continue` mode, hence the per-round growth. `repl_info_jobs()` (src/scheduler/src/repl_info.c:172, `info jobs`) has the same missing `iter_destroy()`. The leak makes LeakSanitizer turn the exit status into 1 at exit, and the process ends before stdio flushes the redirected stdout.\n\nSince: `repl_count_jobs()` since 02f7f48f (scheduler-core REPL task 12.1); `repl_info_jobs()` since c705cada. repl_rollup.c's own comment acknowledged the missing destroy as a \"low-impact quirk\", but its own per-round caller made it a growing leak.\n\nScope: every `iter_make()` in src/scheduler/src (core.c, core_session.c, plugin.c, plugins/limit.c, host_iter.c, scheduler.c, snapshot.c, sim_loop.c, sim_picker.c, repl_info.c, repl_rollup.c, repl_trace_store.c) was checked; all others destroy the iterator on every path, and their early returns are only on `iter_make()` failure, where kvmgr frees the iterator itself. `host_iter.c` keeps its iterator in the host-iterator private data and destroys it in its fini.", + "fix": "- src/scheduler/src/repl.c `repl_count_jobs()`: call `iter_destroy(&it)` after the walk.\n- src/scheduler/src/repl_info.c `repl_info_jobs()`: same.\n- src/scheduler/src/repl_rollup.c: replaced the stale comment that called the missing destroy a quirk.\n- src/scheduler/testsuite/meson.build: `scheduler repl trace baseline test` and `scheduler repl trace wiring test` now run with leak detection on (removed `ASAN_OPTIONS=detect_leaks=0` and the comment blaming this leak). The other REPL/sim tests still leak from their own fixtures (cores and replay writers never torn down), so they keep `detect_leaks=0`.\n\nEvery iterator the REPL makes is now destroyed, so nothing is left for LeakSanitizer at exit.\n\nVerification:\n- Test: `scheduler repl trace baseline test` (test_repl_trace_baseline.c, leak detection now on): RED before the fix (`test_transcript_matches_baseline: Expected 0 Was 1`, the child exits 1 from LeakSanitizer), GREEN after (4 Tests 0 Failures). `scheduler repl trace wiring test` (test_repl_trace_wiring.c, leak detection now on): RED before (exit 1, LeakSanitizer with `repl_count_jobs` frames), GREEN after (17 Tests 0 Failures).\n- Leak-on sweep of all 42 sched_repl_*/sched_sim_* test binaries: `repl_count_jobs`/`repl_info_jobs` frames 0 in every log after the fix (before: present in 13).\n- Suites: `bash src/tools/xce-build.sh scheduler --jobs 12 --incremental`: scheduler suite 118/118 OK, asan (log: /home/claude/tmp/bug-1928/gate-scheduler-suite.log).\n- Live: the standalone command in Reproduction step 2 now exits 0 with leak detection on and prints its 18-line transcript (/home/claude/tmp/bug-1928/green/).", + "tags": [ + "repl", + "sim", + "leak", + "iterator", + "asan" + ], + "related_bugs": [ + "bug-1884" + ], + "fix_commits": [ + "ba6f42cd" + ], + "record": ".wolf/bugs/bug-1928.md" + }, + { + "id": "bug-1929", + "status": "fixed", + "status_updated": "2026-10-09", + "status_note": "", + "component": "scheduler", + "severity": "wrong-result", + "timestamp": "2026-10-09T03:21:26Z", + "last_seen": "2026-10-09T03:21:26Z", + "occurrences": 1, + "error_message": "xce_scheduler --sim REPL continue round summary prints empty jobids= for a dispatch-only round\n\n**Observed:** in `continue` mode a round whose only decision is a dispatch prints a round summary with an empty job list, with tracing off as well as on. From the 6.3 trace-equivalence harness (`XCE_TRACE_EQUIV_DUMP=1`, extra replay, 8 occurrences):\n```\n[sim] round 1-1 (1 round(s), t=1970-01-01T00:00:00Z..1970-01-01T00:00:00Z): 4 dispatch, 1 pend, 0 preempt -- jobids=0-0-0-202\n[sim] round 2-2 (1 round(s), t=1970-01-01T00:00:00Z..1970-01-01T00:00:00Z): 1 dispatch, 0 pend, 0 preempt -- jobids=\n```\nWith the 1.1 contention fixture the dispatched job is missing even when a pend job is listed (round 8 dispatched 0-0-0-108):\n```\n[sim] round 8-8 (1 round(s), t=1970-01-01T00:00:00Z..1970-01-01T00:00:00Z): 1 dispatch, 1 pend, 0 preempt -- jobids=0-0-0-107\n```\n**Expected:** \"jobids=\" names the jobs the round's decisions affected (scheduler-sim-repl-verbosity Requirement 2.1: \"the count of each decision type and the identifiers of the jobs affected\"), so the dispatched (and preempted) jobs are listed: `jobids=0-0-0-202` (the job pending in round 1 and dispatched in round 2) and `jobids=0-0-0-107,0-0-0-108`.\n**Impact:** wrong/misleading REPL output only: the operator cannot see which job a round dispatched or preempted. A second effect of the same cause: rounds with the same counts and pend set but different dispatched jobs collapse into one range (Requirement 3.2 says a range needs the same affected job identifiers). Scheduling decisions are not affected.\n**Found by:** scheduler-sim-decision-trace validation (rollup output in the 6.1/6.3 e2e and equivalence harness dumps).", + "file": "src/scheduler/src/repl_rollup.c, src/scheduler/src/repl_break.c, src/scheduler/src/repl_internal.h, src/scheduler/testsuite/test_repl_round_rollup.c, src/scheduler/testsuite/test_repl_trace_emit.c, src/scheduler/testsuite/test_repl_trace_wiring.c, src/scheduler/testsuite/fixtures/repl_trace_baseline/transcript.txt", + "root_cause": "`repl_round_emit()` (src/scheduler/src/repl_rollup.c:178, listing built at :201-202 before the fix) built the \"jobids=\" listing from `pend_jobids` only, and `repl_round_rollup_process()` (repl_rollup.c:285) only ever collected the pend set (`repl_query_pend_jobids()`, a walk of core->jobs for JOB_IS_PEND jobs). The dispatch and preempt decisions were counted in `xce_repl_on_decision()` (src/scheduler/src/repl_break.c:953-959, `last_n_dispatch`/`last_n_preempt`) but their job ids were never recorded anywhere. A dispatched job is no longer pending after the round, so a dispatch-only round has an empty pend set and printed `jobids=` with nothing after it; in mixed rounds the dispatched jobs were silently left out. The same pend-only fingerprint made rounds that dispatched different jobs compare equal and collapse.\n\nSince: 921c72bb (scheduler-sim-repl-verbosity task 2.2, Round Rollup), written while bug-1611 meant the simulator produced no DISPATCH decisions, so the pend set was the only affected-job information then.\n\nScope: the traced-round path (scheduler-sim-decision-trace 7.3/7.4) and the allocation-failure fallback go through the same `repl_round_emit()`; step mode prints per-decision lines and was not affected.", + "fix": "- src/scheduler/src/repl_internal.h: new state in `xce_sched_repl_state_s` -- `round_decided_jobids`/`round_n_decided`/`round_cap_decided`/`round_decided_lost` (this round's dispatch/preempt jobids) and `collapse_decided_jobids`/`collapse_n_decided_jobids` (the active range's); declarations of `repl_round_note_decided()`/`repl_round_reset_decided()`.\n- src/scheduler/src/repl_break.c: `xce_repl_on_decision()` records the jobid of each DISPATCH and PREEMPT decision (a decision with no body, jobid 0, is skipped); `xce_repl_on_begin_round()` resets the per-round list.\n- src/scheduler/src/repl_rollup.c: `repl_round_rollup_process()` takes over the round's list (sorted, de-duplicated) and makes it part of the fingerprint next to the counts and the pend set, so rounds collapse only when the affected jobs match; `repl_round_emit()` lists the union of the pend set and the dispatch/preempt set (`repl_fmt_affected()`, through the existing `repl_fmt_jobid_range()` compressor) and takes the pend count separately; `repl_round_collapse_flush()` frees the new list. If the per-round list cannot grow (out of memory) or the pend set cannot be built, the round is reported on its own with `jobids=(unavailable)` instead of an empty or partial list.\n- Tests: test_repl_trace_emit.c and test_repl_trace_wiring.c teardowns free the new lists (both run with leak detection on).\n- src/scheduler/testsuite/fixtures/repl_trace_baseline/transcript.txt regenerated (XCE_REPL_BASELINE_REGEN=1, .replay fixtures restored from the backup): exactly two lines change, both `jobids=` lines -- round 5-5 `jobids=0-0-0-107` -> `jobids=0-0-0-104,0-0-0-105,0-0-0-106,0-0-0-107` and round 8-8 `jobids=0-0-0-107` -> `jobids=0-0-0-107,0-0-0-108`.\n\nOutput format change: \"jobids=\" now means every job the range's decisions affected (Requirement 2.1), not only the jobs left pending; the counts are unchanged.\n\nVerification:\n- Test: `test_round_summary_lists_dispatched_jobids_golden` in src/scheduler/testsuite/test_repl_round_rollup.c (exact golden lines for two dispatch-only rounds of different jobs, and a dispatch+preempt round): RED before the fix -- `Expected '[sim] round 1-1 ...: 1 dispatch, 0 pend, 0 preempt -- jobids=0-0-0-5101\\n[sim] round 2-2 ...: 1 dispatch, 0 pend, 0 preempt -- jobids=0-0-0-5102\\n' Was '[sim] round 1-2 (2 round(s), ...): 1 dispatch, 0 pend, 0 preempt -- jobids=\\n'` -- GREEN after (8 Tests 0 Failures). `scheduler repl trace baseline test` with the regenerated transcript: GREEN (4 Tests 0 Failures, leak detection on).\n- Harness: equivalence dump (`XCE_TRACE_EQUIV_DUMP=1`) has 8 empty `jobids=` lines before and 0 after; all 8 dispatch-only rounds now read `round 2-2 ... 1 dispatch, 0 pend, 0 preempt -- jobids=0-0-0-202` (equiv-dump-green.txt).\n- Suites: `bash src/tools/xce-build.sh scheduler --jobs 12 --incremental`: scheduler suite 118/118 OK, asan (log: /home/claude/tmp/bug-1929/gate-scheduler-suite.log).\n- Live: standalone contention run (Reproduction step 3) after the fix: round 8-8 `jobids=0-0-0-107,0-0-0-108`, round 1-1 lists 101-107 (contention-continue-green.txt).", + "tags": [ + "repl", + "sim", + "rollup", + "continue", + "output" + ], + "related_bugs": [ + "bug-1610", + "bug-1611" + ], + "fix_commits": [ + "ba6f42cd" + ], + "record": ".wolf/bugs/bug-1929.md" + }, + { + "id": "bug-1930", + "status": "fixed", + "status_updated": "2026-10-09", + "status_note": "", + "component": "scheduler", + "severity": "wrong-result", + "timestamp": "2026-10-09T03:28:14Z", + "last_seen": "2026-10-09T03:28:14Z", + "occurrences": 1, + "error_message": "xce_scheduler --sim REPL trace on job rejects the raw decimal job id info trace accepts\n\n**Observed:** `trace on job` rejects a raw packed decimal job id, the form `info trace` and `info job` accept for the same job:\n```\n[sim] trace: invalid job id: 104 (invalid job identity: expected --- optionally followed by [k] or [a-b] (decimal digits only, no sign or whitespace))\n```\nwhile `info trace 104` and `info job 104` both resolve job 0-0-0-104. In the unit test the same rejection for a full packed id:\n```\nunexpected output for 'trace on job 81064794341244970': [sim] trace: invalid job id: 81064794341244970 (invalid job identity: expected --- optionally followed by [k] or [a-b] (decimal digits only, no sign or whitespace))\n```\n**Expected:** `trace on job` accepts the same forms as `info trace`/`info job` -- the dashed id, `X[k]`, `X[a-b]` and a raw packed decimal id -- and a raw decimal id selects exactly what its dashed bare form selects (for an array-aligned id, the exact job plus the whole array, tasks.md Implementation Notes 4.3).\n**Impact:** inconsistent REPL grammar: an operator who copies a raw id (as `info job`/`info trace` accept it) into `trace on job` gets an error and no trace. No scheduling effect.\n**Found by:** user report during scheduler-sim-decision-trace validation.", + "file": "src/scheduler/src/repl_trace.c, src/scheduler/src/repl_trace_store.c, src/scheduler/src/repl_internal.h, src/scheduler/testsuite/test_repl_trace_cmd.c", + "root_cause": "`parse_sel()` in src/scheduler/src/repl_trace.c:210 (before the fix) passes each JOBSEL word straight to libxce's `parse_jobsel()`, whose grammar is only the dashed `cluster-apiserver-timestamp-sequence` id with an optional `[k]`/`[a-b]` (utils.h), and reports any error. `info trace`'s `info_trace_sel()` (src/scheduler/src/repl_trace_store.c:456) adds a fallback after a failed `parse_jobsel()`: an all-digits argument is taken as a raw packed id (`strtoull`), matching `info job`. The two selector parsers were written separately and only `info trace` got the fallback.\n\nSince: aa4c43c8 (scheduler-sim-decision-trace task 4.3, the `trace` command parser).\n\nScope: `break ... jobid=` has the opposite problem (it takes only the raw form) and is tracked separately as bug-1926; not changed here.", + "fix": "- src/scheduler/src/repl_trace_store.c: new `repl_parse_jobsel_arg(arg, out, raw)` (declared in repl_internal.h): `parse_jobsel()` plus the raw packed decimal fallback, returned as `JOBSEL_BARE` with `key` = the number and `*raw` = true; a decimal too wide for 64 bits is `JOBSEL_EWIDTH` (before, `info trace` passed it to `strtoull`, which saturated to UINT64_MAX). `info_trace_sel()` uses it and keeps its existing behaviour (a raw id names that job).\n- src/scheduler/src/repl_trace.c `parse_sel()`: uses `repl_parse_jobsel_arg()`. A raw id comes back as the same `JOBSEL_BARE` selector its dashed form produces, so the existing bare-id rule applies unchanged: the exact selector, plus the whole-array selector when the id is array-aligned (Implementation Notes 4.3). repl_trace.c is 652 lines.\n- src/scheduler/testsuite/test_repl_trace_cmd.c: new `test_on_job_raw_decimal_ids`; the old \"`7` is rejected\" case now uses `7x` (a raw `7` is a valid id now), and an over-wide decimal is rejected with the width error.\n\nVerification:\n- Test: `test_on_job_raw_decimal_ids` in src/scheduler/testsuite/test_repl_trace_cmd.c (raw plain id, raw array-aligned id and `104` each give the same selectors as their dashed form; mixed raw/dashed/range line) and the over-wide case in `test_invalid_arguments_leave_settings`: RED before the fix (`test_on_job_raw_decimal_ids: FAIL ... unexpected output for 'trace on job 81064794341244970': [sim] trace: invalid job id ...`; the width case got the syntax error instead), GREEN after (14 Tests 0 Failures, leak detection on).\n- Suites: `bash src/tools/xce-build.sh scheduler --jobs 12 --incremental`: scheduler suite 118/118 OK, asan (log: /home/claude/tmp/bug-1930/gate-scheduler-suite.log).\n- Live: Reproduction step 2 after the fix: `[sim] trace: enabled; applies from the next session`, then `[trace] job 0-0-0-104 ...` blocks for sessions 1-5 under `continue`, and `info trace 104` shows the stored trace (live-green.txt).", + "tags": [ + "repl", + "sim", + "trace", + "jobid" + ], + "related_bugs": [ + "bug-1926" + ], + "fix_commits": [ + "ba6f42cd" + ], + "record": ".wolf/bugs/bug-1930.md" } ] } diff --git a/.wolf/bugs/bug-1928.md b/.wolf/bugs/bug-1928.md new file mode 100644 index 00000000..34b15ddb --- /dev/null +++ b/.wolf/bugs/bug-1928.md @@ -0,0 +1,77 @@ +--- +id: "bug-1928" +status: "fixed" +status_updated: "2026-10-09" +status_note: "" +component: "scheduler" +severity: "leak" +timestamp: "2026-10-09T03:14:08Z" +last_seen: "2026-10-09T03:14:08Z" +occurrences: 1 +files: ["src/scheduler/src/repl.c", "src/scheduler/src/repl_info.c", "src/scheduler/src/repl_rollup.c", "src/scheduler/testsuite/meson.build"] +tags: ["repl", "sim", "leak", "iterator", "asan"] +related: ["bug-1884"] +fix_commits: ["ba6f42cd"] +--- + +# bug-1928: xce_scheduler --sim REPL leaks a job-table iterator every continue round (LeakSanitizer at exit) + +## Description +**Observed:** a normal interactive simulator session (`continue`, then `quit`) on the ASAN build exits 1 from LeakSanitizer, and the buffered console output is lost (0 bytes on stdout): +``` +==4149978==ERROR: LeakSanitizer: detected memory leaks + +Direct leak of 144 byte(s) in 9 object(s) allocated from: + #0 0x79e7779215a3 in calloc (.../libasan.so.8+0x1215a3) + #1 0x5907bfd480ae in xkvmgr_make_iter ../lib/kvmgr.c:159 + #2 0x5907bfde1eca in iter_make (.../xce_scheduler+0x652eca) + #3 0x5907bfcf1004 in repl_count_jobs ../scheduler/src/repl.c:189 + #4 0x5907bfcf5464 in repl_query_pend_jobids ../scheduler/src/repl_rollup.c:94 + #5 0x5907bfcf64bc in repl_round_rollup_process ../scheduler/src/repl_rollup.c:295 + #6 0x5907bfcf3238 in xce_repl_post_session_check ../scheduler/src/repl.c:626 +... +SUMMARY: AddressSanitizer: 160 byte(s) leaked in 10 allocation(s). +``` +**Expected:** the session exits 0 with leak detection on, and its console output reaches stdout. +**Impact:** leak of 16 bytes (one kvmgr iterator) per call of `repl_count_jobs()`: once at the load banner and once per completed round in `continue` mode (Round Rollup), so it grows linearly with the replay length; `info jobs` leaks one more per call. Small, but it makes every standalone ASAN REPL run fail with exit 1 and drop its stdout, and forced several REPL tests to run with `detect_leaks=0` (the trace wiring test's meson comment named this leak as the reason). +**Found by:** ASAN run of the simulator during scheduler-sim-decision-trace validation; the second site (`repl_info_jobs()`) by audit. A grep of every `iter_make()` in src/scheduler/src for a matching `iter_destroy()` on every path found no others. + +## Environment +- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64 +- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2) +- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 +- Build: asan (xce.build), default ASAN_OPTIONS (leak detection on) for the standalone run; the affected meson tests ran with `ASAN_OPTIONS=detect_leaks=0` +- Code: scheduler-framework @ 5ca59dc9-dirty +- Deployment: standalone simulator only (`xce_scheduler --sim ... --interactive`), no controller, no workers; single dev host + +## Reproduction +1. Build: `bash src/tools/xce-build.sh scheduler --jobs 12 --incremental` at the Environment commit. +2. In a temp dir: `printf 'continue\nquit\n' | TERM=dumb src/xce.build/scheduler/xce_scheduler --sim src/scheduler/testsuite/fixtures/repl_trace_baseline/contention.replay --interactive --logfile /sched.log` (paths relative to the repo root) -- exit status 1, LeakSanitizer report on stderr, stdout empty. +3. Smaller: `src/xce.build/scheduler/testsuite/sched_repl_trace_baseline_test` with leak detection on (the child inherits it) fails `test_transcript_matches_baseline: Expected 0 Was 1. the scripted session must end with quit's exit(0)`; `sched_repl_trace_wiring_test` with leak detection on exits 1 with 5 `repl_count_jobs` frames. +- Frequency: deterministic (1/1 standalone runs, every leak-on test run, 2026-10-09) +- Evidence: /home/claude/tmp/bug-1928/red/ (stderr.txt, stdout.txt, per-test logs of a leak-on sweep of every sched_repl_*/sched_sim_* test before the fix), /home/claude/tmp/bug-1928/green/ (same after the fix) + +## Root cause +`repl_count_jobs()` (src/scheduler/src/repl.c:189) creates an iterator over `core->jobs` with `iter_make()`, walks it, and returns the count without calling `iter_destroy()`. For a kvmgr container `iter_make()` heap-allocates the inner iterator (`xkvmgr_make_iter`, lib/kvmgr.c:159) and only `xkvmgr_iter_destroy()` frees it, so each call leaks 16 bytes. It is called from `repl_print_banner()` once and from `repl_query_pend_jobids()` (repl_rollup.c) on every completed round in `continue` mode, hence the per-round growth. `repl_info_jobs()` (src/scheduler/src/repl_info.c:172, `info jobs`) has the same missing `iter_destroy()`. The leak makes LeakSanitizer turn the exit status into 1 at exit, and the process ends before stdio flushes the redirected stdout. + +Since: `repl_count_jobs()` since 02f7f48f (scheduler-core REPL task 12.1); `repl_info_jobs()` since c705cada. repl_rollup.c's own comment acknowledged the missing destroy as a "low-impact quirk", but its own per-round caller made it a growing leak. + +Scope: every `iter_make()` in src/scheduler/src (core.c, core_session.c, plugin.c, plugins/limit.c, host_iter.c, scheduler.c, snapshot.c, sim_loop.c, sim_picker.c, repl_info.c, repl_rollup.c, repl_trace_store.c) was checked; all others destroy the iterator on every path, and their early returns are only on `iter_make()` failure, where kvmgr frees the iterator itself. `host_iter.c` keeps its iterator in the host-iterator private data and destroys it in its fini. + +## Fix +- src/scheduler/src/repl.c `repl_count_jobs()`: call `iter_destroy(&it)` after the walk. +- src/scheduler/src/repl_info.c `repl_info_jobs()`: same. +- src/scheduler/src/repl_rollup.c: replaced the stale comment that called the missing destroy a quirk. +- src/scheduler/testsuite/meson.build: `scheduler repl trace baseline test` and `scheduler repl trace wiring test` now run with leak detection on (removed `ASAN_OPTIONS=detect_leaks=0` and the comment blaming this leak). The other REPL/sim tests still leak from their own fixtures (cores and replay writers never torn down), so they keep `detect_leaks=0`. + +Every iterator the REPL makes is now destroyed, so nothing is left for LeakSanitizer at exit. + +## Verification +- Test: `scheduler repl trace baseline test` (test_repl_trace_baseline.c, leak detection now on): RED before the fix (`test_transcript_matches_baseline: Expected 0 Was 1`, the child exits 1 from LeakSanitizer), GREEN after (4 Tests 0 Failures). `scheduler repl trace wiring test` (test_repl_trace_wiring.c, leak detection now on): RED before (exit 1, LeakSanitizer with `repl_count_jobs` frames), GREEN after (17 Tests 0 Failures). +- Leak-on sweep of all 42 sched_repl_*/sched_sim_* test binaries: `repl_count_jobs`/`repl_info_jobs` frames 0 in every log after the fix (before: present in 13). +- Suites: `bash src/tools/xce-build.sh scheduler --jobs 12 --incremental`: scheduler suite 118/118 OK, asan (log: /home/claude/tmp/bug-1928/gate-scheduler-suite.log). +- Live: the standalone command in Reproduction step 2 now exits 0 with leak detection on and prints its 18-line transcript (/home/claude/tmp/bug-1928/green/). + +## References +- Related: bug-1884 (earlier standalone `--sim` LeakSanitizer exit-1 fix; this is a REPL-only leak it did not cover) +- Docs: src/scheduler/CLAUDE.md (Interactive REPL) diff --git a/.wolf/bugs/bug-1929.md b/.wolf/bugs/bug-1929.md new file mode 100644 index 00000000..0aa06cc1 --- /dev/null +++ b/.wolf/bugs/bug-1929.md @@ -0,0 +1,73 @@ +--- +id: "bug-1929" +status: "fixed" +status_updated: "2026-10-09" +status_note: "" +component: "scheduler" +severity: "wrong-result" +timestamp: "2026-10-09T03:21:26Z" +last_seen: "2026-10-09T03:21:26Z" +occurrences: 1 +files: ["src/scheduler/src/repl_rollup.c", "src/scheduler/src/repl_break.c", "src/scheduler/src/repl_internal.h", "src/scheduler/testsuite/test_repl_round_rollup.c", "src/scheduler/testsuite/test_repl_trace_emit.c", "src/scheduler/testsuite/test_repl_trace_wiring.c", "src/scheduler/testsuite/fixtures/repl_trace_baseline/transcript.txt"] +tags: ["repl", "sim", "rollup", "continue", "output"] +related: ["bug-1610", "bug-1611"] +fix_commits: ["ba6f42cd"] +--- + +# bug-1929: xce_scheduler --sim REPL continue round summary prints empty jobids= for a dispatch-only round + +## Description +**Observed:** in `continue` mode a round whose only decision is a dispatch prints a round summary with an empty job list, with tracing off as well as on. From the 6.3 trace-equivalence harness (`XCE_TRACE_EQUIV_DUMP=1`, extra replay, 8 occurrences): +``` +[sim] round 1-1 (1 round(s), t=1970-01-01T00:00:00Z..1970-01-01T00:00:00Z): 4 dispatch, 1 pend, 0 preempt -- jobids=0-0-0-202 +[sim] round 2-2 (1 round(s), t=1970-01-01T00:00:00Z..1970-01-01T00:00:00Z): 1 dispatch, 0 pend, 0 preempt -- jobids= +``` +With the 1.1 contention fixture the dispatched job is missing even when a pend job is listed (round 8 dispatched 0-0-0-108): +``` +[sim] round 8-8 (1 round(s), t=1970-01-01T00:00:00Z..1970-01-01T00:00:00Z): 1 dispatch, 1 pend, 0 preempt -- jobids=0-0-0-107 +``` +**Expected:** "jobids=" names the jobs the round's decisions affected (scheduler-sim-repl-verbosity Requirement 2.1: "the count of each decision type and the identifiers of the jobs affected"), so the dispatched (and preempted) jobs are listed: `jobids=0-0-0-202` (the job pending in round 1 and dispatched in round 2) and `jobids=0-0-0-107,0-0-0-108`. +**Impact:** wrong/misleading REPL output only: the operator cannot see which job a round dispatched or preempted. A second effect of the same cause: rounds with the same counts and pend set but different dispatched jobs collapse into one range (Requirement 3.2 says a range needs the same affected job identifiers). Scheduling decisions are not affected. +**Found by:** scheduler-sim-decision-trace validation (rollup output in the 6.1/6.3 e2e and equivalence harness dumps). + +## Environment +- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64 +- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2) +- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 +- Build: asan (xce.build), tests run with `ASAN_OPTIONS=detect_leaks=0` (suite convention for these binaries); the symptom does not depend on sanitizer options or on tracing +- Code: scheduler-framework @ 5ca59dc9-dirty +- Deployment: standalone simulator only (`xce_scheduler --sim ... --interactive`), no controller, no workers; single dev host + +## Reproduction +1. Build: `bash src/tools/xce-build.sh scheduler --jobs 12 --incremental` at the Environment commit. +2. `cd src/xce.build/scheduler/testsuite && XCE_TRACE_EQUIV_DUMP=1 ASAN_OPTIONS=detect_leaks=0 ./sched_trace_equivalence_test 2>&1 | grep 'jobids=$'` -- 8 lines `... 1 dispatch, 0 pend, 0 preempt -- jobids=`. +3. Contention fixture: `printf 'continue\nquit\n' | TERM=dumb src/xce.build/scheduler/xce_scheduler --sim src/scheduler/testsuite/fixtures/repl_trace_baseline/contention.replay --interactive --logfile /sched.log` (repo root; with bug-1928 unfixed add `ASAN_OPTIONS=detect_leaks=0`) -- round 8-8 lists only 0-0-0-107; rounds 1-1 and 5-5 list only the pend set. +4. Smallest: `test_round_summary_lists_dispatched_jobids_golden` in test_repl_round_rollup.c (drives xce_repl_on_begin_round/xce_repl_on_decision/repl_round_rollup_process directly). +- Frequency: deterministic (every run, 2026-10-09) +- Evidence: /home/claude/tmp/bug-1929/ -- equiv-dump-red.txt / equiv-dump-green.txt, e2e-dump-red.txt, contention-continue-red.txt / contention-continue-green.txt, rollup-red.txt, transcript.diff, fixture-backup/ (the baseline fixtures before regeneration) + +## Root cause +`repl_round_emit()` (src/scheduler/src/repl_rollup.c:178, listing built at :201-202 before the fix) built the "jobids=" listing from `pend_jobids` only, and `repl_round_rollup_process()` (repl_rollup.c:285) only ever collected the pend set (`repl_query_pend_jobids()`, a walk of core->jobs for JOB_IS_PEND jobs). The dispatch and preempt decisions were counted in `xce_repl_on_decision()` (src/scheduler/src/repl_break.c:953-959, `last_n_dispatch`/`last_n_preempt`) but their job ids were never recorded anywhere. A dispatched job is no longer pending after the round, so a dispatch-only round has an empty pend set and printed `jobids=` with nothing after it; in mixed rounds the dispatched jobs were silently left out. The same pend-only fingerprint made rounds that dispatched different jobs compare equal and collapse. + +Since: 921c72bb (scheduler-sim-repl-verbosity task 2.2, Round Rollup), written while bug-1611 meant the simulator produced no DISPATCH decisions, so the pend set was the only affected-job information then. + +Scope: the traced-round path (scheduler-sim-decision-trace 7.3/7.4) and the allocation-failure fallback go through the same `repl_round_emit()`; step mode prints per-decision lines and was not affected. + +## Fix +- src/scheduler/src/repl_internal.h: new state in `xce_sched_repl_state_s` -- `round_decided_jobids`/`round_n_decided`/`round_cap_decided`/`round_decided_lost` (this round's dispatch/preempt jobids) and `collapse_decided_jobids`/`collapse_n_decided_jobids` (the active range's); declarations of `repl_round_note_decided()`/`repl_round_reset_decided()`. +- src/scheduler/src/repl_break.c: `xce_repl_on_decision()` records the jobid of each DISPATCH and PREEMPT decision (a decision with no body, jobid 0, is skipped); `xce_repl_on_begin_round()` resets the per-round list. +- src/scheduler/src/repl_rollup.c: `repl_round_rollup_process()` takes over the round's list (sorted, de-duplicated) and makes it part of the fingerprint next to the counts and the pend set, so rounds collapse only when the affected jobs match; `repl_round_emit()` lists the union of the pend set and the dispatch/preempt set (`repl_fmt_affected()`, through the existing `repl_fmt_jobid_range()` compressor) and takes the pend count separately; `repl_round_collapse_flush()` frees the new list. If the per-round list cannot grow (out of memory) or the pend set cannot be built, the round is reported on its own with `jobids=(unavailable)` instead of an empty or partial list. +- Tests: test_repl_trace_emit.c and test_repl_trace_wiring.c teardowns free the new lists (both run with leak detection on). +- src/scheduler/testsuite/fixtures/repl_trace_baseline/transcript.txt regenerated (XCE_REPL_BASELINE_REGEN=1, .replay fixtures restored from the backup): exactly two lines change, both `jobids=` lines -- round 5-5 `jobids=0-0-0-107` -> `jobids=0-0-0-104,0-0-0-105,0-0-0-106,0-0-0-107` and round 8-8 `jobids=0-0-0-107` -> `jobids=0-0-0-107,0-0-0-108`. + +Output format change: "jobids=" now means every job the range's decisions affected (Requirement 2.1), not only the jobs left pending; the counts are unchanged. + +## Verification +- Test: `test_round_summary_lists_dispatched_jobids_golden` in src/scheduler/testsuite/test_repl_round_rollup.c (exact golden lines for two dispatch-only rounds of different jobs, and a dispatch+preempt round): RED before the fix -- `Expected '[sim] round 1-1 ...: 1 dispatch, 0 pend, 0 preempt -- jobids=0-0-0-5101\n[sim] round 2-2 ...: 1 dispatch, 0 pend, 0 preempt -- jobids=0-0-0-5102\n' Was '[sim] round 1-2 (2 round(s), ...): 1 dispatch, 0 pend, 0 preempt -- jobids=\n'` -- GREEN after (8 Tests 0 Failures). `scheduler repl trace baseline test` with the regenerated transcript: GREEN (4 Tests 0 Failures, leak detection on). +- Harness: equivalence dump (`XCE_TRACE_EQUIV_DUMP=1`) has 8 empty `jobids=` lines before and 0 after; all 8 dispatch-only rounds now read `round 2-2 ... 1 dispatch, 0 pend, 0 preempt -- jobids=0-0-0-202` (equiv-dump-green.txt). +- Suites: `bash src/tools/xce-build.sh scheduler --jobs 12 --incremental`: scheduler suite 118/118 OK, asan (log: /home/claude/tmp/bug-1929/gate-scheduler-suite.log). +- Live: standalone contention run (Reproduction step 3) after the fix: round 8-8 `jobids=0-0-0-107,0-0-0-108`, round 1-1 lists 101-107 (contention-continue-green.txt). + +## References +- Related: bug-1611 (no DISPATCH decisions in sim at the time the rollup was written), bug-1610 (rollup introduced to replace per-decision lines) +- Docs: .kiro/specs/scheduler-sim-repl-verbosity/requirements.md (Requirement 2.1, 3.2); src/scheduler/testsuite/fixtures/repl_trace_baseline/transcript.txt diff --git a/.wolf/bugs/bug-1930.md b/.wolf/bugs/bug-1930.md new file mode 100644 index 00000000..cea0b1a5 --- /dev/null +++ b/.wolf/bugs/bug-1930.md @@ -0,0 +1,66 @@ +--- +id: "bug-1930" +status: "fixed" +status_updated: "2026-10-09" +status_note: "" +component: "scheduler" +severity: "wrong-result" +timestamp: "2026-10-09T03:28:14Z" +last_seen: "2026-10-09T03:28:14Z" +occurrences: 1 +files: ["src/scheduler/src/repl_trace.c", "src/scheduler/src/repl_trace_store.c", "src/scheduler/src/repl_internal.h", "src/scheduler/testsuite/test_repl_trace_cmd.c"] +tags: ["repl", "sim", "trace", "jobid"] +related: ["bug-1926"] +fix_commits: ["ba6f42cd"] +--- + +# bug-1930: xce_scheduler --sim REPL trace on job rejects the raw decimal job id info trace accepts + +## Description +**Observed:** `trace on job` rejects a raw packed decimal job id, the form `info trace` and `info job` accept for the same job: +``` +[sim] trace: invalid job id: 104 (invalid job identity: expected --- optionally followed by [k] or [a-b] (decimal digits only, no sign or whitespace)) +``` +while `info trace 104` and `info job 104` both resolve job 0-0-0-104. In the unit test the same rejection for a full packed id: +``` +unexpected output for 'trace on job 81064794341244970': [sim] trace: invalid job id: 81064794341244970 (invalid job identity: expected --- optionally followed by [k] or [a-b] (decimal digits only, no sign or whitespace)) +``` +**Expected:** `trace on job` accepts the same forms as `info trace`/`info job` -- the dashed id, `X[k]`, `X[a-b]` and a raw packed decimal id -- and a raw decimal id selects exactly what its dashed bare form selects (for an array-aligned id, the exact job plus the whole array, tasks.md Implementation Notes 4.3). +**Impact:** inconsistent REPL grammar: an operator who copies a raw id (as `info job`/`info trace` accept it) into `trace on job` gets an error and no trace. No scheduling effect. +**Found by:** user report during scheduler-sim-decision-trace validation. + +## Environment +- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64 +- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2) +- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 +- Build: asan (xce.build), default ASAN_OPTIONS; the symptom does not depend on sanitizer options +- Code: scheduler-framework @ 5ca59dc9-dirty +- Deployment: standalone simulator only (`xce_scheduler --sim ... --interactive`), no controller, no workers; single dev host + +## Reproduction +1. Build: `bash src/tools/xce-build.sh scheduler --jobs 12 --incremental` at the Environment commit. +2. `printf 'trace on job 104\ninfo trace 104\nquit\n' | TERM=dumb ASAN_OPTIONS=detect_leaks=0 src/xce.build/scheduler/xce_scheduler --sim src/scheduler/testsuite/fixtures/repl_trace_baseline/contention.replay --interactive --logfile /sched.log` (repo root) -- `trace on job 104` prints the "invalid job id" error above; `info trace 104` accepts the id. +3. Smaller: `test_on_job_raw_decimal_ids` in src/scheduler/testsuite/test_repl_trace_cmd.c. +- Frequency: deterministic (every run, 2026-10-09) +- Evidence: /home/claude/tmp/bug-1930/ -- trace-cmd-red.txt (unit test before the fix), trace-cmd-green.txt, live-green.txt (`trace on job 104` after the fix, tracing 0-0-0-104 through `continue` and `info trace 104`) + +## Root cause +`parse_sel()` in src/scheduler/src/repl_trace.c:210 (before the fix) passes each JOBSEL word straight to libxce's `parse_jobsel()`, whose grammar is only the dashed `cluster-apiserver-timestamp-sequence` id with an optional `[k]`/`[a-b]` (utils.h), and reports any error. `info trace`'s `info_trace_sel()` (src/scheduler/src/repl_trace_store.c:456) adds a fallback after a failed `parse_jobsel()`: an all-digits argument is taken as a raw packed id (`strtoull`), matching `info job`. The two selector parsers were written separately and only `info trace` got the fallback. + +Since: aa4c43c8 (scheduler-sim-decision-trace task 4.3, the `trace` command parser). + +Scope: `break ... jobid=` has the opposite problem (it takes only the raw form) and is tracked separately as bug-1926; not changed here. + +## Fix +- src/scheduler/src/repl_trace_store.c: new `repl_parse_jobsel_arg(arg, out, raw)` (declared in repl_internal.h): `parse_jobsel()` plus the raw packed decimal fallback, returned as `JOBSEL_BARE` with `key` = the number and `*raw` = true; a decimal too wide for 64 bits is `JOBSEL_EWIDTH` (before, `info trace` passed it to `strtoull`, which saturated to UINT64_MAX). `info_trace_sel()` uses it and keeps its existing behaviour (a raw id names that job). +- src/scheduler/src/repl_trace.c `parse_sel()`: uses `repl_parse_jobsel_arg()`. A raw id comes back as the same `JOBSEL_BARE` selector its dashed form produces, so the existing bare-id rule applies unchanged: the exact selector, plus the whole-array selector when the id is array-aligned (Implementation Notes 4.3). repl_trace.c is 652 lines. +- src/scheduler/testsuite/test_repl_trace_cmd.c: new `test_on_job_raw_decimal_ids`; the old "`7` is rejected" case now uses `7x` (a raw `7` is a valid id now), and an over-wide decimal is rejected with the width error. + +## Verification +- Test: `test_on_job_raw_decimal_ids` in src/scheduler/testsuite/test_repl_trace_cmd.c (raw plain id, raw array-aligned id and `104` each give the same selectors as their dashed form; mixed raw/dashed/range line) and the over-wide case in `test_invalid_arguments_leave_settings`: RED before the fix (`test_on_job_raw_decimal_ids: FAIL ... unexpected output for 'trace on job 81064794341244970': [sim] trace: invalid job id ...`; the width case got the syntax error instead), GREEN after (14 Tests 0 Failures, leak detection on). +- Suites: `bash src/tools/xce-build.sh scheduler --jobs 12 --incremental`: scheduler suite 118/118 OK, asan (log: /home/claude/tmp/bug-1930/gate-scheduler-suite.log). +- Live: Reproduction step 2 after the fix: `[sim] trace: enabled; applies from the next session`, then `[trace] job 0-0-0-104 ...` blocks for sessions 1-5 under `continue`, and `info trace 104` shows the stored trace (live-green.txt). + +## References +- Related: bug-1926 (`break ... jobid=` accepts only the raw form; separate fix) +- Docs: .kiro/specs/scheduler-sim-decision-trace/tasks.md (Implementation Notes 4.3), requirements.md 1.2-1.3