docs(buglog): bug-1928, bug-1929, bug-1930 fixed (ba6f42cd)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
ba6f42cdc2
commit
c3cdf2f515
GPG key ID:
80533536B0EB651C
4 changed files with 303 additions and 0 deletions
|
|
@ -17566,6 +17566,93 @@
|
|||
"record": ".wolf/bugs/bug-1927.md",
|
||||
"gitea_issue": 352,
|
||||
"gitea_synced_status": "open"
|
||||
},
|
||||
{
|
||||
"id": "bug-1928",
|
||||
"status": "fixed",
|
||||
"status_updated": "2026-10-09",
|
||||
"status_note": "",
|
||||
"component": "scheduler",
|
||||
"severity": "leak",
|
||||
"timestamp": "2026-10-09T03:14:08Z",
|
||||
"last_seen": "2026-10-09T03:14:08Z",
|
||||
"occurrences": 1,
|
||||
"error_message": "xce_scheduler --sim REPL leaks a job-table iterator every continue round (LeakSanitizer at exit)\n\n**Observed:** a normal interactive simulator session (`continue`, then `quit`) on the ASAN build exits 1 from LeakSanitizer, and the buffered console output is lost (0 bytes on stdout):\n```\n==4149978==ERROR: LeakSanitizer: detected memory leaks\n\nDirect leak of 144 byte(s) in 9 object(s) allocated from:\n #0 0x79e7779215a3 in calloc (.../libasan.so.8+0x1215a3)\n #1 0x5907bfd480ae in xkvmgr_make_iter ../lib/kvmgr.c:159\n #2 0x5907bfde1eca in iter_make (.../xce_scheduler+0x652eca)\n #3 0x5907bfcf1004 in repl_count_jobs ../scheduler/src/repl.c:189\n #4 0x5907bfcf5464 in repl_query_pend_jobids ../scheduler/src/repl_rollup.c:94\n #5 0x5907bfcf64bc in repl_round_rollup_process ../scheduler/src/repl_rollup.c:295\n #6 0x5907bfcf3238 in xce_repl_post_session_check ../scheduler/src/repl.c:626\n...\nSUMMARY: AddressSanitizer: 160 byte(s) leaked in 10 allocation(s).\n```\n**Expected:** the session exits 0 with leak detection on, and its console output reaches stdout.\n**Impact:** leak of 16 bytes (one kvmgr iterator) per call of `repl_count_jobs()`: once at the load banner and once per completed round in `continue` mode (Round Rollup), so it grows linearly with the replay length; `info jobs` leaks one more per call. Small, but it makes every standalone ASAN REPL run fail with exit 1 and drop its stdout, and forced several REPL tests to run with `detect_leaks=0` (the trace wiring test's meson comment named this leak as the reason).\n**Found by:** ASAN run of the simulator during scheduler-sim-decision-trace validation; the second site (`repl_info_jobs()`) by audit. A grep of every `iter_make()` in src/scheduler/src for a matching `iter_destroy()` on every path found no others.",
|
||||
"file": "src/scheduler/src/repl.c, src/scheduler/src/repl_info.c, src/scheduler/src/repl_rollup.c, src/scheduler/testsuite/meson.build",
|
||||
"root_cause": "`repl_count_jobs()` (src/scheduler/src/repl.c:189) creates an iterator over `core->jobs` with `iter_make()`, walks it, and returns the count without calling `iter_destroy()`. For a kvmgr container `iter_make()` heap-allocates the inner iterator (`xkvmgr_make_iter`, lib/kvmgr.c:159) and only `xkvmgr_iter_destroy()` frees it, so each call leaks 16 bytes. It is called from `repl_print_banner()` once and from `repl_query_pend_jobids()` (repl_rollup.c) on every completed round in `continue` mode, hence the per-round growth. `repl_info_jobs()` (src/scheduler/src/repl_info.c:172, `info jobs`) has the same missing `iter_destroy()`. The leak makes LeakSanitizer turn the exit status into 1 at exit, and the process ends before stdio flushes the redirected stdout.\n\nSince: `repl_count_jobs()` since 02f7f48f (scheduler-core REPL task 12.1); `repl_info_jobs()` since c705cada. repl_rollup.c's own comment acknowledged the missing destroy as a \"low-impact quirk\", but its own per-round caller made it a growing leak.\n\nScope: every `iter_make()` in src/scheduler/src (core.c, core_session.c, plugin.c, plugins/limit.c, host_iter.c, scheduler.c, snapshot.c, sim_loop.c, sim_picker.c, repl_info.c, repl_rollup.c, repl_trace_store.c) was checked; all others destroy the iterator on every path, and their early returns are only on `iter_make()` failure, where kvmgr frees the iterator itself. `host_iter.c` keeps its iterator in the host-iterator private data and destroys it in its fini.",
|
||||
"fix": "- src/scheduler/src/repl.c `repl_count_jobs()`: call `iter_destroy(&it)` after the walk.\n- src/scheduler/src/repl_info.c `repl_info_jobs()`: same.\n- src/scheduler/src/repl_rollup.c: replaced the stale comment that called the missing destroy a quirk.\n- src/scheduler/testsuite/meson.build: `scheduler repl trace baseline test` and `scheduler repl trace wiring test` now run with leak detection on (removed `ASAN_OPTIONS=detect_leaks=0` and the comment blaming this leak). The other REPL/sim tests still leak from their own fixtures (cores and replay writers never torn down), so they keep `detect_leaks=0`.\n\nEvery iterator the REPL makes is now destroyed, so nothing is left for LeakSanitizer at exit.\n\nVerification:\n- Test: `scheduler repl trace baseline test` (test_repl_trace_baseline.c, leak detection now on): RED before the fix (`test_transcript_matches_baseline: Expected 0 Was 1`, the child exits 1 from LeakSanitizer), GREEN after (4 Tests 0 Failures). `scheduler repl trace wiring test` (test_repl_trace_wiring.c, leak detection now on): RED before (exit 1, LeakSanitizer with `repl_count_jobs` frames), GREEN after (17 Tests 0 Failures).\n- Leak-on sweep of all 42 sched_repl_*/sched_sim_* test binaries: `repl_count_jobs`/`repl_info_jobs` frames 0 in every log after the fix (before: present in 13).\n- Suites: `bash src/tools/xce-build.sh scheduler --jobs 12 --incremental`: scheduler suite 118/118 OK, asan (log: /home/claude/tmp/bug-1928/gate-scheduler-suite.log).\n- Live: the standalone command in Reproduction step 2 now exits 0 with leak detection on and prints its 18-line transcript (/home/claude/tmp/bug-1928/green/).",
|
||||
"tags": [
|
||||
"repl",
|
||||
"sim",
|
||||
"leak",
|
||||
"iterator",
|
||||
"asan"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1884"
|
||||
],
|
||||
"fix_commits": [
|
||||
"ba6f42cd"
|
||||
],
|
||||
"record": ".wolf/bugs/bug-1928.md"
|
||||
},
|
||||
{
|
||||
"id": "bug-1929",
|
||||
"status": "fixed",
|
||||
"status_updated": "2026-10-09",
|
||||
"status_note": "",
|
||||
"component": "scheduler",
|
||||
"severity": "wrong-result",
|
||||
"timestamp": "2026-10-09T03:21:26Z",
|
||||
"last_seen": "2026-10-09T03:21:26Z",
|
||||
"occurrences": 1,
|
||||
"error_message": "xce_scheduler --sim REPL continue round summary prints empty jobids= for a dispatch-only round\n\n**Observed:** in `continue` mode a round whose only decision is a dispatch prints a round summary with an empty job list, with tracing off as well as on. From the 6.3 trace-equivalence harness (`XCE_TRACE_EQUIV_DUMP=1`, extra replay, 8 occurrences):\n```\n[sim] round 1-1 (1 round(s), t=1970-01-01T00:00:00Z..1970-01-01T00:00:00Z): 4 dispatch, 1 pend, 0 preempt -- jobids=0-0-0-202\n[sim] round 2-2 (1 round(s), t=1970-01-01T00:00:00Z..1970-01-01T00:00:00Z): 1 dispatch, 0 pend, 0 preempt -- jobids=\n```\nWith the 1.1 contention fixture the dispatched job is missing even when a pend job is listed (round 8 dispatched 0-0-0-108):\n```\n[sim] round 8-8 (1 round(s), t=1970-01-01T00:00:00Z..1970-01-01T00:00:00Z): 1 dispatch, 1 pend, 0 preempt -- jobids=0-0-0-107\n```\n**Expected:** \"jobids=\" names the jobs the round's decisions affected (scheduler-sim-repl-verbosity Requirement 2.1: \"the count of each decision type and the identifiers of the jobs affected\"), so the dispatched (and preempted) jobs are listed: `jobids=0-0-0-202` (the job pending in round 1 and dispatched in round 2) and `jobids=0-0-0-107,0-0-0-108`.\n**Impact:** wrong/misleading REPL output only: the operator cannot see which job a round dispatched or preempted. A second effect of the same cause: rounds with the same counts and pend set but different dispatched jobs collapse into one range (Requirement 3.2 says a range needs the same affected job identifiers). Scheduling decisions are not affected.\n**Found by:** scheduler-sim-decision-trace validation (rollup output in the 6.1/6.3 e2e and equivalence harness dumps).",
|
||||
"file": "src/scheduler/src/repl_rollup.c, src/scheduler/src/repl_break.c, src/scheduler/src/repl_internal.h, src/scheduler/testsuite/test_repl_round_rollup.c, src/scheduler/testsuite/test_repl_trace_emit.c, src/scheduler/testsuite/test_repl_trace_wiring.c, src/scheduler/testsuite/fixtures/repl_trace_baseline/transcript.txt",
|
||||
"root_cause": "`repl_round_emit()` (src/scheduler/src/repl_rollup.c:178, listing built at :201-202 before the fix) built the \"jobids=\" listing from `pend_jobids` only, and `repl_round_rollup_process()` (repl_rollup.c:285) only ever collected the pend set (`repl_query_pend_jobids()`, a walk of core->jobs for JOB_IS_PEND jobs). The dispatch and preempt decisions were counted in `xce_repl_on_decision()` (src/scheduler/src/repl_break.c:953-959, `last_n_dispatch`/`last_n_preempt`) but their job ids were never recorded anywhere. A dispatched job is no longer pending after the round, so a dispatch-only round has an empty pend set and printed `jobids=` with nothing after it; in mixed rounds the dispatched jobs were silently left out. The same pend-only fingerprint made rounds that dispatched different jobs compare equal and collapse.\n\nSince: 921c72bb (scheduler-sim-repl-verbosity task 2.2, Round Rollup), written while bug-1611 meant the simulator produced no DISPATCH decisions, so the pend set was the only affected-job information then.\n\nScope: the traced-round path (scheduler-sim-decision-trace 7.3/7.4) and the allocation-failure fallback go through the same `repl_round_emit()`; step mode prints per-decision lines and was not affected.",
|
||||
"fix": "- src/scheduler/src/repl_internal.h: new state in `xce_sched_repl_state_s` -- `round_decided_jobids`/`round_n_decided`/`round_cap_decided`/`round_decided_lost` (this round's dispatch/preempt jobids) and `collapse_decided_jobids`/`collapse_n_decided_jobids` (the active range's); declarations of `repl_round_note_decided()`/`repl_round_reset_decided()`.\n- src/scheduler/src/repl_break.c: `xce_repl_on_decision()` records the jobid of each DISPATCH and PREEMPT decision (a decision with no body, jobid 0, is skipped); `xce_repl_on_begin_round()` resets the per-round list.\n- src/scheduler/src/repl_rollup.c: `repl_round_rollup_process()` takes over the round's list (sorted, de-duplicated) and makes it part of the fingerprint next to the counts and the pend set, so rounds collapse only when the affected jobs match; `repl_round_emit()` lists the union of the pend set and the dispatch/preempt set (`repl_fmt_affected()`, through the existing `repl_fmt_jobid_range()` compressor) and takes the pend count separately; `repl_round_collapse_flush()` frees the new list. If the per-round list cannot grow (out of memory) or the pend set cannot be built, the round is reported on its own with `jobids=(unavailable)` instead of an empty or partial list.\n- Tests: test_repl_trace_emit.c and test_repl_trace_wiring.c teardowns free the new lists (both run with leak detection on).\n- src/scheduler/testsuite/fixtures/repl_trace_baseline/transcript.txt regenerated (XCE_REPL_BASELINE_REGEN=1, .replay fixtures restored from the backup): exactly two lines change, both `jobids=` lines -- round 5-5 `jobids=0-0-0-107` -> `jobids=0-0-0-104,0-0-0-105,0-0-0-106,0-0-0-107` and round 8-8 `jobids=0-0-0-107` -> `jobids=0-0-0-107,0-0-0-108`.\n\nOutput format change: \"jobids=\" now means every job the range's decisions affected (Requirement 2.1), not only the jobs left pending; the counts are unchanged.\n\nVerification:\n- Test: `test_round_summary_lists_dispatched_jobids_golden` in src/scheduler/testsuite/test_repl_round_rollup.c (exact golden lines for two dispatch-only rounds of different jobs, and a dispatch+preempt round): RED before the fix -- `Expected '[sim] round 1-1 ...: 1 dispatch, 0 pend, 0 preempt -- jobids=0-0-0-5101\\n[sim] round 2-2 ...: 1 dispatch, 0 pend, 0 preempt -- jobids=0-0-0-5102\\n' Was '[sim] round 1-2 (2 round(s), ...): 1 dispatch, 0 pend, 0 preempt -- jobids=\\n'` -- GREEN after (8 Tests 0 Failures). `scheduler repl trace baseline test` with the regenerated transcript: GREEN (4 Tests 0 Failures, leak detection on).\n- Harness: equivalence dump (`XCE_TRACE_EQUIV_DUMP=1`) has 8 empty `jobids=` lines before and 0 after; all 8 dispatch-only rounds now read `round 2-2 ... 1 dispatch, 0 pend, 0 preempt -- jobids=0-0-0-202` (equiv-dump-green.txt).\n- Suites: `bash src/tools/xce-build.sh scheduler --jobs 12 --incremental`: scheduler suite 118/118 OK, asan (log: /home/claude/tmp/bug-1929/gate-scheduler-suite.log).\n- Live: standalone contention run (Reproduction step 3) after the fix: round 8-8 `jobids=0-0-0-107,0-0-0-108`, round 1-1 lists 101-107 (contention-continue-green.txt).",
|
||||
"tags": [
|
||||
"repl",
|
||||
"sim",
|
||||
"rollup",
|
||||
"continue",
|
||||
"output"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1610",
|
||||
"bug-1611"
|
||||
],
|
||||
"fix_commits": [
|
||||
"ba6f42cd"
|
||||
],
|
||||
"record": ".wolf/bugs/bug-1929.md"
|
||||
},
|
||||
{
|
||||
"id": "bug-1930",
|
||||
"status": "fixed",
|
||||
"status_updated": "2026-10-09",
|
||||
"status_note": "",
|
||||
"component": "scheduler",
|
||||
"severity": "wrong-result",
|
||||
"timestamp": "2026-10-09T03:28:14Z",
|
||||
"last_seen": "2026-10-09T03:28:14Z",
|
||||
"occurrences": 1,
|
||||
"error_message": "xce_scheduler --sim REPL trace on job rejects the raw decimal job id info trace accepts\n\n**Observed:** `trace on job` rejects a raw packed decimal job id, the form `info trace` and `info job` accept for the same job:\n```\n[sim] trace: invalid job id: 104 (invalid job identity: expected <cluster>-<apiserver>-<timestamp>-<sequence> optionally followed by [k] or [a-b] (decimal digits only, no sign or whitespace))\n```\nwhile `info trace 104` and `info job 104` both resolve job 0-0-0-104. In the unit test the same rejection for a full packed id:\n```\nunexpected output for 'trace on job 81064794341244970': [sim] trace: invalid job id: 81064794341244970 (invalid job identity: expected <cluster>-<apiserver>-<timestamp>-<sequence> optionally followed by [k] or [a-b] (decimal digits only, no sign or whitespace))\n```\n**Expected:** `trace on job` accepts the same forms as `info trace`/`info job` -- the dashed id, `X[k]`, `X[a-b]` and a raw packed decimal id -- and a raw decimal id selects exactly what its dashed bare form selects (for an array-aligned id, the exact job plus the whole array, tasks.md Implementation Notes 4.3).\n**Impact:** inconsistent REPL grammar: an operator who copies a raw id (as `info job`/`info trace` accept it) into `trace on job` gets an error and no trace. No scheduling effect.\n**Found by:** user report during scheduler-sim-decision-trace validation.",
|
||||
"file": "src/scheduler/src/repl_trace.c, src/scheduler/src/repl_trace_store.c, src/scheduler/src/repl_internal.h, src/scheduler/testsuite/test_repl_trace_cmd.c",
|
||||
"root_cause": "`parse_sel()` in src/scheduler/src/repl_trace.c:210 (before the fix) passes each JOBSEL word straight to libxce's `parse_jobsel()`, whose grammar is only the dashed `cluster-apiserver-timestamp-sequence` id with an optional `[k]`/`[a-b]` (utils.h), and reports any error. `info trace`'s `info_trace_sel()` (src/scheduler/src/repl_trace_store.c:456) adds a fallback after a failed `parse_jobsel()`: an all-digits argument is taken as a raw packed id (`strtoull`), matching `info job`. The two selector parsers were written separately and only `info trace` got the fallback.\n\nSince: aa4c43c8 (scheduler-sim-decision-trace task 4.3, the `trace` command parser).\n\nScope: `break ... jobid=` has the opposite problem (it takes only the raw form) and is tracked separately as bug-1926; not changed here.",
|
||||
"fix": "- src/scheduler/src/repl_trace_store.c: new `repl_parse_jobsel_arg(arg, out, raw)` (declared in repl_internal.h): `parse_jobsel()` plus the raw packed decimal fallback, returned as `JOBSEL_BARE` with `key` = the number and `*raw` = true; a decimal too wide for 64 bits is `JOBSEL_EWIDTH` (before, `info trace` passed it to `strtoull`, which saturated to UINT64_MAX). `info_trace_sel()` uses it and keeps its existing behaviour (a raw id names that job).\n- src/scheduler/src/repl_trace.c `parse_sel()`: uses `repl_parse_jobsel_arg()`. A raw id comes back as the same `JOBSEL_BARE` selector its dashed form produces, so the existing bare-id rule applies unchanged: the exact selector, plus the whole-array selector when the id is array-aligned (Implementation Notes 4.3). repl_trace.c is 652 lines.\n- src/scheduler/testsuite/test_repl_trace_cmd.c: new `test_on_job_raw_decimal_ids`; the old \"`7` is rejected\" case now uses `7x` (a raw `7` is a valid id now), and an over-wide decimal is rejected with the width error.\n\nVerification:\n- Test: `test_on_job_raw_decimal_ids` in src/scheduler/testsuite/test_repl_trace_cmd.c (raw plain id, raw array-aligned id and `104` each give the same selectors as their dashed form; mixed raw/dashed/range line) and the over-wide case in `test_invalid_arguments_leave_settings`: RED before the fix (`test_on_job_raw_decimal_ids: FAIL ... unexpected output for 'trace on job 81064794341244970': [sim] trace: invalid job id ...`; the width case got the syntax error instead), GREEN after (14 Tests 0 Failures, leak detection on).\n- Suites: `bash src/tools/xce-build.sh scheduler --jobs 12 --incremental`: scheduler suite 118/118 OK, asan (log: /home/claude/tmp/bug-1930/gate-scheduler-suite.log).\n- Live: Reproduction step 2 after the fix: `[sim] trace: enabled; applies from the next session`, then `[trace] job 0-0-0-104 ...` blocks for sessions 1-5 under `continue`, and `info trace 104` shows the stored trace (live-green.txt).",
|
||||
"tags": [
|
||||
"repl",
|
||||
"sim",
|
||||
"trace",
|
||||
"jobid"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1926"
|
||||
],
|
||||
"fix_commits": [
|
||||
"ba6f42cd"
|
||||
],
|
||||
"record": ".wolf/bugs/bug-1930.md"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
77
.wolf/bugs/bug-1928.md
Normal file
77
.wolf/bugs/bug-1928.md
Normal file
|
|
@ -0,0 +1,77 @@
|
|||
---
|
||||
id: "bug-1928"
|
||||
status: "fixed"
|
||||
status_updated: "2026-10-09"
|
||||
status_note: ""
|
||||
component: "scheduler"
|
||||
severity: "leak"
|
||||
timestamp: "2026-10-09T03:14:08Z"
|
||||
last_seen: "2026-10-09T03:14:08Z"
|
||||
occurrences: 1
|
||||
files: ["src/scheduler/src/repl.c", "src/scheduler/src/repl_info.c", "src/scheduler/src/repl_rollup.c", "src/scheduler/testsuite/meson.build"]
|
||||
tags: ["repl", "sim", "leak", "iterator", "asan"]
|
||||
related: ["bug-1884"]
|
||||
fix_commits: ["ba6f42cd"]
|
||||
---
|
||||
|
||||
# bug-1928: xce_scheduler --sim REPL leaks a job-table iterator every continue round (LeakSanitizer at exit)
|
||||
|
||||
## Description
|
||||
**Observed:** a normal interactive simulator session (`continue`, then `quit`) on the ASAN build exits 1 from LeakSanitizer, and the buffered console output is lost (0 bytes on stdout):
|
||||
```
|
||||
==4149978==ERROR: LeakSanitizer: detected memory leaks
|
||||
|
||||
Direct leak of 144 byte(s) in 9 object(s) allocated from:
|
||||
#0 0x79e7779215a3 in calloc (.../libasan.so.8+0x1215a3)
|
||||
#1 0x5907bfd480ae in xkvmgr_make_iter ../lib/kvmgr.c:159
|
||||
#2 0x5907bfde1eca in iter_make (.../xce_scheduler+0x652eca)
|
||||
#3 0x5907bfcf1004 in repl_count_jobs ../scheduler/src/repl.c:189
|
||||
#4 0x5907bfcf5464 in repl_query_pend_jobids ../scheduler/src/repl_rollup.c:94
|
||||
#5 0x5907bfcf64bc in repl_round_rollup_process ../scheduler/src/repl_rollup.c:295
|
||||
#6 0x5907bfcf3238 in xce_repl_post_session_check ../scheduler/src/repl.c:626
|
||||
...
|
||||
SUMMARY: AddressSanitizer: 160 byte(s) leaked in 10 allocation(s).
|
||||
```
|
||||
**Expected:** the session exits 0 with leak detection on, and its console output reaches stdout.
|
||||
**Impact:** leak of 16 bytes (one kvmgr iterator) per call of `repl_count_jobs()`: once at the load banner and once per completed round in `continue` mode (Round Rollup), so it grows linearly with the replay length; `info jobs` leaks one more per call. Small, but it makes every standalone ASAN REPL run fail with exit 1 and drop its stdout, and forced several REPL tests to run with `detect_leaks=0` (the trace wiring test's meson comment named this leak as the reason).
|
||||
**Found by:** ASAN run of the simulator during scheduler-sim-decision-trace validation; the second site (`repl_info_jobs()`) by audit. A grep of every `iter_make()` in src/scheduler/src for a matching `iter_destroy()` on every path found no others.
|
||||
|
||||
## Environment
|
||||
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
||||
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
||||
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
||||
- Build: asan (xce.build), default ASAN_OPTIONS (leak detection on) for the standalone run; the affected meson tests ran with `ASAN_OPTIONS=detect_leaks=0`
|
||||
- Code: scheduler-framework @ 5ca59dc9-dirty
|
||||
- Deployment: standalone simulator only (`xce_scheduler --sim ... --interactive`), no controller, no workers; single dev host
|
||||
|
||||
## Reproduction
|
||||
1. Build: `bash src/tools/xce-build.sh scheduler --jobs 12 --incremental` at the Environment commit.
|
||||
2. In a temp dir: `printf 'continue\nquit\n' | TERM=dumb src/xce.build/scheduler/xce_scheduler --sim src/scheduler/testsuite/fixtures/repl_trace_baseline/contention.replay --interactive --logfile <tmp>/sched.log` (paths relative to the repo root) -- exit status 1, LeakSanitizer report on stderr, stdout empty.
|
||||
3. Smaller: `src/xce.build/scheduler/testsuite/sched_repl_trace_baseline_test` with leak detection on (the child inherits it) fails `test_transcript_matches_baseline: Expected 0 Was 1. the scripted session must end with quit's exit(0)`; `sched_repl_trace_wiring_test` with leak detection on exits 1 with 5 `repl_count_jobs` frames.
|
||||
- Frequency: deterministic (1/1 standalone runs, every leak-on test run, 2026-10-09)
|
||||
- Evidence: /home/claude/tmp/bug-1928/red/ (stderr.txt, stdout.txt, per-test logs of a leak-on sweep of every sched_repl_*/sched_sim_* test before the fix), /home/claude/tmp/bug-1928/green/ (same after the fix)
|
||||
|
||||
## Root cause
|
||||
`repl_count_jobs()` (src/scheduler/src/repl.c:189) creates an iterator over `core->jobs` with `iter_make()`, walks it, and returns the count without calling `iter_destroy()`. For a kvmgr container `iter_make()` heap-allocates the inner iterator (`xkvmgr_make_iter`, lib/kvmgr.c:159) and only `xkvmgr_iter_destroy()` frees it, so each call leaks 16 bytes. It is called from `repl_print_banner()` once and from `repl_query_pend_jobids()` (repl_rollup.c) on every completed round in `continue` mode, hence the per-round growth. `repl_info_jobs()` (src/scheduler/src/repl_info.c:172, `info jobs`) has the same missing `iter_destroy()`. The leak makes LeakSanitizer turn the exit status into 1 at exit, and the process ends before stdio flushes the redirected stdout.
|
||||
|
||||
Since: `repl_count_jobs()` since 02f7f48f (scheduler-core REPL task 12.1); `repl_info_jobs()` since c705cada. repl_rollup.c's own comment acknowledged the missing destroy as a "low-impact quirk", but its own per-round caller made it a growing leak.
|
||||
|
||||
Scope: every `iter_make()` in src/scheduler/src (core.c, core_session.c, plugin.c, plugins/limit.c, host_iter.c, scheduler.c, snapshot.c, sim_loop.c, sim_picker.c, repl_info.c, repl_rollup.c, repl_trace_store.c) was checked; all others destroy the iterator on every path, and their early returns are only on `iter_make()` failure, where kvmgr frees the iterator itself. `host_iter.c` keeps its iterator in the host-iterator private data and destroys it in its fini.
|
||||
|
||||
## Fix
|
||||
- src/scheduler/src/repl.c `repl_count_jobs()`: call `iter_destroy(&it)` after the walk.
|
||||
- src/scheduler/src/repl_info.c `repl_info_jobs()`: same.
|
||||
- src/scheduler/src/repl_rollup.c: replaced the stale comment that called the missing destroy a quirk.
|
||||
- src/scheduler/testsuite/meson.build: `scheduler repl trace baseline test` and `scheduler repl trace wiring test` now run with leak detection on (removed `ASAN_OPTIONS=detect_leaks=0` and the comment blaming this leak). The other REPL/sim tests still leak from their own fixtures (cores and replay writers never torn down), so they keep `detect_leaks=0`.
|
||||
|
||||
Every iterator the REPL makes is now destroyed, so nothing is left for LeakSanitizer at exit.
|
||||
|
||||
## Verification
|
||||
- Test: `scheduler repl trace baseline test` (test_repl_trace_baseline.c, leak detection now on): RED before the fix (`test_transcript_matches_baseline: Expected 0 Was 1`, the child exits 1 from LeakSanitizer), GREEN after (4 Tests 0 Failures). `scheduler repl trace wiring test` (test_repl_trace_wiring.c, leak detection now on): RED before (exit 1, LeakSanitizer with `repl_count_jobs` frames), GREEN after (17 Tests 0 Failures).
|
||||
- Leak-on sweep of all 42 sched_repl_*/sched_sim_* test binaries: `repl_count_jobs`/`repl_info_jobs` frames 0 in every log after the fix (before: present in 13).
|
||||
- Suites: `bash src/tools/xce-build.sh scheduler --jobs 12 --incremental`: scheduler suite 118/118 OK, asan (log: /home/claude/tmp/bug-1928/gate-scheduler-suite.log).
|
||||
- Live: the standalone command in Reproduction step 2 now exits 0 with leak detection on and prints its 18-line transcript (/home/claude/tmp/bug-1928/green/).
|
||||
|
||||
## References
|
||||
- Related: bug-1884 (earlier standalone `--sim` LeakSanitizer exit-1 fix; this is a REPL-only leak it did not cover)
|
||||
- Docs: src/scheduler/CLAUDE.md (Interactive REPL)
|
||||
73
.wolf/bugs/bug-1929.md
Normal file
73
.wolf/bugs/bug-1929.md
Normal file
|
|
@ -0,0 +1,73 @@
|
|||
---
|
||||
id: "bug-1929"
|
||||
status: "fixed"
|
||||
status_updated: "2026-10-09"
|
||||
status_note: ""
|
||||
component: "scheduler"
|
||||
severity: "wrong-result"
|
||||
timestamp: "2026-10-09T03:21:26Z"
|
||||
last_seen: "2026-10-09T03:21:26Z"
|
||||
occurrences: 1
|
||||
files: ["src/scheduler/src/repl_rollup.c", "src/scheduler/src/repl_break.c", "src/scheduler/src/repl_internal.h", "src/scheduler/testsuite/test_repl_round_rollup.c", "src/scheduler/testsuite/test_repl_trace_emit.c", "src/scheduler/testsuite/test_repl_trace_wiring.c", "src/scheduler/testsuite/fixtures/repl_trace_baseline/transcript.txt"]
|
||||
tags: ["repl", "sim", "rollup", "continue", "output"]
|
||||
related: ["bug-1610", "bug-1611"]
|
||||
fix_commits: ["ba6f42cd"]
|
||||
---
|
||||
|
||||
# bug-1929: xce_scheduler --sim REPL continue round summary prints empty jobids= for a dispatch-only round
|
||||
|
||||
## Description
|
||||
**Observed:** in `continue` mode a round whose only decision is a dispatch prints a round summary with an empty job list, with tracing off as well as on. From the 6.3 trace-equivalence harness (`XCE_TRACE_EQUIV_DUMP=1`, extra replay, 8 occurrences):
|
||||
```
|
||||
[sim] round 1-1 (1 round(s), t=1970-01-01T00:00:00Z..1970-01-01T00:00:00Z): 4 dispatch, 1 pend, 0 preempt -- jobids=0-0-0-202
|
||||
[sim] round 2-2 (1 round(s), t=1970-01-01T00:00:00Z..1970-01-01T00:00:00Z): 1 dispatch, 0 pend, 0 preempt -- jobids=
|
||||
```
|
||||
With the 1.1 contention fixture the dispatched job is missing even when a pend job is listed (round 8 dispatched 0-0-0-108):
|
||||
```
|
||||
[sim] round 8-8 (1 round(s), t=1970-01-01T00:00:00Z..1970-01-01T00:00:00Z): 1 dispatch, 1 pend, 0 preempt -- jobids=0-0-0-107
|
||||
```
|
||||
**Expected:** "jobids=" names the jobs the round's decisions affected (scheduler-sim-repl-verbosity Requirement 2.1: "the count of each decision type and the identifiers of the jobs affected"), so the dispatched (and preempted) jobs are listed: `jobids=0-0-0-202` (the job pending in round 1 and dispatched in round 2) and `jobids=0-0-0-107,0-0-0-108`.
|
||||
**Impact:** wrong/misleading REPL output only: the operator cannot see which job a round dispatched or preempted. A second effect of the same cause: rounds with the same counts and pend set but different dispatched jobs collapse into one range (Requirement 3.2 says a range needs the same affected job identifiers). Scheduling decisions are not affected.
|
||||
**Found by:** scheduler-sim-decision-trace validation (rollup output in the 6.1/6.3 e2e and equivalence harness dumps).
|
||||
|
||||
## Environment
|
||||
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
||||
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
||||
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
||||
- Build: asan (xce.build), tests run with `ASAN_OPTIONS=detect_leaks=0` (suite convention for these binaries); the symptom does not depend on sanitizer options or on tracing
|
||||
- Code: scheduler-framework @ 5ca59dc9-dirty
|
||||
- Deployment: standalone simulator only (`xce_scheduler --sim ... --interactive`), no controller, no workers; single dev host
|
||||
|
||||
## Reproduction
|
||||
1. Build: `bash src/tools/xce-build.sh scheduler --jobs 12 --incremental` at the Environment commit.
|
||||
2. `cd src/xce.build/scheduler/testsuite && XCE_TRACE_EQUIV_DUMP=1 ASAN_OPTIONS=detect_leaks=0 ./sched_trace_equivalence_test 2>&1 | grep 'jobids=$'` -- 8 lines `... 1 dispatch, 0 pend, 0 preempt -- jobids=`.
|
||||
3. Contention fixture: `printf 'continue\nquit\n' | TERM=dumb src/xce.build/scheduler/xce_scheduler --sim src/scheduler/testsuite/fixtures/repl_trace_baseline/contention.replay --interactive --logfile <tmp>/sched.log` (repo root; with bug-1928 unfixed add `ASAN_OPTIONS=detect_leaks=0`) -- round 8-8 lists only 0-0-0-107; rounds 1-1 and 5-5 list only the pend set.
|
||||
4. Smallest: `test_round_summary_lists_dispatched_jobids_golden` in test_repl_round_rollup.c (drives xce_repl_on_begin_round/xce_repl_on_decision/repl_round_rollup_process directly).
|
||||
- Frequency: deterministic (every run, 2026-10-09)
|
||||
- Evidence: /home/claude/tmp/bug-1929/ -- equiv-dump-red.txt / equiv-dump-green.txt, e2e-dump-red.txt, contention-continue-red.txt / contention-continue-green.txt, rollup-red.txt, transcript.diff, fixture-backup/ (the baseline fixtures before regeneration)
|
||||
|
||||
## Root cause
|
||||
`repl_round_emit()` (src/scheduler/src/repl_rollup.c:178, listing built at :201-202 before the fix) built the "jobids=" listing from `pend_jobids` only, and `repl_round_rollup_process()` (repl_rollup.c:285) only ever collected the pend set (`repl_query_pend_jobids()`, a walk of core->jobs for JOB_IS_PEND jobs). The dispatch and preempt decisions were counted in `xce_repl_on_decision()` (src/scheduler/src/repl_break.c:953-959, `last_n_dispatch`/`last_n_preempt`) but their job ids were never recorded anywhere. A dispatched job is no longer pending after the round, so a dispatch-only round has an empty pend set and printed `jobids=` with nothing after it; in mixed rounds the dispatched jobs were silently left out. The same pend-only fingerprint made rounds that dispatched different jobs compare equal and collapse.
|
||||
|
||||
Since: 921c72bb (scheduler-sim-repl-verbosity task 2.2, Round Rollup), written while bug-1611 meant the simulator produced no DISPATCH decisions, so the pend set was the only affected-job information then.
|
||||
|
||||
Scope: the traced-round path (scheduler-sim-decision-trace 7.3/7.4) and the allocation-failure fallback go through the same `repl_round_emit()`; step mode prints per-decision lines and was not affected.
|
||||
|
||||
## Fix
|
||||
- src/scheduler/src/repl_internal.h: new state in `xce_sched_repl_state_s` -- `round_decided_jobids`/`round_n_decided`/`round_cap_decided`/`round_decided_lost` (this round's dispatch/preempt jobids) and `collapse_decided_jobids`/`collapse_n_decided_jobids` (the active range's); declarations of `repl_round_note_decided()`/`repl_round_reset_decided()`.
|
||||
- src/scheduler/src/repl_break.c: `xce_repl_on_decision()` records the jobid of each DISPATCH and PREEMPT decision (a decision with no body, jobid 0, is skipped); `xce_repl_on_begin_round()` resets the per-round list.
|
||||
- src/scheduler/src/repl_rollup.c: `repl_round_rollup_process()` takes over the round's list (sorted, de-duplicated) and makes it part of the fingerprint next to the counts and the pend set, so rounds collapse only when the affected jobs match; `repl_round_emit()` lists the union of the pend set and the dispatch/preempt set (`repl_fmt_affected()`, through the existing `repl_fmt_jobid_range()` compressor) and takes the pend count separately; `repl_round_collapse_flush()` frees the new list. If the per-round list cannot grow (out of memory) or the pend set cannot be built, the round is reported on its own with `jobids=(unavailable)` instead of an empty or partial list.
|
||||
- Tests: test_repl_trace_emit.c and test_repl_trace_wiring.c teardowns free the new lists (both run with leak detection on).
|
||||
- src/scheduler/testsuite/fixtures/repl_trace_baseline/transcript.txt regenerated (XCE_REPL_BASELINE_REGEN=1, .replay fixtures restored from the backup): exactly two lines change, both `jobids=` lines -- round 5-5 `jobids=0-0-0-107` -> `jobids=0-0-0-104,0-0-0-105,0-0-0-106,0-0-0-107` and round 8-8 `jobids=0-0-0-107` -> `jobids=0-0-0-107,0-0-0-108`.
|
||||
|
||||
Output format change: "jobids=" now means every job the range's decisions affected (Requirement 2.1), not only the jobs left pending; the counts are unchanged.
|
||||
|
||||
## Verification
|
||||
- Test: `test_round_summary_lists_dispatched_jobids_golden` in src/scheduler/testsuite/test_repl_round_rollup.c (exact golden lines for two dispatch-only rounds of different jobs, and a dispatch+preempt round): RED before the fix -- `Expected '[sim] round 1-1 ...: 1 dispatch, 0 pend, 0 preempt -- jobids=0-0-0-5101\n[sim] round 2-2 ...: 1 dispatch, 0 pend, 0 preempt -- jobids=0-0-0-5102\n' Was '[sim] round 1-2 (2 round(s), ...): 1 dispatch, 0 pend, 0 preempt -- jobids=\n'` -- GREEN after (8 Tests 0 Failures). `scheduler repl trace baseline test` with the regenerated transcript: GREEN (4 Tests 0 Failures, leak detection on).
|
||||
- Harness: equivalence dump (`XCE_TRACE_EQUIV_DUMP=1`) has 8 empty `jobids=` lines before and 0 after; all 8 dispatch-only rounds now read `round 2-2 ... 1 dispatch, 0 pend, 0 preempt -- jobids=0-0-0-202` (equiv-dump-green.txt).
|
||||
- Suites: `bash src/tools/xce-build.sh scheduler --jobs 12 --incremental`: scheduler suite 118/118 OK, asan (log: /home/claude/tmp/bug-1929/gate-scheduler-suite.log).
|
||||
- Live: standalone contention run (Reproduction step 3) after the fix: round 8-8 `jobids=0-0-0-107,0-0-0-108`, round 1-1 lists 101-107 (contention-continue-green.txt).
|
||||
|
||||
## References
|
||||
- Related: bug-1611 (no DISPATCH decisions in sim at the time the rollup was written), bug-1610 (rollup introduced to replace per-decision lines)
|
||||
- Docs: .kiro/specs/scheduler-sim-repl-verbosity/requirements.md (Requirement 2.1, 3.2); src/scheduler/testsuite/fixtures/repl_trace_baseline/transcript.txt
|
||||
66
.wolf/bugs/bug-1930.md
Normal file
66
.wolf/bugs/bug-1930.md
Normal file
|
|
@ -0,0 +1,66 @@
|
|||
---
|
||||
id: "bug-1930"
|
||||
status: "fixed"
|
||||
status_updated: "2026-10-09"
|
||||
status_note: ""
|
||||
component: "scheduler"
|
||||
severity: "wrong-result"
|
||||
timestamp: "2026-10-09T03:28:14Z"
|
||||
last_seen: "2026-10-09T03:28:14Z"
|
||||
occurrences: 1
|
||||
files: ["src/scheduler/src/repl_trace.c", "src/scheduler/src/repl_trace_store.c", "src/scheduler/src/repl_internal.h", "src/scheduler/testsuite/test_repl_trace_cmd.c"]
|
||||
tags: ["repl", "sim", "trace", "jobid"]
|
||||
related: ["bug-1926"]
|
||||
fix_commits: ["ba6f42cd"]
|
||||
---
|
||||
|
||||
# bug-1930: xce_scheduler --sim REPL trace on job rejects the raw decimal job id info trace accepts
|
||||
|
||||
## Description
|
||||
**Observed:** `trace on job` rejects a raw packed decimal job id, the form `info trace` and `info job` accept for the same job:
|
||||
```
|
||||
[sim] trace: invalid job id: 104 (invalid job identity: expected <cluster>-<apiserver>-<timestamp>-<sequence> optionally followed by [k] or [a-b] (decimal digits only, no sign or whitespace))
|
||||
```
|
||||
while `info trace 104` and `info job 104` both resolve job 0-0-0-104. In the unit test the same rejection for a full packed id:
|
||||
```
|
||||
unexpected output for 'trace on job 81064794341244970': [sim] trace: invalid job id: 81064794341244970 (invalid job identity: expected <cluster>-<apiserver>-<timestamp>-<sequence> optionally followed by [k] or [a-b] (decimal digits only, no sign or whitespace))
|
||||
```
|
||||
**Expected:** `trace on job` accepts the same forms as `info trace`/`info job` -- the dashed id, `X[k]`, `X[a-b]` and a raw packed decimal id -- and a raw decimal id selects exactly what its dashed bare form selects (for an array-aligned id, the exact job plus the whole array, tasks.md Implementation Notes 4.3).
|
||||
**Impact:** inconsistent REPL grammar: an operator who copies a raw id (as `info job`/`info trace` accept it) into `trace on job` gets an error and no trace. No scheduling effect.
|
||||
**Found by:** user report during scheduler-sim-decision-trace validation.
|
||||
|
||||
## Environment
|
||||
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
||||
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
||||
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
||||
- Build: asan (xce.build), default ASAN_OPTIONS; the symptom does not depend on sanitizer options
|
||||
- Code: scheduler-framework @ 5ca59dc9-dirty
|
||||
- Deployment: standalone simulator only (`xce_scheduler --sim ... --interactive`), no controller, no workers; single dev host
|
||||
|
||||
## Reproduction
|
||||
1. Build: `bash src/tools/xce-build.sh scheduler --jobs 12 --incremental` at the Environment commit.
|
||||
2. `printf 'trace on job 104\ninfo trace 104\nquit\n' | TERM=dumb ASAN_OPTIONS=detect_leaks=0 src/xce.build/scheduler/xce_scheduler --sim src/scheduler/testsuite/fixtures/repl_trace_baseline/contention.replay --interactive --logfile <tmp>/sched.log` (repo root) -- `trace on job 104` prints the "invalid job id" error above; `info trace 104` accepts the id.
|
||||
3. Smaller: `test_on_job_raw_decimal_ids` in src/scheduler/testsuite/test_repl_trace_cmd.c.
|
||||
- Frequency: deterministic (every run, 2026-10-09)
|
||||
- Evidence: /home/claude/tmp/bug-1930/ -- trace-cmd-red.txt (unit test before the fix), trace-cmd-green.txt, live-green.txt (`trace on job 104` after the fix, tracing 0-0-0-104 through `continue` and `info trace 104`)
|
||||
|
||||
## Root cause
|
||||
`parse_sel()` in src/scheduler/src/repl_trace.c:210 (before the fix) passes each JOBSEL word straight to libxce's `parse_jobsel()`, whose grammar is only the dashed `cluster-apiserver-timestamp-sequence` id with an optional `[k]`/`[a-b]` (utils.h), and reports any error. `info trace`'s `info_trace_sel()` (src/scheduler/src/repl_trace_store.c:456) adds a fallback after a failed `parse_jobsel()`: an all-digits argument is taken as a raw packed id (`strtoull`), matching `info job`. The two selector parsers were written separately and only `info trace` got the fallback.
|
||||
|
||||
Since: aa4c43c8 (scheduler-sim-decision-trace task 4.3, the `trace` command parser).
|
||||
|
||||
Scope: `break ... jobid=` has the opposite problem (it takes only the raw form) and is tracked separately as bug-1926; not changed here.
|
||||
|
||||
## Fix
|
||||
- src/scheduler/src/repl_trace_store.c: new `repl_parse_jobsel_arg(arg, out, raw)` (declared in repl_internal.h): `parse_jobsel()` plus the raw packed decimal fallback, returned as `JOBSEL_BARE` with `key` = the number and `*raw` = true; a decimal too wide for 64 bits is `JOBSEL_EWIDTH` (before, `info trace` passed it to `strtoull`, which saturated to UINT64_MAX). `info_trace_sel()` uses it and keeps its existing behaviour (a raw id names that job).
|
||||
- src/scheduler/src/repl_trace.c `parse_sel()`: uses `repl_parse_jobsel_arg()`. A raw id comes back as the same `JOBSEL_BARE` selector its dashed form produces, so the existing bare-id rule applies unchanged: the exact selector, plus the whole-array selector when the id is array-aligned (Implementation Notes 4.3). repl_trace.c is 652 lines.
|
||||
- src/scheduler/testsuite/test_repl_trace_cmd.c: new `test_on_job_raw_decimal_ids`; the old "`7` is rejected" case now uses `7x` (a raw `7` is a valid id now), and an over-wide decimal is rejected with the width error.
|
||||
|
||||
## Verification
|
||||
- Test: `test_on_job_raw_decimal_ids` in src/scheduler/testsuite/test_repl_trace_cmd.c (raw plain id, raw array-aligned id and `104` each give the same selectors as their dashed form; mixed raw/dashed/range line) and the over-wide case in `test_invalid_arguments_leave_settings`: RED before the fix (`test_on_job_raw_decimal_ids: FAIL ... unexpected output for 'trace on job 81064794341244970': [sim] trace: invalid job id ...`; the width case got the syntax error instead), GREEN after (14 Tests 0 Failures, leak detection on).
|
||||
- Suites: `bash src/tools/xce-build.sh scheduler --jobs 12 --incremental`: scheduler suite 118/118 OK, asan (log: /home/claude/tmp/bug-1930/gate-scheduler-suite.log).
|
||||
- Live: Reproduction step 2 after the fix: `[sim] trace: enabled; applies from the next session`, then `[trace] job 0-0-0-104 ...` blocks for sessions 1-5 under `continue`, and `info trace 104` shows the stored trace (live-green.txt).
|
||||
|
||||
## References
|
||||
- Related: bug-1926 (`break ... jobid=` accepts only the raw form; separate fix)
|
||||
- Docs: .kiro/specs/scheduler-sim-decision-trace/tasks.md (Implementation Notes 4.3), requirements.md 1.2-1.3
|
||||
Loading…
Add table
Add a link
Reference in a new issue