From 7b001cb0f7c2b84012ff9a468edb483e575336aa Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Tue, 29 Sep 2026 10:35:43 +0000 Subject: [PATCH] chore(buglog): bug-1885..1889 fixed Co-Authored-By: Claude Opus 5.5 (1M context) --- .wolf/buglog.json | 89 +++++++++++++++++++++++++++++++++++------- .wolf/bugs/bug-1885.md | 15 +++---- .wolf/bugs/bug-1886.md | 15 +++---- .wolf/bugs/bug-1887.md | 15 +++---- .wolf/bugs/bug-1888.md | 59 ++++++++++++++++++++++++++++ .wolf/bugs/bug-1889.md | 57 +++++++++++++++++++++++++++ 6 files changed, 214 insertions(+), 36 deletions(-) create mode 100644 .wolf/bugs/bug-1888.md create mode 100644 .wolf/bugs/bug-1889.md diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 55d888ee..0c31e457 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -16310,9 +16310,9 @@ }, { "id": "bug-1885", - "status": "open", + "status": "fixed", "status_updated": "2026-09-29", - "status_note": "newly logged", + "status_note": "", "component": "scheduler", "severity": "leak", "timestamp": "2026-09-29T09:48:44Z", @@ -16320,8 +16320,8 @@ "occurrences": 1, "error_message": "scheduler normal mode never frees decision structs: 64 bytes leaked per published decision\n\n**Observed:** (code reading) the normal-mode round loop in\n`xce_sched_driver_run_normal_mode_round()` (src/scheduler/src/driver.c)\ndrains each `xce_decision_t*` from `core->decisions`, records it and passes\nit to `on_decision`, but never frees the struct `make_*_decision()`\n(decision.c) calloc()s:\n```\ncbuf_get(&(sched->core.decisions), &decision);\n... sched->core.on_decision(sched->core.driver_ctx, decision);\n/* decision never freed */\n```\n\n**Expected:** the round loop frees the decision struct (not its alloc/pend reason, which the job owns) once published.\n**Impact:** leak of 64 bytes per dispatch/pend decision for the life of a normal-mode scheduler. Simulation mode was fixed in bug-1884 (ebfe4794). Three tests (test_core_full_cycle.c, test_core_session.c, test_snapshot_save_load.c) keep decision pointers from their callback, so the fix changes the callback contract (\"valid only during the call\") and those tests.\n**Found by:** review (while fixing bug-1884, 2026-09-29)", "file": "src/scheduler/src/driver.c, src/scheduler/src/decision.c", - "root_cause": "TODO: required for fixed/partial", - "fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why", + "root_cause": "`xce_sched_driver_run_normal_mode_round()` (driver.c) drains each decision from `core->decisions`, records it and passes it to `on_decision`, and never frees the struct `make_*_decision()` (decision.c) allocates.", + "fix": "The round frees the decision struct after the callback returns (d21b3d8f). Its allocation/pend reason belong to the job; the publish callback passes those pointers on and the recorder copies, so nothing keeps the struct. Tests with their own drain loops (test_core_full_cycle.c, test_core_session.c, test_snapshot_save_load.c) are unaffected.\n\nVerification:\n- Test: test_round_frees_decision_after_callback in src/scheduler/testsuite/test_driver_session_ordering.c (drives the real round with a stand-in allocator emitting one PEND; checks the struct is freed via __asan_address_is_poisoned and the reason is not): RED before, GREEN after\n- Suites: ASAN debug build (src/xce.build.discrete), `meson test --no-suite xceadmin`: 117/117 OK, 0 sanitizer reports; xcelib_test 695 tests 0 failures (2026-09-29)\n- Live: 3-round sequential HA soak on rebuilt ASAN binaries (/home/claude/tmp/xce-bug1885-soak, 2026-09-29): PASS, 15 jobs, zero job loss, all 3 nodes led; SIMVERIFY 4 files 0 mismatches; each recording replays with detect_leaks=1 at exit 0, 0 mismatches; no sanitizer report in the scheduler or controller stderr", "tags": [ "scheduler", "leak", @@ -16330,14 +16330,16 @@ "related_bugs": [ "bug-1884" ], - "fix_commits": [], + "fix_commits": [ + "d21b3d8f" + ], "record": ".wolf/bugs/bug-1885.md" }, { "id": "bug-1886", - "status": "open", + "status": "fixed", "status_updated": "2026-09-29", - "status_note": "newly logged", + "status_note": "", "component": "scheduler", "severity": "leak", "timestamp": "2026-09-29T09:48:47Z", @@ -16345,8 +16347,8 @@ "occurrences": 1, "error_message": "scheduler leaks the rejected queue on each config re-apply and the old config on each CONFIG_PUSH\n\n**Observed:** (code reading)\n- `sched_init_queues()` (src/scheduler/src/scheduler.c) builds a new\n `xce_sched_queue_t` for every configured queue on each config apply and\n `sched_add_queue()`s it; for an existing name the add fails and the new\n queue (with its interned name/resreq) is still handed to the picker but\n owned by nothing.\n- `CONFIG_PUSH` (core_events.c) sets `core->config = config` without freeing\n the previous config.\n\n**Expected:** re-applied queues update the existing queue in place (as `core_upsert_queue()` does for QUEUE_UPDATE), and a replaced config is freed once nothing points into it.\n**Impact:** leak per config push/re-apply (one queue per configured queue plus one whole `xce_conf4sched_t`). Freeing the old config needs care: the picker and limit plugin may keep pointers into it. The queue table's datakiller must stay a no-op for as long as `sched_init_queues()` keeps using a rejected queue (bug-1884 found a use-after-free when it freed).\n**Found by:** review (while fixing bug-1884, 2026-09-29)", "file": "src/scheduler/src/scheduler.c, src/scheduler/src/core_events.c", - "root_cause": "TODO: required for fixed/partial", - "fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why", + "root_cause": "`sched_init_queues()` built a new `xce_sched_queue_t` for every configured queue on each config apply; for an existing name `sched_add_queue()` failed and the unregistered queue was handed to the picker (which then tracked it instead of the registered one) and leaked, while the registered queue kept its old settings. `sched_init_config()` and the core's CONFIG_PUSH replaced `core->config` without freeing the previous one.", + "fix": "`sched_init_queues()` updates the registered queue in place (attrs, status, priority, accept_interval, sched_delay -- what make_sched_queue() copies), as QUEUE_UPDATE's core_upsert_queue() does; a replaced config is freed after the new one is applied -- nothing points into it (queues, resources, groups and limits built from it are copies) (3e873f0a).\n\nVerification:\n- Test: test_config_reapply_updates_queue_in_place_and_frees_old in src/scheduler/testsuite/test_config_push_deferred.c (applies a one-queue config twice, priority 1 then 7): RED before (priority stayed 1), GREEN after (same queue object, priority 7, first config freed per ASAN)\n- Suites: ASAN debug build (src/xce.build.discrete), `meson test --no-suite xceadmin`: 117/117 OK, 0 sanitizer reports; xcelib_test 695 tests 0 failures (2026-09-29)\n- Live: 3-round sequential HA soak on rebuilt ASAN binaries (/home/claude/tmp/xce-bug1885-soak, 2026-09-29): PASS, 15 jobs, zero job loss, all 3 nodes led; SIMVERIFY 4 files 0 mismatches; each recording replays with detect_leaks=1 at exit 0, 0 mismatches; no sanitizer report in the scheduler or controller stderr", "tags": [ "scheduler", "leak", @@ -16355,14 +16357,16 @@ "related_bugs": [ "bug-1884" ], - "fix_commits": [], + "fix_commits": [ + "3e873f0a" + ], "record": ".wolf/bugs/bug-1886.md" }, { "id": "bug-1887", - "status": "open", + "status": "fixed", "status_updated": "2026-09-29", - "status_note": "newly logged", + "status_note": "", "component": "lib", "severity": "wrong-result", "timestamp": "2026-09-29T09:48:49Z", @@ -16370,8 +16374,8 @@ "occurrences": 1, "error_message": "resreq parser ignores a primary's select/rusage parse error and accepts the job without that clause\n\n**Observed:** (code reading) the `primary_resreq` action in\nsrc/lib/xce_resreq.peg calls\n`parse_resreq_primary(primary, $0, auxil->resmgr, &(auxil->errmsg))`\nand ignores its return value; nothing sets `has_error`, so\n`parse_resreq()` returns success. A primary whose select or rusage clause\nfails to parse is kept without that clause (select NULL / rusages empty),\ni.e. the job is scheduled as if it had no such constraint. While fixing\nbug-1884 a LeakSanitizer report showed such an error message being\nproduced during a normal replay fixture (its job's resreq parsed \"OK\").\n\n**Expected:** a primary sub-parse error makes `parse_resreq()` fail with that message (the job is rejected/pended with a parse error), not silently dropped.\n**Impact:** wrong result: a job with a malformed select (e.g. a typo) may dispatch onto hosts its select would have excluded.\n**Found by:** review (while fixing bug-1884, 2026-09-29)", "file": "src/lib/xce_resreq.peg, src/lib/resreq.c", - "root_cause": "TODO: required for fixed/partial", - "fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why", + "root_cause": "The `primary_resreq` action in src/lib/xce_resreq.peg ignored `parse_resreq_primary()`'s result and nothing set `has_error`, so `parse_resreq()` succeeded for a primary whose select/rusage clause failed to parse and the job kept no such clause. Confirmed with a probe (2026-09-29): `select[type==]`, `select[mem>]`, `select[type==any && ]` all returned rc=1 with select NULL; `select[gpu>1]` with no gpu resource likewise.", + "fix": "A malformed clause now fails the whole resreq and `parse_resreq()` returns the clause's own message (b5712c05). A select naming an unregistered resource is still dropped rather than rejected: parse_select() reports it as -ENOENT (new select_context_t.unknown_resource) and the action lets it pass, because jobs rely on it -- the scheduler's own \"type == any\" default among them when no \"type\" resource is registered (making it fatal failed 36 scheduler suites). Whether an unknown resource in a select should reject the job is left as a separate decision. Making syntax errors fatal exposed bug-1888.\n\nVerification:\n- Test: test_resreq_primary_select_error_fails_parse in src/lib/testsuite/test_resreq.c (xkvmgr resmgr with type/mem; syntax errors incl. inside a compound fail with a message; unknown resource still parses with select NULL; valid select kept): RED before (all accepted), GREEN after\n- Suites: ASAN debug build (src/xce.build.discrete), `meson test --no-suite xceadmin`: 117/117 OK, 0 sanitizer reports; xcelib_test 695 tests 0 failures (2026-09-29)\n- Live: 3-round sequential HA soak on rebuilt ASAN binaries (/home/claude/tmp/xce-bug1885-soak, 2026-09-29): PASS, 15 jobs, zero job loss, all 3 nodes led; SIMVERIFY 4 files 0 mismatches; each recording replays with detect_leaks=1 at exit 0, 0 mismatches; no sanitizer report in the scheduler or controller stderr", "tags": [ "lib", "resreq", @@ -16380,8 +16384,63 @@ "related_bugs": [ "bug-1884" ], - "fix_commits": [], + "fix_commits": [ + "b5712c05" + ], "record": ".wolf/bugs/bug-1887.md" + }, + { + "id": "bug-1888", + "status": "fixed", + "status_updated": "2026-09-29", + "status_note": "", + "component": "scheduler", + "severity": "wrong-result", + "timestamp": "2026-09-29T10:32:13Z", + "last_seen": "2026-09-29T10:32:13Z", + "occurrences": 1, + "error_message": "snapshots save a job's resreq as a debug dump; restored jobs lose select and rusage\n\n**Observed:** saving a snapshot of a pending job whose resreq is `rusage[mem=2]` stores `submit->resreq` as\n```\nPrimary:\n Tasks: min = 1, max = 1\n Rusages:\n res mem value 2.000000 per 3 duration -1.000000 decay 0\n```\nand the loaded job has no rusage (and no select). With bug-1887 fixed the loader rejects that text outright (snapshot tests failed: \"parse_resreq failed for jobid 201: Primary: ...\").\n\n**Expected:** the snapshot stores the resreq text the job was submitted with, and the restored job parses to the same request.\n**Impact:** wrong result: every job restored from a snapshot -- a `.replay` start snapshot (simulator replays), REPL snapshots -- lost its select and rusage, so replays of sessions with pending jobs at snapshot time could decide differently from the live scheduler. A save/restore field-completeness violation (persistence.md).\n**Found by:** test (while fixing bug-1887, 2026-09-29)", + "file": "src/scheduler/src/snapshot_wire.c, src/scheduler/src/snapshot_load.c", + "root_cause": "`build_wire_job()` (snapshot_wire.c) filled `submit->resreq` with `format_resreq()`, a human-readable debug dump, because the scheduler job kept only the parsed resreq, not its text. The loader's parser accepted the dump as a primary with neither select nor rusage (the bug-1887 leniency).", + "fix": "`xce_sched_job_t` keeps the resreq text (`resreq_str`, set at ingest, JOB_UPDATE and snapshot load, freed at teardown) and the snapshot saves it; a job without it restores as the loader's \"type == any\" default (077b1213).\n\nVerification:\n- Test: test_snapshot_round_trip_keeps_job_resreq_text in src/scheduler/testsuite/test_snapshot_save_load.c (push a job with rusage[mem=2], save, load into a second core): RED before (saved text was the dump), GREEN after (text kept, restored rusage count 1)\n- Suites: ASAN debug build (src/xce.build.discrete), `meson test --no-suite xceadmin`: 117/117 OK, 0 sanitizer reports; xcelib_test 695 tests 0 failures (2026-09-29)\n- Live: 3-round sequential HA soak on rebuilt ASAN binaries (/home/claude/tmp/xce-bug1885-soak, 2026-09-29): PASS, 15 jobs, zero job loss, all 3 nodes led; SIMVERIFY 4 files 0 mismatches; each recording replays with detect_leaks=1 at exit 0, 0 mismatches; no sanitizer report in the scheduler or controller stderr", + "tags": [ + "scheduler", + "snapshot", + "persistence", + "resreq" + ], + "related_bugs": [ + "bug-1887" + ], + "fix_commits": [ + "077b1213" + ], + "record": ".wolf/bugs/bug-1888.md" + }, + { + "id": "bug-1889", + "status": "fixed", + "status_updated": "2026-09-29", + "status_note": "", + "component": "lib", + "severity": "wrong-result", + "timestamp": "2026-09-29T10:32:13Z", + "last_seen": "2026-09-29T10:32:13Z", + "occurrences": 1, + "error_message": "xcelib_test exits 0 on failures: nested UNITY_BEGIN in test_libxce_retry resets counts\n\n**Observed:** `xcelib_test` printed\n```\n../lib/testsuite/test.c:479:test_fmt_limit_units_percent_limit_ignores_units_mode:FAIL: a percentage-based limit's value must render identically regardless of units_mode ...\n56 Tests 0 Failures 0 Ignored\n```\nand exited 0, so meson reported the libxce suite OK on every run.\n\n**Expected:** a failing test fails xcelib_test and the suite.\n**Impact:** wrong result (test harness): any failure in a test group run before test_libxce_retry was silently dropped. One real failure was hidden: fmt_limit_units() printed a 0.8 percent limit as \"80.000001%\".\n**Found by:** test (reported by the user as a failing test, 2026-09-29)", + "file": "src/lib/testsuite/test_libxce_retry.c, src/lib/testsuite/test.c, src/lib/xcelimit.c", + "root_cause": "`test_libxce_retry()`, one group of test.c's single runner, called `UNITY_BEGIN()`/`UNITY_END()` itself; `UNITY_BEGIN()` resets Unity's test and failure counters, so failures from earlier groups were lost and the final `UNITY_END()` returned 0. The hidden failure: fmt_limit_units() widened the float limit value before multiplying by 100 (0.8f -> 0.800000011920929 -> \"80.000001%\").", + "fix": "Removed the nested UNITY_BEGIN/UNITY_END (the runner now reports 695 tests; this was the only hidden failure), and fmt_limit_units() scales in float then widens, giving the configured 80 (9c51de88). A scan of every testsuite found no other nested runner.\n\nVerification:\n- Test: test_fmt_limit_units_percent_limit_ignores_units_mode in src/lib/testsuite/test_limit.c: FAIL (now counted: 695 tests, 1 failure, exit 1) before the formatter fix, PASS after\n- Suites: ASAN debug build (src/xce.build.discrete), `meson test --no-suite xceadmin`: 117/117 OK, 0 sanitizer reports; xcelib_test 695 tests 0 failures (2026-09-29)\n- Live: 3-round sequential HA soak on rebuilt ASAN binaries (/home/claude/tmp/xce-bug1885-soak, 2026-09-29): PASS, 15 jobs, zero job loss, all 3 nodes led; SIMVERIFY 4 files 0 mismatches; each recording replays with detect_leaks=1 at exit 0, 0 mismatches; no sanitizer report in the scheduler or controller stderr", + "tags": [ + "lib", + "test", + "harness" + ], + "related_bugs": [], + "fix_commits": [ + "9c51de88" + ], + "record": ".wolf/bugs/bug-1889.md" } ] } diff --git a/.wolf/bugs/bug-1885.md b/.wolf/bugs/bug-1885.md index b8c32c9b..aeccd063 100644 --- a/.wolf/bugs/bug-1885.md +++ b/.wolf/bugs/bug-1885.md @@ -1,8 +1,8 @@ --- id: "bug-1885" -status: "open" +status: "fixed" status_updated: "2026-09-29" -status_note: "newly logged" +status_note: "" component: "scheduler" severity: "leak" timestamp: "2026-09-29T09:48:44Z" @@ -11,6 +11,7 @@ occurrences: 1 files: ["src/scheduler/src/driver.c", "src/scheduler/src/decision.c"] tags: ["scheduler", "leak", "decision"] related: ["bug-1884"] +fix_commits: ["d21b3d8f"] --- # bug-1885: scheduler normal mode never frees decision structs: 64 bytes leaked per published decision @@ -46,15 +47,15 @@ cbuf_get(&(sched->core.decisions), &decision); - Evidence: not kept; code reading ## Root cause -TODO: required for fixed/partial +`xce_sched_driver_run_normal_mode_round()` (driver.c) drains each decision from `core->decisions`, records it and passes it to `on_decision`, and never frees the struct `make_*_decision()` (decision.c) allocates. ## Fix -TODO: required for fixed/partial +The round frees the decision struct after the callback returns (d21b3d8f). Its allocation/pend reason belong to the job; the publish callback passes those pointers on and the recorder copies, so nothing keeps the struct. Tests with their own drain loops (test_core_full_cycle.c, test_core_session.c, test_snapshot_save_load.c) are unaffected. ## Verification -- Test: TODO: name + file; RED before the fix, GREEN after -- Suites: TODO: suite results and build profile -- Live: TODO: soak/deploy run, or n/a and why +- Test: test_round_frees_decision_after_callback in src/scheduler/testsuite/test_driver_session_ordering.c (drives the real round with a stand-in allocator emitting one PEND; checks the struct is freed via __asan_address_is_poisoned and the reason is not): RED before, GREEN after +- Suites: ASAN debug build (src/xce.build.discrete), `meson test --no-suite xceadmin`: 117/117 OK, 0 sanitizer reports; xcelib_test 695 tests 0 failures (2026-09-29) +- Live: 3-round sequential HA soak on rebuilt ASAN binaries (/home/claude/tmp/xce-bug1885-soak, 2026-09-29): PASS, 15 jobs, zero job loss, all 3 nodes led; SIMVERIFY 4 files 0 mismatches; each recording replays with detect_leaks=1 at exit 0, 0 mismatches; no sanitizer report in the scheduler or controller stderr ## References - Related: bug-1884 diff --git a/.wolf/bugs/bug-1886.md b/.wolf/bugs/bug-1886.md index efd6618f..24674132 100644 --- a/.wolf/bugs/bug-1886.md +++ b/.wolf/bugs/bug-1886.md @@ -1,8 +1,8 @@ --- id: "bug-1886" -status: "open" +status: "fixed" status_updated: "2026-09-29" -status_note: "newly logged" +status_note: "" component: "scheduler" severity: "leak" timestamp: "2026-09-29T09:48:47Z" @@ -11,6 +11,7 @@ occurrences: 1 files: ["src/scheduler/src/scheduler.c", "src/scheduler/src/core_events.c"] tags: ["scheduler", "leak", "config"] related: ["bug-1884"] +fix_commits: ["3e873f0a"] --- # bug-1886: scheduler leaks the rejected queue on each config re-apply and the old config on each CONFIG_PUSH @@ -44,15 +45,15 @@ related: ["bug-1884"] - Evidence: not kept; code reading plus the test_config_push_deferred UAF seen with a freeing killer ## Root cause -TODO: required for fixed/partial +`sched_init_queues()` built a new `xce_sched_queue_t` for every configured queue on each config apply; for an existing name `sched_add_queue()` failed and the unregistered queue was handed to the picker (which then tracked it instead of the registered one) and leaked, while the registered queue kept its old settings. `sched_init_config()` and the core's CONFIG_PUSH replaced `core->config` without freeing the previous one. ## Fix -TODO: required for fixed/partial +`sched_init_queues()` updates the registered queue in place (attrs, status, priority, accept_interval, sched_delay -- what make_sched_queue() copies), as QUEUE_UPDATE's core_upsert_queue() does; a replaced config is freed after the new one is applied -- nothing points into it (queues, resources, groups and limits built from it are copies) (3e873f0a). ## Verification -- Test: TODO: name + file; RED before the fix, GREEN after -- Suites: TODO: suite results and build profile -- Live: TODO: soak/deploy run, or n/a and why +- Test: test_config_reapply_updates_queue_in_place_and_frees_old in src/scheduler/testsuite/test_config_push_deferred.c (applies a one-queue config twice, priority 1 then 7): RED before (priority stayed 1), GREEN after (same queue object, priority 7, first config freed per ASAN) +- Suites: ASAN debug build (src/xce.build.discrete), `meson test --no-suite xceadmin`: 117/117 OK, 0 sanitizer reports; xcelib_test 695 tests 0 failures (2026-09-29) +- Live: 3-round sequential HA soak on rebuilt ASAN binaries (/home/claude/tmp/xce-bug1885-soak, 2026-09-29): PASS, 15 jobs, zero job loss, all 3 nodes led; SIMVERIFY 4 files 0 mismatches; each recording replays with detect_leaks=1 at exit 0, 0 mismatches; no sanitizer report in the scheduler or controller stderr ## References - Related: bug-1884 diff --git a/.wolf/bugs/bug-1887.md b/.wolf/bugs/bug-1887.md index e08a381b..5c7720df 100644 --- a/.wolf/bugs/bug-1887.md +++ b/.wolf/bugs/bug-1887.md @@ -1,8 +1,8 @@ --- id: "bug-1887" -status: "open" +status: "fixed" status_updated: "2026-09-29" -status_note: "newly logged" +status_note: "" component: "lib" severity: "wrong-result" timestamp: "2026-09-29T09:48:49Z" @@ -11,6 +11,7 @@ occurrences: 1 files: ["src/lib/xce_resreq.peg", "src/lib/resreq.c"] tags: ["lib", "resreq", "parser"] related: ["bug-1884"] +fix_commits: ["b5712c05"] --- # bug-1887: resreq parser ignores a primary's select/rusage parse error and accepts the job without that clause @@ -45,15 +46,15 @@ produced during a normal replay fixture (its job's resreq parsed "OK"). - Evidence: not kept; LeakSanitizer stack from the bug-1884 fixture (strdup in parse_select error path under core_build_job) ## Root cause -TODO: required for fixed/partial +The `primary_resreq` action in src/lib/xce_resreq.peg ignored `parse_resreq_primary()`'s result and nothing set `has_error`, so `parse_resreq()` succeeded for a primary whose select/rusage clause failed to parse and the job kept no such clause. Confirmed with a probe (2026-09-29): `select[type==]`, `select[mem>]`, `select[type==any && ]` all returned rc=1 with select NULL; `select[gpu>1]` with no gpu resource likewise. ## Fix -TODO: required for fixed/partial +A malformed clause now fails the whole resreq and `parse_resreq()` returns the clause's own message (b5712c05). A select naming an unregistered resource is still dropped rather than rejected: parse_select() reports it as -ENOENT (new select_context_t.unknown_resource) and the action lets it pass, because jobs rely on it -- the scheduler's own "type == any" default among them when no "type" resource is registered (making it fatal failed 36 scheduler suites). Whether an unknown resource in a select should reject the job is left as a separate decision. Making syntax errors fatal exposed bug-1888. ## Verification -- Test: TODO: name + file; RED before the fix, GREEN after -- Suites: TODO: suite results and build profile -- Live: TODO: soak/deploy run, or n/a and why +- Test: test_resreq_primary_select_error_fails_parse in src/lib/testsuite/test_resreq.c (xkvmgr resmgr with type/mem; syntax errors incl. inside a compound fail with a message; unknown resource still parses with select NULL; valid select kept): RED before (all accepted), GREEN after +- Suites: ASAN debug build (src/xce.build.discrete), `meson test --no-suite xceadmin`: 117/117 OK, 0 sanitizer reports; xcelib_test 695 tests 0 failures (2026-09-29) +- Live: 3-round sequential HA soak on rebuilt ASAN binaries (/home/claude/tmp/xce-bug1885-soak, 2026-09-29): PASS, 15 jobs, zero job loss, all 3 nodes led; SIMVERIFY 4 files 0 mismatches; each recording replays with detect_leaks=1 at exit 0, 0 mismatches; no sanitizer report in the scheduler or controller stderr ## References - Related: bug-1884 diff --git a/.wolf/bugs/bug-1888.md b/.wolf/bugs/bug-1888.md new file mode 100644 index 00000000..254b9524 --- /dev/null +++ b/.wolf/bugs/bug-1888.md @@ -0,0 +1,59 @@ +--- +id: "bug-1888" +status: "fixed" +status_updated: "2026-09-29" +status_note: "" +component: "scheduler" +severity: "wrong-result" +timestamp: "2026-09-29T10:32:13Z" +last_seen: "2026-09-29T10:32:13Z" +occurrences: 1 +files: ["src/scheduler/src/snapshot_wire.c", "src/scheduler/src/snapshot_load.c"] +tags: ["scheduler", "snapshot", "persistence", "resreq"] +related: ["bug-1887"] +fix_commits: ["077b1213"] +--- + +# bug-1888: snapshots save a job's resreq as a debug dump; restored jobs lose select and rusage + +## Description +**Observed:** saving a snapshot of a pending job whose resreq is `rusage[mem=2]` stores `submit->resreq` as +``` +Primary: + Tasks: min = 1, max = 1 + Rusages: + res mem value 2.000000 per 3 duration -1.000000 decay 0 +``` +and the loaded job has no rusage (and no select). With bug-1887 fixed the loader rejects that text outright (snapshot tests failed: "parse_resreq failed for jobid 201: Primary: ..."). + +**Expected:** the snapshot stores the resreq text the job was submitted with, and the restored job parses to the same request. +**Impact:** wrong result: every job restored from a snapshot -- a `.replay` start snapshot (simulator replays), REPL snapshots -- lost its select and rusage, so replays of sessions with pending jobs at snapshot time could decide differently from the live scheduler. A save/restore field-completeness violation (persistence.md). +**Found by:** test (while fixing bug-1887, 2026-09-29) + +## Environment +- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64 +- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2) +- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 +- Build: asan (src/xce.build.discrete) +- Code: scheduler-framework @ 9c51de88-dirty +- Deployment: n/a (unit test) + +## Reproduction +1. Build the scheduler tests; run `src/xce.build.discrete/scheduler/testsuite/sched_snapshot_save_load_test` at a commit before 077b1213 with test_snapshot_round_trip_keeps_job_resreq_text added. +2. Observe the saved text is the dump, not `rusage[mem=2]`. +- Frequency: deterministic +- Evidence: test output quoted above + +## Root cause +`build_wire_job()` (snapshot_wire.c) filled `submit->resreq` with `format_resreq()`, a human-readable debug dump, because the scheduler job kept only the parsed resreq, not its text. The loader's parser accepted the dump as a primary with neither select nor rusage (the bug-1887 leniency). + +## Fix +`xce_sched_job_t` keeps the resreq text (`resreq_str`, set at ingest, JOB_UPDATE and snapshot load, freed at teardown) and the snapshot saves it; a job without it restores as the loader's "type == any" default (077b1213). + +## Verification +- Test: test_snapshot_round_trip_keeps_job_resreq_text in src/scheduler/testsuite/test_snapshot_save_load.c (push a job with rusage[mem=2], save, load into a second core): RED before (saved text was the dump), GREEN after (text kept, restored rusage count 1) +- Suites: ASAN debug build (src/xce.build.discrete), `meson test --no-suite xceadmin`: 117/117 OK, 0 sanitizer reports; xcelib_test 695 tests 0 failures (2026-09-29) +- Live: 3-round sequential HA soak on rebuilt ASAN binaries (/home/claude/tmp/xce-bug1885-soak, 2026-09-29): PASS, 15 jobs, zero job loss, all 3 nodes led; SIMVERIFY 4 files 0 mismatches; each recording replays with detect_leaks=1 at exit 0, 0 mismatches; no sanitizer report in the scheduler or controller stderr + +## References +- Related: none diff --git a/.wolf/bugs/bug-1889.md b/.wolf/bugs/bug-1889.md new file mode 100644 index 00000000..d8e8490e --- /dev/null +++ b/.wolf/bugs/bug-1889.md @@ -0,0 +1,57 @@ +--- +id: "bug-1889" +status: "fixed" +status_updated: "2026-09-29" +status_note: "" +component: "lib" +severity: "wrong-result" +timestamp: "2026-09-29T10:32:13Z" +last_seen: "2026-09-29T10:32:13Z" +occurrences: 1 +files: ["src/lib/testsuite/test_libxce_retry.c", "src/lib/testsuite/test.c", "src/lib/xcelimit.c"] +tags: ["lib", "test", "harness"] +related: [] +fix_commits: ["9c51de88"] +--- + +# bug-1889: xcelib_test exits 0 on failures: nested UNITY_BEGIN in test_libxce_retry resets counts + +## Description +**Observed:** `xcelib_test` printed +``` +../lib/testsuite/test.c:479:test_fmt_limit_units_percent_limit_ignores_units_mode:FAIL: a percentage-based limit's value must render identically regardless of units_mode ... +56 Tests 0 Failures 0 Ignored +``` +and exited 0, so meson reported the libxce suite OK on every run. + +**Expected:** a failing test fails xcelib_test and the suite. +**Impact:** wrong result (test harness): any failure in a test group run before test_libxce_retry was silently dropped. One real failure was hidden: fmt_limit_units() printed a 0.8 percent limit as "80.000001%". +**Found by:** test (reported by the user as a failing test, 2026-09-29) + +## Environment +- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64 +- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2) +- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 +- Build: asan (src/xce.build.discrete) +- Code: scheduler-framework @ 9c51de88-dirty +- Deployment: n/a (unit test) + +## Reproduction +1. `src/xce.build.discrete/lib/xcelib_test; echo $?` at a commit before 9c51de88. +2. Observe the FAIL line, "0 Failures" and exit 0. +- Frequency: deterministic +- Evidence: output quoted above + +## Root cause +`test_libxce_retry()`, one group of test.c's single runner, called `UNITY_BEGIN()`/`UNITY_END()` itself; `UNITY_BEGIN()` resets Unity's test and failure counters, so failures from earlier groups were lost and the final `UNITY_END()` returned 0. The hidden failure: fmt_limit_units() widened the float limit value before multiplying by 100 (0.8f -> 0.800000011920929 -> "80.000001%"). + +## Fix +Removed the nested UNITY_BEGIN/UNITY_END (the runner now reports 695 tests; this was the only hidden failure), and fmt_limit_units() scales in float then widens, giving the configured 80 (9c51de88). A scan of every testsuite found no other nested runner. + +## Verification +- Test: test_fmt_limit_units_percent_limit_ignores_units_mode in src/lib/testsuite/test_limit.c: FAIL (now counted: 695 tests, 1 failure, exit 1) before the formatter fix, PASS after +- Suites: ASAN debug build (src/xce.build.discrete), `meson test --no-suite xceadmin`: 117/117 OK, 0 sanitizer reports; xcelib_test 695 tests 0 failures (2026-09-29) +- Live: 3-round sequential HA soak on rebuilt ASAN binaries (/home/claude/tmp/xce-bug1885-soak, 2026-09-29): PASS, 15 jobs, zero job loss, all 3 nodes led; SIMVERIFY 4 files 0 mismatches; each recording replays with detect_leaks=1 at exit 0, 0 mismatches; no sanitizer report in the scheduler or controller stderr + +## References +- Related: none