chore(buglog): bug-1897 fixed (0fe27d4b)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
0fe27d4be3
commit
da5c3065ed
GPG key ID:
80533536B0EB651C
2 changed files with 17 additions and 14 deletions
|
|
@ -16665,9 +16665,9 @@
|
|||
},
|
||||
{
|
||||
"id": "bug-1897",
|
||||
"status": "open",
|
||||
"status_updated": "2026-09-30",
|
||||
"status_note": "newly logged",
|
||||
"status": "fixed",
|
||||
"status_updated": "2026-10-01",
|
||||
"status_note": "",
|
||||
"component": "scheduler",
|
||||
"severity": "crash",
|
||||
"timestamp": "2026-09-30T10:19:59Z",
|
||||
|
|
@ -16675,8 +16675,8 @@
|
|||
"occurrences": 1,
|
||||
"error_message": "xce_scheduler --sim segfaults in limit_job_election on a job whose snapshot resreq failed to parse\n\n**Observed:** replaying `data/scheduler-replay-20260928-064842.replay` in simulation mode crashes before any decision, 3/3 runs:\n```\n[ERROR] [xce_sched_core_load_snapshot] xce_sched_core_load_snapshot parse_resreq failed for jobid 4667606771186008064: Primary: ...\n==2723571==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000008 (pc 0x555555a7cb6e ... T3)\n #0 limit_job_election ../scheduler/src/plugins/limit.c:336\n #1 plugin_job_election ../scheduler/src/plugin.c:601\n #2 sched_allocator ../scheduler/src/sched_election.c:236\n```\nThe replay's snapshot stores resreqs in the debug-dump format that bug-1888 wrote before its fix. Since the bug-1887 fix, `parse_resreq` rejects that text, so the snapshot loader logs the error but keeps the job with no schedobj. The allocator thread's job election then reads `job->schedobj` as NULL in the limit plugin.\n\n**Expected:** a job whose resreq cannot be parsed from a snapshot is either dropped (with the error logged) or kept pending with a reason, and never reaches election with a NULL schedobj; the simulator must not crash on an old replay.\n\n**Impact:** crash (SIGSEGV) of `xce_scheduler --sim` on any replay whose snapshot has unparsable resreqs, including every replay captured before the bug-1888 fix. Suspected: the same path could crash a normal-mode scheduler loading such a snapshot (not tried).\n\n**Found by:** feature-level validation smoke run of spec `scheduler-allocator-runner` (2026-09-30); not caused by that feature (limit.c, snapshot_load.c, sched_election.c untouched by it).",
|
||||
"file": "src/scheduler/src/plugins/limit.c, src/scheduler/src/snapshot_load.c",
|
||||
"root_cause": "TODO: required for fixed/partial",
|
||||
"fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why",
|
||||
"root_cause": "UPDATE (2026-10-01): A job whose snapshot resreq fails to parse is kept by xce_sched_core_load_snapshot() (snapshot_load.c) with resreq and schedobj both NULL, so a running job's restored allocation stays charged. Nothing kept it out of the picker: the loader pushed it with on_job_push(), and in simulation mode sim_bootstrap_picker() (sim_picker.c) re-pushes every core job once the picker exists (bug-1567), which is the path the replay took, since the sim loads its snapshot before any picker is registered. The allocator thread (sched_allocator(), sched_election.c) then ran plugin_job_election() and sched_make_allocation() on the NULL schedobj, and limit_job_election() (limit.c:336) dereferenced it. Exposed since the bug-1887 fix made parse_resreq() reject the debug-dump resreq text that snapshots wrote before the bug-1888 fix, i.e. every replay captured before that fix; the live JOBS_PUSH path was never affected (core_build_job() drops such a job).",
|
||||
"fix": "UPDATE (2026-10-01): A job without a schedobj is never made a picker candidate: snapshot_load.c skips on_job_push() for it (logging once that it will not be scheduled) and sim_picker.c's picker rebuild skips it too. The job stays in the core, so its restored allocation is still charged and later events still find it. As a backstop, sched_allocator() (sched_election.c) skips any picked job whose schedobj is NULL, the same way it skips a job it cannot find, so no other route into the picker can crash the allocator thread.\n\nVerification:\n- Test: `test_snapshot_load_leaves_schedobj_null_when_resreq_fails_to_parse` (testsuite/test_snapshot_save_load.c, new no-picker-candidate assertion) RED (\"Expected 0 Was 1\"), GREEN after; `test_sched_allocator_skips_job_without_schedobj` (testsuite/test_sched_election_null_schedobj.c, new) RED (ASan SEGV in sched_make_allocation, scheduler.c:447), GREEN after; `test_snapshot_job_with_unparsable_resreq_is_not_scheduled` (testsuite/test_sim_event_loop.c, new) RED (a PEND decision for the job, with the sim_picker.c and sched_election.c changes reverted), GREEN after.\n- Suites: scheduler ASAN suite 97/97 (build.alloc), 2026-10-01.\n- Live: n/a for HA (no HA path touched). The original replay (`scheduler-replay-20260928-064842.replay`, `--sim`) completes all 1511 sessions in 3/3 runs: exit 0, no ASan report, the 50 affected jobs reported once at load as unschedulable, 0 allocator skips.",
|
||||
"tags": [
|
||||
"scheduler",
|
||||
"sim",
|
||||
|
|
@ -16687,7 +16687,9 @@
|
|||
"bug-1887",
|
||||
"bug-1888"
|
||||
],
|
||||
"fix_commits": [],
|
||||
"fix_commits": [
|
||||
"0fe27d4b"
|
||||
],
|
||||
"record": ".wolf/bugs/bug-1897.md"
|
||||
}
|
||||
]
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
---
|
||||
id: "bug-1897"
|
||||
status: "open"
|
||||
status_updated: "2026-09-30"
|
||||
status_note: "newly logged"
|
||||
status: "fixed"
|
||||
status_updated: "2026-10-01"
|
||||
status_note: ""
|
||||
component: "scheduler"
|
||||
severity: "crash"
|
||||
timestamp: "2026-09-30T10:19:59Z"
|
||||
|
|
@ -11,6 +11,7 @@ occurrences: 1
|
|||
files: ["src/scheduler/src/plugins/limit.c", "src/scheduler/src/snapshot_load.c"]
|
||||
tags: ["scheduler", "sim", "snapshot", "limit"]
|
||||
related: ["bug-1887", "bug-1888"]
|
||||
fix_commits: ["0fe27d4b"]
|
||||
---
|
||||
|
||||
# bug-1897: xce_scheduler --sim segfaults in limit_job_election on a job whose snapshot resreq failed to parse
|
||||
|
|
@ -48,15 +49,15 @@ The replay's snapshot stores resreqs in the debug-dump format that bug-1888 wrot
|
|||
- Evidence: `/home/claude/tmp/bug-1897/` (replay file copy, `run.out` with the ASan report, `sim.log`)
|
||||
|
||||
## Root cause
|
||||
TODO: required for fixed/partial
|
||||
UPDATE (2026-10-01): A job whose snapshot resreq fails to parse is kept by xce_sched_core_load_snapshot() (snapshot_load.c) with resreq and schedobj both NULL, so a running job's restored allocation stays charged. Nothing kept it out of the picker: the loader pushed it with on_job_push(), and in simulation mode sim_bootstrap_picker() (sim_picker.c) re-pushes every core job once the picker exists (bug-1567), which is the path the replay took, since the sim loads its snapshot before any picker is registered. The allocator thread (sched_allocator(), sched_election.c) then ran plugin_job_election() and sched_make_allocation() on the NULL schedobj, and limit_job_election() (limit.c:336) dereferenced it. Exposed since the bug-1887 fix made parse_resreq() reject the debug-dump resreq text that snapshots wrote before the bug-1888 fix, i.e. every replay captured before that fix; the live JOBS_PUSH path was never affected (core_build_job() drops such a job).
|
||||
|
||||
## Fix
|
||||
TODO: required for fixed/partial
|
||||
UPDATE (2026-10-01): A job without a schedobj is never made a picker candidate: snapshot_load.c skips on_job_push() for it (logging once that it will not be scheduled) and sim_picker.c's picker rebuild skips it too. The job stays in the core, so its restored allocation is still charged and later events still find it. As a backstop, sched_allocator() (sched_election.c) skips any picked job whose schedobj is NULL, the same way it skips a job it cannot find, so no other route into the picker can crash the allocator thread.
|
||||
|
||||
## Verification
|
||||
- Test: TODO: name + file; RED before the fix, GREEN after
|
||||
- Suites: TODO: suite results and build profile
|
||||
- Live: TODO: soak/deploy run, or n/a and why
|
||||
- Test: `test_snapshot_load_leaves_schedobj_null_when_resreq_fails_to_parse` (testsuite/test_snapshot_save_load.c, new no-picker-candidate assertion) RED ("Expected 0 Was 1"), GREEN after; `test_sched_allocator_skips_job_without_schedobj` (testsuite/test_sched_election_null_schedobj.c, new) RED (ASan SEGV in sched_make_allocation, scheduler.c:447), GREEN after; `test_snapshot_job_with_unparsable_resreq_is_not_scheduled` (testsuite/test_sim_event_loop.c, new) RED (a PEND decision for the job, with the sim_picker.c and sched_election.c changes reverted), GREEN after.
|
||||
- Suites: scheduler ASAN suite 97/97 (build.alloc), 2026-10-01.
|
||||
- Live: n/a for HA (no HA path touched). The original replay (`scheduler-replay-20260928-064842.replay`, `--sim`) completes all 1511 sessions in 3/3 runs: exit 0, no ASan report, the 50 affected jobs reported once at load as unschedulable, 0 allocator skips.
|
||||
|
||||
## References
|
||||
- Related: none
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue