fix: scheduler empty-resreq JOB_UPDATE crash (bug-1904); xcecli jobs list rejection text (bug-1906)
bug-1904: both scheduler JOB_UPDATE paths parsed "" as-is:
parse_resreq() returns success with a PRIMARY container whose body is
NULL, and apply_job_task_count() wrote through it. Use ingest's default
("type == any") instead.
bug-1906: xcecli jobs list sent local selection-parse errors through
ExtractErrorMessage, which strips brackets and the quoted selection;
report them as written.
Also logs bug-1905 (xcecd_virtual ignores kill of a running job). All
three found by job-array-one-based-index task 10.2's live REST flow.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
6439e5852f
commit
6e631caa22
GPG key ID:
80533536B0EB651C
9 changed files with 392 additions and 12 deletions
|
|
@ -16855,6 +16855,78 @@
|
|||
"related_bugs": [],
|
||||
"fix_commits": [],
|
||||
"record": ".wolf/bugs/bug-1903.md"
|
||||
},
|
||||
{
|
||||
"id": "bug-1904",
|
||||
"status": "open",
|
||||
"status_updated": "2026-10-01",
|
||||
"status_note": "newly logged",
|
||||
"component": "scheduler",
|
||||
"severity": "crash",
|
||||
"timestamp": "2026-10-01T14:32:17Z",
|
||||
"last_seen": "2026-10-01T14:32:17Z",
|
||||
"occurrences": 1,
|
||||
"error_message": "scheduler crashes on a JOB_UPDATE whose resreq is empty: apply_job_task_count writes through NULL\n\n**Observed:**\n```\n==1580007==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x5c8c80c00867 ...)\n==1580007==The signal is caused by a WRITE memory access.\n #0 apply_job_task_count ../lib/resreq.c:589\n #1 xce_core_push_events ../scheduler/src/core_events.c:1388\n #2 xce_sched_driver_translate_pending_events ../scheduler/src/driver.c:850\n #3 xce_sched_driver_run_normal_mode_round ../scheduler/src/driver.c:1021\n```\nThe scheduler process died (stderr was /dev/null under start-all.sh, so it vanished silently) right after the controller pushed `schedUpdateJob` for a REST-submitted job with no `-R`; every later job stayed PEND. Restarting the scheduler crashed again on the same replayed update.\n\n**Expected:** an update with an empty resource request is treated as ingest treats it (`\"type == any\"`) and the scheduler keeps running.\n**Impact:** any job submitted without a resource request (every REST submit without `-R`) kills the scheduler on its first JOB_UPDATE, and keeps killing it on restart; scheduling stops cluster-wide.\n**Found by:** spec task (job-array-one-based-index task 10.2 live REST flow)",
|
||||
"file": "src/scheduler/src/core_events.c, src/scheduler/src/schedobj.c",
|
||||
"root_cause": "Both JOB_UPDATE paths (`core_events.c` `xce_core_push_events`, `schedobj.c` update of an existing job) parsed `updated->submit->resreq` as-is. For `\"\"`, `parse_resreq()` zeroes the container (tag 0 = `XCE_RESREQ_PRIMARY`, `u.primary` NULL), parses nothing and returns success; `apply_job_task_count()` (added there by the bug-1773 fix, 41c3694d) then writes `u.primary->minTasks` through NULL. Ingest (`core_build_job`, `make_sched_job`) already maps an empty string to `\"type == any\"`; the update paths did not.",
|
||||
"fix": "Both update paths now apply the ingest default: an empty resreq string is parsed as `\"type == any\"`, and `resreq_str` records the string actually parsed.\n\nVerification:\n- Test: `test_job_update_with_empty_resreq_uses_default` in `src/scheduler/testsuite/test_maxtasks_wiring_ingest.c`; RED before the fix (the same ASAN SEGV at resreq.c:589), GREEN after\n- Suites: scheduler meson suite 97/97 OK (asan, src/build.jai)\n- Live: restarted the fixed scheduler on the same deployment: it replayed the update without crashing and every stuck PEND job (plain, arr2[1], arr2[3]) ran to DONE",
|
||||
"tags": [
|
||||
"resreq",
|
||||
"job-update",
|
||||
"asan",
|
||||
"bug-1773-regression"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1773"
|
||||
],
|
||||
"fix_commits": [],
|
||||
"record": ".wolf/bugs/bug-1904.md"
|
||||
},
|
||||
{
|
||||
"id": "bug-1905",
|
||||
"status": "open",
|
||||
"status_updated": "2026-10-01",
|
||||
"status_note": "newly logged",
|
||||
"component": "worker",
|
||||
"severity": "wrong-result",
|
||||
"timestamp": "2026-10-01T14:32:18Z",
|
||||
"last_seen": "2026-10-01T14:32:18Z",
|
||||
"occurrences": 1,
|
||||
"error_message": "xcecd_virtual ignores kill of a running job: job keeps RUN until its sleep time elapses\n\n**Observed:**\n```\n14:17:47,625 controller_connector.c:1546 Received job control request to send SIGINFO for job ID: 1-0-1790864259-0\n14:17:47,625 controller_connector.c:1610 Job control task handled directly for job ID: 1-0-1790864259-0\n14:27:40,008 xcecd_worker_utils.c:847 Sleep time 600 >= 600 seconds elapsed for worker 1009 (job 1-0-1790864259-0)\nctrl-1: on_job_finish job finish jobid 1-0-1790864259-0 exit code 0\n```\n`xcecli jobs kill` of a RUN job (plain `sleep 600`, and array elements `arr3[2]`, `arr3[3]`) is accepted (apiserver: `overallStatus accepted`, SIGTERM) and reaches the virtual worker, but the job stays RUN and finishes DONE with exit 0 when its sleep time elapses. Killing a PEND job works (EXIT).\n\n**Expected:** the virtual worker ends a killed running job as EXITED with exit code 128+signal on its next poll (the `killed_signal` branch in `xcecd_dbus_manager.c:259-262`).\n**Impact:** kill of running jobs cannot be exercised on virtual-worker deployments (tests, soaks); the real worker path (`ipc_master_send_job_signal`) is not affected by this record.\n**Found by:** spec task (job-array-one-based-index task 10.2 live REST flow)",
|
||||
"file": "src/worker/src/controller_connector.c, src/worker/src/xcecd_dbus_manager.c",
|
||||
"root_cause": "TODO: required for fixed/partial",
|
||||
"fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why",
|
||||
"tags": [
|
||||
"virtual-cluster",
|
||||
"job-control",
|
||||
"kill"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"fix_commits": [],
|
||||
"record": ".wolf/bugs/bug-1905.md"
|
||||
},
|
||||
{
|
||||
"id": "bug-1906",
|
||||
"status": "open",
|
||||
"status_updated": "2026-10-01",
|
||||
"status_note": "newly logged",
|
||||
"component": "xcecli",
|
||||
"severity": "wrong-result",
|
||||
"timestamp": "2026-10-01T14:32:21Z",
|
||||
"last_seen": "2026-10-01T14:32:21Z",
|
||||
"occurrences": 1,
|
||||
"error_message": "xcecli jobs list drops the selection and its brackets from rejection messages\n\n**Observed:**\n```\n$ xcecli jobs list '1-0-1790863351-0[1,3]'\nfailed to get parameter: [expected <job-id>, <array-id>k or <array-id>a-b]\n$ xcecli jobs kill '1-0-1790863351-0[1,3]'\ninvalid job identity or element selection '1-0-1790863351-0[1,3]': expected <job-id>, <array-id>[k] or <array-id>[a-b]\n```\n`jobs list` passed local selection-parse errors through `request.ExtractErrorMessage`, which strips every `[`/`]` and keeps only the text after the last `\": \"`.\n\n**Expected:** the rejection names the selection as written and keeps the `[k]` syntax, as `jobs kill` does (job-array-one-based-index Requirement 5: a rejection names the rejected selection).\n**Impact:** users of `jobs list` cannot tell which selection was rejected, and the hint shows `<array-id>k` instead of `<array-id>[k]`.\n**Found by:** spec task (job-array-one-based-index task 10.2 live REST flow)",
|
||||
"file": "src/frontend/xcecli/cmd/jobs/list.go",
|
||||
"root_cause": "`cmd/jobs/list.go` (`RunJobsList`, `runArrayJobStats`) formatted `HandleJobFilter` errors with `request.ExtractErrorMessage`, a sanitizer meant for transport errors: it removes all brackets and keeps only the last `\": \"`-separated part, which drops the quoted selection.",
|
||||
"fix": "Both sites report the local parse error as written (`failed to get parameter: [%v]`, err), without `ExtractErrorMessage`.\n\nVerification:\n- Test: `TestRunJobsListReportsRejectedSelectionVerbatim` in `src/frontend/xcecli/cmd/jobs/list_test.go`; RED before the fix (6 assertion failures), GREEN after\n- Suites: xcecli `go vet ./...` clean, `go test ./...` all ok (CGO_ENABLED=0); `/xce-build xcecli` asan rc=0\n- Live: rebuilt xcecli against the deployment: `jobs list` rejections now read e.g. `invalid element selection '1-0-1790863351-0[40000]': element numbers must be at most 32767`",
|
||||
"tags": [
|
||||
"job-array",
|
||||
"error-message",
|
||||
"selection"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"fix_commits": [],
|
||||
"record": ".wolf/bugs/bug-1906.md"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue