fix(scheduler): stop re-dispatching running jobs (bug-1498) + resval double-free (bug-1497)
bug-1498 (re-dispatch): sched_job_is_ready() was a stub returning 1, so any job lingering in the fcfs pendJobs queue was always elected. A dispatched job was never marked non-PEND (the controller defers its schedStartJob notify to the not-yet-implemented apply pump, controller/job.c), so fcfs_update_job re-enqueued it on every job update and the election thread re-dispatched the running job — the worker got conflicting signals and the job was SIGTERM'd. - sched_job_is_ready: elect only PEND, non-suspended jobs - sched_commit_decision (dispatch): mark the job JOB_STAT_RUN on dispatch so it leaves the PEND set and is no longer re-electable - fcfs_update_job: dedup via ordq_peek before enqueue bug-1497 (double-free): sched_update's resval paths transferred ownership of rv->val to sharedInst/w->resvals but left the request pointer set, so sched_pend_update_killer double-freed it -> use-after-free in build_host_eval_ctx. Clear rv->val = NULL on transfer. Tests: new scheduler/testsuite/test_redispatch.c (8 cases): sched_job_is_ready PEND gating + dispatch-commit marks the job RUN. Full scheduler meson suite green. Live-verified: a `sleep 60` job is elected exactly once, runs the full duration and exits 0 (`completed successfully`); before the fix it was re-dispatched at ~11s and killed (SIGTERM 143). Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
lei.chen
parent
07ab05671d
commit
ee7dee0ed4
5 changed files with 381 additions and 2 deletions
|
|
@ -14643,5 +14643,81 @@
|
|||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-08-03"
|
||||
},
|
||||
{
|
||||
"id": "bug-1496",
|
||||
"timestamp": "2026-08-03",
|
||||
"error_message": "AppImage daemon binaries hardcode /nix/store absolute paths in DT_NEEDED (sqlite) and DT_RUNPATH (all libs), so they fail to load on hosts without /nix/store (t2/t3): \"error while loading shared libraries: libsqlite3.so/libcap.so.2/libm.so.6 cannot open shared object file\".",
|
||||
"file": "src/tools/appimage/ (entry.sh), xce.nix build",
|
||||
"root_cause": "Daemons (xce_controller/xcecd/xce_scheduler/xce_apiserver) are built by nix with DT_RUNPATH = /nix/store/<hash>/lib absolute paths, and xcecd has DT_NEEDED = /nix/store/<hash>-sqlite/lib/libsqlite3.so (absolute). AppImage packaging does NOT rewrite these, so the binaries are NOT self-contained: they depend on the build machine's /nix/store. t1 happens to have /nix/store (nix env) so it works; t2/t3 without it fail.",
|
||||
"fix": "VERIFIED FIX (pilot): use patchelf to make binaries self-contained \u2014 collect non-glibc deps into lib/ dir; patchelf --set-rpath $ORIGIN/lib on the binary; patchelf --set-rpath $ORIGIN on every lib/*.so (lib-to-lib deps also have absolute RUNPATH); replace sqlite absolute DT_NEEDED with short name; set interpreter to system /lib64/ld-linux (glibc stays on host). Verified on t2 (no /nix/store): self-contained xcecd loaded all 12 non-glibc libs and connected to t1 controller. NOT YET integrated into AppImage package.sh \u2014 pending.",
|
||||
"tags": [
|
||||
"appimage",
|
||||
"rpath",
|
||||
"dt-needed",
|
||||
"self-contained",
|
||||
"nix-store-dependency",
|
||||
"verified-fix-pilot"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-08-03"
|
||||
},
|
||||
{
|
||||
"id": "bug-1497",
|
||||
"timestamp": "2026-08-03",
|
||||
"error_message": "scheduler ASAN: heap-use-after-free in build_host_eval_ctx (resource.c:170) \u2014 scheduling thread reads worker resvals freed by sched_pend_update_killer. Scheduler crashes (ASAN abort) after scheduling a job.",
|
||||
"file": "src/scheduler/src/scheduler.c (sched_handle_resvals XCE_SCHED_OP_UPDATE_RESVALS / _FULL)",
|
||||
"root_cause": "Ownership conflict: sched_handle_resvals does zarray_put(&w->resvals, r->idx, rv->val) (and &sched->sharedInst in the FULL path), transferring rv->val to the zarray whose killer (kill_sched_resval) will free_Value() it. But sched_pend_update_killer then calls free_ResValList_ptr on the request, which frees rv->val again (double-free) and leaves the worker zarray pointer dangling. The scheduling thread later reads it in build_host_eval_ctx -> use-after-free. The worker-construction path (schedobj.c:302) correctly uses MOVE_PTR(val, nv->val) to transfer ownership; sched_handle_resvals does not \u2014 inconsistency.",
|
||||
"fix": "FIXED: in sched_handle_resvals, after zarray_put(&w->resvals, r->idx, rv->val) and zarray_put(&sched->sharedInst, r->idx, rv->val), set rv->val = NULL to transfer ownership to the zarray (whose kill_sched_resval frees it). free_ResValList_ptr then skips the NULL val (free_Value_ptr is NULL-safe), eliminating the double-free/use-after-free. Verified: ASAN scheduler schedules a job to RUN->DONE with zero ASAN errors and stays alive.",
|
||||
"tags": [
|
||||
"scheduler",
|
||||
"use-after-free",
|
||||
"resvals",
|
||||
"ownership",
|
||||
"asan",
|
||||
"fixed"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1495"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-08-03"
|
||||
},
|
||||
{
|
||||
"id": "bug-1498",
|
||||
"timestamp": "2026-08-03",
|
||||
"error_message": "scheduler \u91cd\u590d\u8c03\u5ea6\u5df2\u5b8c\u6210\u7684\u4f5c\u4e1a: e2e-test (73930087014072320, sleep 10) \u5728 worker \u4e0a\u88ab\u8c03\u5ea6\u6267\u884c 3 \u6b21 (fcfs_elect_job x4), \u524d\u4e24\u6b21\u6210\u529f\u5b8c\u6210, \u7b2c\u4e09\u6b21\u88ab SIGKILL (code 143); \u6700\u7ec8 MCP \u72b6\u6001\u663e\u793a failed/EXIT, exec_hosts None; controller \u53cd\u590d\u65e5\u5fd7 rejecting backward START on terminal state",
|
||||
"file": "src/scheduler/src/scheduler.c / src/scheduler/src/decision.c (job \u5b8c\u6210\u72b6\u6001\u672a\u4ece runnable \u96c6\u5408\u79fb\u9664)",
|
||||
"root_cause": "1) sched_job_is_ready \u6052\u771f stub: pendJobs \u91cc\u4efb\u4f55\u4f5c\u4e1a\u90fd\u88ab elect; 2) \u6d3e\u53d1\u540e scheduler \u5185\u4f5c\u4e1a\u72b6\u6001\u4ecd\u4e3a PEND (controller \u4ece\u4e0d\u53d1 schedStartJob, record_job_start_fact \u7684 AFTER_EVENT \u901a\u77e5\u88ab\u63a8\u8fdf\u5230\u672a\u5b9e\u73b0\u7684 apply pump), fcfs_update_job \u5728\u6bcf\u6b21 job update \u65f6\u628a\u4ecd PEND \u7684\u4f5c\u4e1a\u91cd\u65b0\u585e\u56de pendJobs; 3) fcfs_elect_job \u6bcf\u79d2\u91cd\u626b\u91cd\u65b0 elect \u5df2\u6d3e\u53d1\u4f5c\u4e1a \u2192 \u91cd\u590d dispatch \u2192 worker \u88ab SIGTERM \u6740",
|
||||
"fix": "FIXED: 1) sched_job_is_ready() \u4ece\u6052\u771f stub \u6539\u4e3a\u68c0\u67e5 JOB_STAT_PEND \u4e14\u975e\u6302\u8d77; 2) sched_commit_decision dispatch \u5206\u652f\u628a\u4f5c\u4e1a\u6807\u8bb0\u4e3a JOB_STAT_RUN\uff08\u6d3e\u53d1\u5373\u79bb\u5f00 PEND \u96c6\u5408\uff09; 3) fcfs_update_job \u7528 ordq_peek \u53bb\u91cd\u9632\u91cd\u590d\u5165\u961f\u3002\u65b0\u589e test_redispatch.c \u56de\u5f52\u6d4b\u8bd5 (8 \u7528\u4f8b)\u3002\u5b9e\u673a\u9a8c\u8bc1: sleep 60 \u4f5c\u4e1a\u53ea elect 1 \u6b21, \u8dd1\u6ee1 60s, exit 0 completed successfully (\u5bf9\u6bd4\u4fee\u590d\u524d ~11s \u88ab\u6740 143)\u3002\u6ce8: \u4f5c\u4e1a\u4e0a\u62a5\u72b6\u6001\u4ecd\u53d7 controller apply-pump \u672a\u5b9e\u73b0\u5f71\u54cd, \u5c5e\u72ec\u7acb bug\u3002",
|
||||
"tags": [
|
||||
"scheduler",
|
||||
"re-dispatch",
|
||||
"job-lifecycle",
|
||||
"fcfs",
|
||||
"fixed"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1494"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-08-03"
|
||||
},
|
||||
{
|
||||
"id": "bug-1499",
|
||||
"timestamp": "2026-08-03",
|
||||
"error_message": "apiserver \u91cd\u542f\u540e\u6240\u6709 API key \u5931\u6548 (HTTP 401 Authentication failed / invalid or expired API key); .mcp.json \u4e2d\u7684 xce_ak_988620... key \u5728 09:07 apiserver \u91cd\u542f\u540e\u4e0d\u53ef\u7528",
|
||||
"file": "src/frontend/apiserver/store/apikey.go + cmd/main.go",
|
||||
"root_cause": "APIKeys \u53ea\u5728\u5185\u5b58 store, AddAPIKey \u4ec5\u5728 POST /xce/v1/apikeys \u751f\u6210\u63a5\u53e3\u88ab\u8c03\u7528; InitStores() \u542f\u52a8\u65f6\u4e0d seed, \u91cd\u542f\u5373\u5168\u90e8\u4e22\u5931",
|
||||
"fix": "\u4e34\u65f6: \u7528 admin \u767b\u5f55\u8c03 POST /xce/v1/apikeys \u91cd\u65b0\u751f\u6210 (scheduler-agent/mcp_operator); \u6839\u6cbb\u5f85\u5b9a (\u542f\u52a8\u65f6\u6301\u4e45\u5316/seed API keys)",
|
||||
"tags": [
|
||||
"apiserver",
|
||||
"apikey",
|
||||
"restart-loss"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-08-03"
|
||||
}
|
||||
]
|
||||
Loading…
Add table
Add a link
Reference in a new issue