refactor(scheduler): defer job-array expansion off the vtask thread pool

sched_on_job_array() expanded a compact array-job push (up to
hundreds of elements: a per-element dedup lookup plus a full deep
clone) inline, synchronously, on whatever vtaskmgr thread handled the
incoming request -- the same shared pool the scheduler's own
controller connection traffic depends on, violating ha-async-io.md's
"bounded handler work only" contract for that pool (a project-wide
design invariant). A prior finding already showed a burst of
6x50-element array pushes (300 job events landing within ~2ms of each
other) causing the scheduler to lose its controller connection within
245ms and enter a reconnect/failover loop -- directly consistent with
this mechanism, and the same reconnect storm whose resync races
motivated the companion JOBS_PUSH fix in this branch.

The request handler now does O(1) work only: validate the descriptor
and enqueue it unexpanded (XCE_SCHED_OP_UPDATE_JOB_ARRAY, mirroring
the existing XCE_SCHED_OP_ADMIN_RECORD deferred-dispatch pattern).
The actual per-element expansion and dedup check moved to
translate_one_pending_update(), which runs on the round-processing
thread (main.c's xce_sched_driver_run_normal_mode_round(), before
xce_core_begin_session() each round) -- a thread with no connection-
liveness responsibility. As a side effect this also closes the
dedup check's own pre-existing cross-thread TOCTOU race (previously
documented as bug-1714 follow-up "Bug B"): only one thread ever runs
it now.

test_job_array_expand.c and test_job_array_wire_e2e.c, which asserted
the old inline-expansion behavior, are updated to assert the new
deferred-enqueue behavior instead; expansion, dedup, and full
drain-push-release ownership coverage moved to new tests in
test_driver_translate.c.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
This commit is contained in:
Rongsong Shen 2026-09-15 09:01:47 +00:00
parent 6c201bb3e8
commit d5ec1eb3ae
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
10 changed files with 838 additions and 517 deletions

View file

@ -1,13 +1,13 @@
{
"version": 1,
"meta": {
"lastScanned": "2026-09-15T03:56:28.973Z",
"fileCount": 567,
"lastScanned": "2026-09-15T08:47:36.542Z",
"fileCount": 572,
"hits": 0,
"misses": 0,
"renderedHash": "a9ac7e8c7eca3b5e2b406896f2877eaecf4b21d63a5ffa783b18205d4002256c",
"storeUpdatedAt": "2026-09-15T03:56:28.981Z",
"rootHash": "7500ad02fc80875a"
"renderedHash": "a6cf6257d3a03c11e763a6e34e7b004b90a51e62156a9ef766f127af914454ed",
"storeUpdatedAt": "2026-09-15T08:47:36.550Z",
"rootHash": "547dd1e8c0823374"
},
"files": {
".agent-shell/transcripts/2026-08-11-11-39-59.md": {
@ -9790,11 +9790,11 @@
},
"src/scheduler/testsuite/test_core_events.c": {
"description": "Declares and",
"tokens": 18252,
"hash": "d4594ba3d4df54ad",
"size": 68442,
"mtimeMs": 1788858634325.683,
"updatedAt": "2026-09-08T09:10:34.359Z",
"tokens": 19786,
"hash": "4489f3d00b674da8",
"size": 74197,
"mtimeMs": 1789458930665.5647,
"updatedAt": "2026-09-15T07:55:30.710Z",
"source": "hook"
},
"src/lib/resreq.c": {
@ -9844,11 +9844,11 @@
},
"src/scheduler/include/scheduler.h": {
"description": "if !defined(_SCHEDULER_H_)",
"tokens": 4716,
"hash": "92201f60362d4928",
"size": 17683,
"mtimeMs": 1788863604566.5034,
"updatedAt": "2026-09-08T10:33:24.601Z",
"tokens": 5360,
"hash": "97ef8ebcef45538e",
"size": 20099,
"mtimeMs": 1789460878191.3936,
"updatedAt": "2026-09-15T08:27:58.244Z",
"source": "hook"
},
"src/scheduler/src/plugin.c": {
@ -10249,11 +10249,11 @@
},
"src/scheduler/src/core_events.c": {
"description": "include <errno.h>",
"tokens": 16287,
"hash": "658f318636cb3ead",
"size": 61075,
"mtimeMs": 1789439677074.2285,
"updatedAt": "2026-09-15T02:34:37.112Z",
"tokens": 16984,
"hash": "4e24abe60faa7bf7",
"size": 63688,
"mtimeMs": 1789459293023.9487,
"updatedAt": "2026-09-15T08:01:33.100Z",
"source": "hook"
},
"src/lib/utils.c": {
@ -10309,6 +10309,51 @@
"mtimeMs": 1789443750926.8965,
"updatedAt": "2026-09-15T03:42:30.970Z",
"source": "hook"
},
"src/scheduler/testsuite/test_driver_translate.c": {
"description": "Declares field",
"tokens": 11757,
"hash": "3351ae8b98718b4e",
"size": 44086,
"mtimeMs": 1789461520794.2683,
"updatedAt": "2026-09-15T08:38:40.839Z",
"source": "hook"
},
"src/scheduler/src/handler.c": {
"description": "include <errno.h>",
"tokens": 5162,
"hash": "d149b77df5b2f981",
"size": 19355,
"mtimeMs": 1789460932712.6296,
"updatedAt": "2026-09-15T08:28:52.774Z",
"source": "hook"
},
"src/scheduler/src/driver.c": {
"description": "include <errno.h>",
"tokens": 11968,
"hash": "e3265c56760407ea",
"size": 44880,
"mtimeMs": 1789461058238.7385,
"updatedAt": "2026-09-15T08:30:58.282Z",
"source": "hook"
},
"src/scheduler/testsuite/test_job_array_expand.c": {
"description": "",
"tokens": 3393,
"hash": "e878cf5068b6e41a",
"size": 12786,
"mtimeMs": 1789461227452.9976,
"updatedAt": "2026-09-15T08:33:47.501Z",
"source": "hook"
},
"src/scheduler/testsuite/test_job_array_wire_e2e.c": {
"description": "Declares vtask_entry",
"tokens": 4026,
"hash": "9e542dae84fa78c1",
"size": 15107,
"mtimeMs": 1789462056502.7026,
"updatedAt": "2026-09-15T08:47:36.542Z",
"source": "hook"
}
}
}