lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

refactor(scheduler): defer job-array expansion off the vtask thread pool

Browse source 
sched_on_job_array() expanded a compact array-job push (up to
hundreds of elements: a per-element dedup lookup plus a full deep
clone) inline, synchronously, on whatever vtaskmgr thread handled the
incoming request -- the same shared pool the scheduler's own
controller connection traffic depends on, violating ha-async-io.md's
"bounded handler work only" contract for that pool (a project-wide
design invariant). A prior finding already showed a burst of
6x50-element array pushes (300 job events landing within ~2ms of each
other) causing the scheduler to lose its controller connection within
245ms and enter a reconnect/failover loop -- directly consistent with
this mechanism, and the same reconnect storm whose resync races
motivated the companion JOBS_PUSH fix in this branch.

The request handler now does O(1) work only: validate the descriptor
and enqueue it unexpanded (XCE_SCHED_OP_UPDATE_JOB_ARRAY, mirroring
the existing XCE_SCHED_OP_ADMIN_RECORD deferred-dispatch pattern).
The actual per-element expansion and dedup check moved to
translate_one_pending_update(), which runs on the round-processing
thread (main.c's xce_sched_driver_run_normal_mode_round(), before
xce_core_begin_session() each round) -- a thread with no connection-
liveness responsibility. As a side effect this also closes the
dedup check's own pre-existing cross-thread TOCTOU race (previously
documented as bug-1714 follow-up "Bug B"): only one thread ever runs
it now.

test_job_array_expand.c and test_job_array_wire_e2e.c, which asserted
the old inline-expansion behavior, are updated to assert the new
deferred-enqueue behavior instead; expansion, dedup, and full
drain-push-release ownership coverage moved to new tests in
test_driver_translate.c.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
This commit is contained in:
Rongsong Shen 2026-09-15 09:01:47 +00:00
parent 6c201bb3e8
commit d5ec1eb3ae
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
10 changed files with 838 additions and 517 deletions
Download patch file Download diff file Expand all files Collapse all files

37
src/scheduler/include/scheduler.h
View file

@ -40,6 +40,27 @@ typedef enum {
* XCE_SCHED_OP_UPDATE_JOB (the dataset bulk-create path) so a notify
* update to an existing job mutates it in place and never creates a job. */
XCE_SCHED_OP_UPDATE_JOB_FIELDS,
/* Compact job-array push (job-array-single-wal-record's xcesched_
* push_jobarray() wire shape: ONE JobSubmit{jobid, arraySize, submit}
* carrying an array's base jobid, element count, and one shared job
* spec) -- deferred from the request-handling thread (handler.c's
* scheduler_req_handler()) to the driver thread, the same pattern
* XCE_SCHED_OP_ADMIN_RECORD below already establishes. Per-element
* expansion (N independent deep clones via clone_job_submit_data()/
* clone_job_startinfo(), driver.c) used to run inline in
* sched_on_job_array() on whatever vtaskmgr thread handled the incoming
* push -- for a large concurrent burst (live-observed: 6x50-element
* array pushes landing within ~2ms of each other) this occupies several
* of the vtaskmgr pool's threads for the duration of the expansion, the
* exact "long/unbounded work on a pool that must stay free for
* liveness-critical messages" anti-pattern ha-async-io.md's Thread-role
* contract warns about (the pool is shared with the scheduler's own
* controller connection traffic). The vtask-thread handler now only
* takes ownership of the compact descriptor and enqueues it -- O(1),
* bounded -- translate_one_pending_update() (driver.c) does the actual
* per-element expansion on the dedicated round-processing thread
* instead, which carries no connection liveness responsibility. */
XCE_SCHED_OP_UPDATE_JOB_ARRAY,
/* Scheduler admin recording control (scheduler-core spec, task 9.4, Req
* 5 AC 10): a `recordStart`/`recordStop` SchedAdminRequest, deferred
* from the request-handling thread (handler.c's scheduler_req_handler())
@ -69,6 +90,22 @@ typedef struct {
struct {
xce_job_t* job;
} jobfieldsupdate;
struct {
/* Input: raw compact array descriptor -- ownership transferred
* here from sched_handle_dataset()'s XceData_data_jobArray
* case. See XCE_SCHED_OP_UPDATE_JOB_ARRAY's own comment above. */
xce_job_submit_t* job_array;
/* Output: populated by translate_one_pending_update()'s
* expansion of job_array above -- the N per-element clones,
* the SAME pointers aliased into the translated
* XCE_CORE_EVENT_JOBS_PUSH event's jobs_push.jobs, freed by
* release_translated_payload() the same way
* XCE_SCHED_OP_UPDATE_JOB's own jobupdate.jobs is freed. Zero/
* NULL until expansion runs (or if it produced nothing to
* push -- an all-duplicate redelivery). */
int expanded_njobs;
xce_job_t** expanded_jobs;
} jobarrayupdate;
struct {
int nworkers;
xce_worker_t** workers;