lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

refactor(scheduler): defer job-array expansion off the vtask thread pool

Browse source 
sched_on_job_array() expanded a compact array-job push (up to
hundreds of elements: a per-element dedup lookup plus a full deep
clone) inline, synchronously, on whatever vtaskmgr thread handled the
incoming request -- the same shared pool the scheduler's own
controller connection traffic depends on, violating ha-async-io.md's
"bounded handler work only" contract for that pool (a project-wide
design invariant). A prior finding already showed a burst of
6x50-element array pushes (300 job events landing within ~2ms of each
other) causing the scheduler to lose its controller connection within
245ms and enter a reconnect/failover loop -- directly consistent with
this mechanism, and the same reconnect storm whose resync races
motivated the companion JOBS_PUSH fix in this branch.

The request handler now does O(1) work only: validate the descriptor
and enqueue it unexpanded (XCE_SCHED_OP_UPDATE_JOB_ARRAY, mirroring
the existing XCE_SCHED_OP_ADMIN_RECORD deferred-dispatch pattern).
The actual per-element expansion and dedup check moved to
translate_one_pending_update(), which runs on the round-processing
thread (main.c's xce_sched_driver_run_normal_mode_round(), before
xce_core_begin_session() each round) -- a thread with no connection-
liveness responsibility. As a side effect this also closes the
dedup check's own pre-existing cross-thread TOCTOU race (previously
documented as bug-1714 follow-up "Bug B"): only one thread ever runs
it now.

test_job_array_expand.c and test_job_array_wire_e2e.c, which asserted
the old inline-expansion behavior, are updated to assert the new
deferred-enqueue behavior instead; expansion, dedup, and full
drain-push-release ownership coverage moved to new tests in
test_driver_translate.c.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
This commit is contained in:
Rongsong Shen 2026-09-15 09:01:47 +00:00
parent 6c201bb3e8
commit d5ec1eb3ae
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
10 changed files with 838 additions and 517 deletions
Download patch file Download diff file Expand all files Collapse all files

39
.wolf/memory.md
View file

@ -4744,3 +4744,42 @@
| 20:42 | Edited src/lib/mylock.c | added 1 condition(s) | ~303 |
| 20:56 | Edited src/lib/vtaskmgr.c | 11→13 lines | ~236 |
| 20:56 | Edited src/lib/vtaskmgr.c | 5→4 lines | ~74 |
| 23:59 | Session end: 17 writes across 5 files (test_vtaskmgr_teardown.c, test.c, meson.build, vtaskmgr.c, mylock.c) | 9 reads | ~10856 tok |
## Session: 2026-09-15 23:59
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
| 00:15 | Edited src/scheduler/testsuite/test_core_events.c | modified 1811() | ~1088 |
| 00:16 | Edited src/scheduler/testsuite/test_core_events.c | modified 1811() | ~1131 |
| 00:16 | Edited src/scheduler/testsuite/test_core_events.c | 2→6 lines | ~115 |
| 00:23 | Edited src/scheduler/src/core_events.c | added 1 condition(s) | ~686 |
| 00:24 | Edited src/scheduler/src/core_events.c | added 2 condition(s) | ~872 |
| 00:32 | Edited src/scheduler/testsuite/test_core_events.c | modified test_push_events_fires_each_hook_once_per_event_with_picker_idle() | ~865 |
| 00:35 | Edited src/scheduler/testsuite/test_driver_translate.c | 4→5 lines | ~31 |
| 00:35 | Edited src/scheduler/testsuite/test_driver_translate.c | expanded (+19 lines) | ~391 |
| 00:55 | Edited src/scheduler/testsuite/test_core_events.c | modified 1812() | ~1505 |
| 00:55 | Edited src/scheduler/testsuite/test_core_events.c | 1→3 lines | ~69 |
| 01:01 | Edited src/scheduler/src/core_events.c | added 1 condition(s) | ~896 |
| 01:27 | Edited src/scheduler/include/scheduler.h | expanded (+21 lines) | ~428 |
| 01:27 | Edited src/scheduler/include/scheduler.h | expanded (+16 lines) | ~266 |
| 01:28 | Edited src/scheduler/src/handler.c | modified REVISED() | ~514 |
| 01:28 | Edited src/scheduler/src/handler.c | added 1 condition(s) | ~252 |
| 01:29 | Edited src/scheduler/src/driver.c | 2→5 lines | ~76 |
| 01:30 | Edited src/scheduler/src/driver.c | added 12 condition(s) | ~1344 |
| 01:30 | Edited src/scheduler/src/driver.c | added 5 condition(s) | ~1031 |
| 01:30 | Edited src/scheduler/src/driver.c | modified for() | ~322 |
| 01:31 | Edited src/scheduler/testsuite/test_job_array_expand.c | modified REVISED() | ~340 |
| 01:31 | Edited src/scheduler/testsuite/test_job_array_expand.c | modified pop_job_array_update() | ~239 |
| 01:31 | Edited src/scheduler/testsuite/test_job_array_expand.c | modified test_job_array_push_enqueues_raw_descriptor_unexpanded() | ~351 |
| 01:32 | Edited src/scheduler/testsuite/test_job_array_expand.c | reduced (-7 lines) | ~21 |
| 01:33 | Edited src/scheduler/testsuite/test_job_array_expand.c | modified test_wire_dispatch_chain_delivers_job_array_descriptor() | ~1152 |
| 01:33 | Edited src/scheduler/testsuite/test_job_array_expand.c | inline fix | ~20 |
| 01:35 | Edited src/scheduler/testsuite/test_driver_translate.c | modified make_wire_job_array() | ~306 |
| 01:36 | Edited src/scheduler/testsuite/test_driver_translate.c | modified follows() | ~1396 |
| 01:36 | Edited src/scheduler/testsuite/test_driver_translate.c | 2→6 lines | ~102 |
| 01:38 | Edited src/scheduler/testsuite/test_driver_translate.c | modified test_drain_translates_and_pushes_job_array_op() | ~472 |
| 01:38 | Edited src/scheduler/testsuite/test_driver_translate.c | 1→2 lines | ~34 |
| 01:46 | Edited src/scheduler/testsuite/test_job_array_wire_e2e.c | modified vtask_entry() | ~1067 |
| 01:47 | Edited src/scheduler/testsuite/test_job_array_wire_e2e.c | modified pop_job_array_update() | ~216 |
| 01:47 | Edited src/scheduler/testsuite/test_job_array_wire_e2e.c | modified test_one_push_enqueues_job_array_descriptor() | ~1041 |