myxce/.wolf/bugs/bug-1813.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	legacy
bug-1813	fixed	2026-09-24		scheduler	crash	2026-09-15T08:35:00Z	2026-09-19T09:00:00Z	1	src/scheduler/include/scheduler.h src/scheduler/src/handler.c src/scheduler/src/driver.c src/scheduler/testsuite/test_job_array_expand.c src/scheduler/testsuite/test_job_array_wire_e2e.c src/scheduler/testsuite/test_driver_translate.c	scheduler vtaskmgr async-io connection-stability job-array architecture user-directed	bug-1812 bug-1734 bug-1739 bug-1714	true

bug-1813: Follow-up to bug-1812's investigation (user-directed architectural fix, not a live-reproduced…

Description

Follow-up to bug-1812's investigation (user-directed architectural fix, not a live-reproduced crash): while investigating why a scheduler reconnect/resync storm collides with in-flight array-job dispatches (bug-1812's own root cause), user asked to investigate the connection-loss trigger itself. Traced the scheduler's array-job receive path (handler.c's sched_on_job_array(), reached via scheduler_req_handler() on the shared vtaskmgr thread pool, default 8 threads) and found it does the FULL per-element expansion (dedup check via sched_get_job() + N independent deep clones via clone_job_submit_data()/clone_job_startinfo()) inline, synchronously, on whatever vtask thread handles the incoming push -- the exact 'long/unbounded work on a pool that must stay free for liveness-critical messages' anti-pattern .kiro/steering/ha-async-io.md's Thread-role contract warns about (the same pool is shared with the scheduler's own controller connection traffic). A prior, independent finding (2026-09-04 discovery, this project) already showed a burst of 6x50-element array pushes (300 job events, landing within ~2ms of each other) caused the scheduler to lose its connection to the primary controller within 245ms and enter a failover/reconnect loop -- directly consistent with this mechanism. User proposed the fix directly: the request-handling thread should just requeue the raw request, with the actual (expensive) work deferred to run alongside 'begin session' -- i.e. the round-processing thread's own xce_sched_driver_translate_pending_events()/xce_core_begin_session() sequence (main.c's xce_sched_driver_run_normal_mode_round(), driver.c), which carries no connection-liveness responsibility and already establishes this exact deferred-dispatch pattern for XCE_SCHED_OP_ADMIN_RECORD.

Environment

Not recorded (migrated from buglog.json).

Reproduction

Not recorded (migrated from buglog.json).

Root cause

CONFIRMED via direct code reading (not yet live-reproduced end-to-end with gdb/ASAN under a real burst -- this fix is the same corrective direction bug-1714's own still-open investigation was heading, applied proactively here rather than waiting for a full live repro). handler.c's sched_on_job_array() (the XceData_data_jobArray case's handler, invoked by scheduler_req_handler() on a vtaskmgr thread) did the ENTIRE per-element expansion synchronously and inline: for each of up to array_size elements, a sched_get_job() dedup lookup plus a full deep clone (clone_job_submit_data()/clone_job_startinfo(), several calloc()+strdup() calls per element) before ever enqueueing anything. For a large concurrent burst (6 arrays x 50 elements landing within ~2ms, the 2026-09-04 finding), this ties up several of the vtaskmgr pool's 8 threads for the duration of 300 elements' worth of cloning -- the same pool the scheduler's own controller connection traffic depends on, a direct instance of ha-async-io.md's documented anti-pattern (a project-wide DESIGN INVARIANT, 'applies to ALL components: controller, worker, scheduler').

Fix

FIXED (2026-09-15), TDD RED->GREEN, user-directed design. Added XCE_SCHED_OP_UPDATE_JOB_ARRAY (scheduler.h) mirroring XCE_SCHED_OP_ADMIN_RECORD's own established deferred-dispatch pattern, with a new xce_sched_update_request_t.u.jobarrayupdate union member carrying the raw, still-unexpanded xce_job_submit_t* descriptor plus (post-expansion) expanded_njobs/expanded_jobs for release_translated_payload()'s later cleanup. handler.c's sched_on_job_array() now does O(1) work only: validate the descriptor shape, wrap it in a request, enq_pend_update() -- no loop, no clone, no dedup check, ownership of the descriptor transferred to the queue on success (sched_handle_dataset()'s XceData_data_jobArray case updated to null dataset->rows on success instead of unconditionally freeing it). The full per-element expansion logic (clone_job_startinfo/clone_job_submit_data/make_array_element, moved verbatim) plus its dedup check now live in driver.c's translate_one_pending_update(), a new XCE_SCHED_OP_UPDATE_JOB_ARRAY case producing the same XCE_CORE_EVENT_JOBS_PUSH shape the old handler.c code did -- this runs on the round-processing thread (main.c's xce_sched_driver_run_normal_mode_round(), invoked before xce_core_begin_session() every round), which carries no connection-liveness responsibility. Side benefit: the dedup check's own pre-existing TOCTOU race (documented in the original code as bug-1714 follow-up 'Bug B', a genuine cross-thread race between the vtask thread's check and the driver thread's own core->jobs mutations) is now eliminated for the single-request case, since only the round-processing thread ever runs this check -- the remaining same-batch-double-delivery edge case is unchanged and was already documented as safe-by-design (sched_add_job()'s key_add() plus the JOBS_PUSH duplicate-merge logic, core_events.c, is the actual correctness boundary, untouched by this fix). release_translated_payload() (driver.c) gained a matching cleanup case: frees the original job_array descriptor unconditionally (never itself pushed into core) plus the N expanded clones (same unconditional free XCE_SCHED_OP_UPDATE_JOB's own jobupdate.jobs already gets). TDD: 4 new unit tests in test_driver_translate.c directly on translate_one_pending_update() (expands into N events with independent clones; skips already-existing elements against a REAL core->jobs kvmgr; all-duplicate redelivery returns 0/no-op; malformed payload returns 0) plus 1 new full drain-push-release end-to-end test (test_drain_translates_and_pushes_job_array_op) proving ownership correctness under real ASAN instrumentation. test_job_array_expand.c's and test_job_array_wire_e2e.c's own tests (which previously asserted the OLD inline-expansion behavior, including test_job_array_wire_e2e.c's tlog early-print-callback log-count capture, since the confirmation log line moved to driver.c) were revised to assert the NEW deferred-enqueue behavior instead -- redelivery dedup and log-firing-count coverage moved to test_driver_translate.c's own tests (log-firing-count itself was not re-added at the new location, a deliberate small scope trim, noted here rather than silently dropped). RED confirmed at each stage (xce-build.sh scheduler asan xce.build --incremental): first pass caught the 4 new driver-level unit test failures as expected plus 2 pre-existing test_job_array_wire_e2e.c failures this refactor had NOT yet been applied to (a test file this investigation initially missed, found only via the full suite run -- both asserted 'Expected 2 Was 11', i.e. the old handler-level XCE_SCHED_OP_UPDATE_JOB=2 check now correctly seeing the new XCE_SCHED_OP_UPDATE_JOB_ARRAY=11). GREEN confirmed after revising that file too: full scheduler suite 77/77 Ok, zero regressions, including test_sim_e2e_original_motivation.c (verified explicitly, given this same session's earlier unrelated mistake on bug-1811/1812's own investigation broke it once already) and all 10 new/revised job-array tests individually confirmed PASS via the raw test log.

Verification

Not recorded (migrated from buglog.json).

References

• Related: bug-1812, bug-1734, bug-1739, bug-1714