lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

refactor(scheduler): defer job-array expansion off the vtask thread pool

Browse source 
sched_on_job_array() expanded a compact array-job push (up to
hundreds of elements: a per-element dedup lookup plus a full deep
clone) inline, synchronously, on whatever vtaskmgr thread handled the
incoming request -- the same shared pool the scheduler's own
controller connection traffic depends on, violating ha-async-io.md's
"bounded handler work only" contract for that pool (a project-wide
design invariant). A prior finding already showed a burst of
6x50-element array pushes (300 job events landing within ~2ms of each
other) causing the scheduler to lose its controller connection within
245ms and enter a reconnect/failover loop -- directly consistent with
this mechanism, and the same reconnect storm whose resync races
motivated the companion JOBS_PUSH fix in this branch.

The request handler now does O(1) work only: validate the descriptor
and enqueue it unexpanded (XCE_SCHED_OP_UPDATE_JOB_ARRAY, mirroring
the existing XCE_SCHED_OP_ADMIN_RECORD deferred-dispatch pattern).
The actual per-element expansion and dedup check moved to
translate_one_pending_update(), which runs on the round-processing
thread (main.c's xce_sched_driver_run_normal_mode_round(), before
xce_core_begin_session() each round) -- a thread with no connection-
liveness responsibility. As a side effect this also closes the
dedup check's own pre-existing cross-thread TOCTOU race (previously
documented as bug-1714 follow-up "Bug B"): only one thread ever runs
it now.

test_job_array_expand.c and test_job_array_wire_e2e.c, which asserted
the old inline-expansion behavior, are updated to assert the new
deferred-enqueue behavior instead; expansion, dedup, and full
drain-push-release ownership coverage moved to new tests in
test_driver_translate.c.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
This commit is contained in:
Rongsong Shen 2026-09-15 09:01:47 +00:00
parent 6c201bb3e8
commit d5ec1eb3ae
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
10 changed files with 838 additions and 517 deletions
Download patch file Download diff file Expand all files Collapse all files

85
.wolf/anatomy-index.json
View file

@ -1,13 +1,13 @@
{
"version": 1,
"meta": {
"lastScanned": "2026-09-15T03:56:28.973Z",
"fileCount": 567,
"lastScanned": "2026-09-15T08:47:36.542Z",
"fileCount": 572,
"hits": 0,
"misses": 0,
"renderedHash": "a9ac7e8c7eca3b5e2b406896f2877eaecf4b21d63a5ffa783b18205d4002256c",
"storeUpdatedAt": "2026-09-15T03:56:28.981Z",
"rootHash": "7500ad02fc80875a"
"renderedHash": "a6cf6257d3a03c11e763a6e34e7b004b90a51e62156a9ef766f127af914454ed",
"storeUpdatedAt": "2026-09-15T08:47:36.550Z",
"rootHash": "547dd1e8c0823374"
},
"files": {
".agent-shell/transcripts/2026-08-11-11-39-59.md": {
@ -9790,11 +9790,11 @@
},
"src/scheduler/testsuite/test_core_events.c": {
"description": "Declares and",
"tokens": 18252,
"hash": "d4594ba3d4df54ad",
"size": 68442,
"mtimeMs": 1788858634325.683,
"updatedAt": "2026-09-08T09:10:34.359Z",
"tokens": 19786,
"hash": "4489f3d00b674da8",
"size": 74197,
"mtimeMs": 1789458930665.5647,
"updatedAt": "2026-09-15T07:55:30.710Z",
"source": "hook"
},
"src/lib/resreq.c": {
@ -9844,11 +9844,11 @@
},
"src/scheduler/include/scheduler.h": {
"description": "if !defined(_SCHEDULER_H_)",
"tokens": 4716,
"hash": "92201f60362d4928",
"size": 17683,
"mtimeMs": 1788863604566.5034,
"updatedAt": "2026-09-08T10:33:24.601Z",
"tokens": 5360,
"hash": "97ef8ebcef45538e",
"size": 20099,
"mtimeMs": 1789460878191.3936,
"updatedAt": "2026-09-15T08:27:58.244Z",
"source": "hook"
},
"src/scheduler/src/plugin.c": {
@ -10249,11 +10249,11 @@
},
"src/scheduler/src/core_events.c": {
"description": "include <errno.h>",
"tokens": 16287,
"hash": "658f318636cb3ead",
"size": 61075,
"mtimeMs": 1789439677074.2285,
"updatedAt": "2026-09-15T02:34:37.112Z",
"tokens": 16984,
"hash": "4e24abe60faa7bf7",
"size": 63688,
"mtimeMs": 1789459293023.9487,
"updatedAt": "2026-09-15T08:01:33.100Z",
"source": "hook"
},
"src/lib/utils.c": {
@ -10309,6 +10309,51 @@
"mtimeMs": 1789443750926.8965,
"updatedAt": "2026-09-15T03:42:30.970Z",
"source": "hook"
},
"src/scheduler/testsuite/test_driver_translate.c": {
"description": "Declares field",
"tokens": 11757,
"hash": "3351ae8b98718b4e",
"size": 44086,
"mtimeMs": 1789461520794.2683,
"updatedAt": "2026-09-15T08:38:40.839Z",
"source": "hook"
},
"src/scheduler/src/handler.c": {
"description": "include <errno.h>",
"tokens": 5162,
"hash": "d149b77df5b2f981",
"size": 19355,
"mtimeMs": 1789460932712.6296,
"updatedAt": "2026-09-15T08:28:52.774Z",
"source": "hook"
},
"src/scheduler/src/driver.c": {
"description": "include <errno.h>",
"tokens": 11968,
"hash": "e3265c56760407ea",
"size": 44880,
"mtimeMs": 1789461058238.7385,
"updatedAt": "2026-09-15T08:30:58.282Z",
"source": "hook"
},
"src/scheduler/testsuite/test_job_array_expand.c": {
"description": "",
"tokens": 3393,
"hash": "e878cf5068b6e41a",
"size": 12786,
"mtimeMs": 1789461227452.9976,
"updatedAt": "2026-09-15T08:33:47.501Z",
"source": "hook"
},
"src/scheduler/testsuite/test_job_array_wire_e2e.c": {
"description": "Declares vtask_entry",
"tokens": 4026,
"hash": "9e542dae84fa78c1",
"size": 15107,
"mtimeMs": 1789462056502.7026,
"updatedAt": "2026-09-15T08:47:36.542Z",
"source": "hook"
}
}
}

15
.wolf/anatomy.md
View file

@ -1,7 +1,7 @@
# anatomy.md
> Auto-maintained by OpenWolf. Last scanned: 2026-09-15T03:56:28.973Z
> Files: 567 tracked | Anatomy hits: 0 | Misses: 0
> Auto-maintained by OpenWolf. Last scanned: 2026-09-15T08:47:36.542Z
> Files: 572 tracked | Anatomy hits: 0 | Misses: 0
## ./
@ -906,12 +906,14 @@
## src/scheduler/include/
- `scheduler.h` — if !defined(_SCHEDULER_H_) (~4716 tok)
- `scheduler.h` — if !defined(_SCHEDULER_H_) (~5360 tok)
- `sim.h` — if !defined(_SIM_H_) (~4213 tok)
## src/scheduler/src/
- `core_events.c` — include <errno.h> (~16287 tok)
- `core_events.c` — include <errno.h> (~16984 tok)
- `driver.c` — include <errno.h> (~11968 tok)
- `handler.c` — include <errno.h> (~5162 tok)
- `plugin.c` — include <errno.h> (~4554 tok)
- `repl_break.c` — include <stdio.h> (~9921 tok)
- `repl_info.c` — include <stdio.h> (~16226 tok)
@ -927,7 +929,10 @@
## src/scheduler/testsuite/
- `test_core_events.c` — Declares and (~18252 tok)
- `test_core_events.c` — Declares and (~19786 tok)
- `test_driver_translate.c` — Declares field (~11757 tok)
- `test_job_array_expand.c` (~3393 tok)
- `test_job_array_wire_e2e.c` — Declares vtask_entry (~4026 tok)
- `test_parjob_combinatorial.c` — Declares of (~12038 tok)
- `test_repl_breakpoints_inspection.c` — Declares and (~23533 tok)
- `test_repl_help.c` — Declares error (~6047 tok)

73
.wolf/buglog.json
View file

@ -22007,6 +22007,79 @@
],
"occurrences": 1,
"last_seen": "2026-09-15T03:55:00Z"
},
{
"id": "bug-1811",
"timestamp": "2026-09-15T07:40:00Z",
"error_message": "User-reported (interactive sim session=35, `info jobs`/`info worker pc13v`): job 0-14-1789369200-0[21] shows RUN in `info jobs` but is absent from `info worker pc13v`'s assigned-jobs listing (which lists [0-20,22], skipping 21) -- user's working theory was slot overcommitment (23 running jobs vs. 22-slot worker capacity).",
"file": "src/scheduler/src/core_events.c, src/scheduler/src/sim_loop.c, .kiro/specs/scheduler-sim-decision-parity/{requirements,design}.md",
"root_cause": "NOT A BUG -- initial diagnosis was WRONG and the attempted fix was reverted. Root-caused (correctly, mechanism-wise) to XCE_CORE_EVENT_JOB_STATE_CHANGE/XCE_CORE_EVENT_JOB_UPDATE (core_events.c) setting job->state=RUN unconditionally even when job->allocs/job->accounts are still NULL -- repl_info_worker_one() (repl_info.c) walks job->allocs to build the assigned-jobs listing, so an unaccounted RUN job is correctly displayed as RUN in `info jobs` but silently absent from `info worker`'s listing. Initially misdiagnosed this AS a defect (a 'phantom RUN' analogous to bug-1739's JOBS_PUSH resync gap) and implemented a guard refusing the RUN transition without an existing allocation. That guard was WRONG: this exact 'RUN with allocs==NULL' transient window is the INTENTIONAL, designed, tested trigger condition for Recorded-Decision Reconciliation (scheduler-sim-decision-parity spec, sim_loop.c's sim_reconcile_unaccounted_jobs(), extended by bug-1804's 2026-09-11 revision to set job->state=RUN synchronously via JOB_STATE_CHANGE/JOB_UPDATE ahead of allocation capture, matching the original live scheduler's own sched_commit_decision() semantics). Confirmed via test_sim_e2e_original_motivation.c's own header comment and ANCHOR_ID fixture: the SAME core_events.c code path is used, by design, to construct a job that is 'already running, with an unaccounted allocation, when a scheduling round's own live decision-making reaches it' -- 'Recorded-Decision Reconciliation's entire reason for existing'. My guard broke this fixture (test_e2e_original_motivation_long_replay) plus test_push_events_fires_each_hook_once_per_event_with_picker_idle and test_drain_translates_and_pushes_all_ten_op_types, both of which legitimately exercise a bare JOB_STATE_CHANGE/JOB_UPDATE(RUN) with no allocation yet. Further confirmed against scheduler-sim-decision-parity/requirements.md Requirement 4 ('Graceful degradation when recorded decision data is unavailable'): AC 4.1/4.2 EXPLICITLY anticipate and accept that a job's originating DISPATCH decision may not be found in the recorded replay data, in which case the scheduler reports that job's usage as unavailable rather than fabricating a value, and continues processing -- this is precisely job 21's observed symptom, not an accident. The user's overcommitment theory is also independently wrong on the accounting side: sched_account_jobs() only charges resInUse for JOB_IS_RUN(state) && accounts != NULL, so an unaccounted RUN job is correctly EXCLUDED from the real slot-usage total (used=22.00 already correctly reflects only the 22 genuinely-accounted jobs) -- there is no real overcommitment, only a display gap for a job whose allocation reconciliation hasn't (yet, or ever, depending on whether its DISPATCH decision exists later in this replay stream) caught up.",
"fix": "REVERTED. Initial fix (guard in core_events.c's XCE_CORE_EVENT_JOB_STATE_CHANGE/JOB_UPDATE refusing a RUN claim without an existing allocation, plus 6 new tests in test_core_events.c and a fixture change in test_driver_translate.c) was implemented, RED/GREEN-tested, then found to break test_sim_e2e_original_motivation.c's test_e2e_original_motivation_long_replay (the live-verify for the entire Recorded-Decision Reconciliation mechanism) plus two pre-existing test_core_events.c/test_driver_translate.c tests. All changes reverted via `git checkout --` before any commit. No production code changed. Left as an open question whether job 21 (and the user's follow-up observation that this 'occurs more frequently') ever actually reconciles later in the same replay session -- if a job's RUN-but-unaccounted display state NEVER resolves even after the replay completes, that would indicate its DISPATCH decision is genuinely missing from the recorded stream (the same category of gap scheduler-snapshot-job-allocation's PREEXIST_ID addresses for pre-recording dispatches, via snapshot-based restoration instead of replay-based reconciliation) -- worth a follow-up investigation with concrete evidence (does the job ever show up in `info worker`'s listing on a later round?) before attempting any further fix. A safe, smaller follow-up actually worth considering: improve `info worker`'s assigned-jobs display (repl_info.c) to explicitly distinguish a RUN-but-unaccounted job (e.g. an annotation) from silent omission, so operators aren't misled into an overcommitment theory the way this session's user was -- proposed to the user, not yet implemented. SUPERSEDED (2026-09-15, same session): the 'not a bug' conclusion above was itself only half right. The transient RUN-before-allocation window IS legitimate (Recorded-Decision Reconciliation), but the specific PERMANENT loss the user kept observing turned out to be a real, separate defect in the JOBS_PUSH duplicate-jobid resync merge -- see bug-1812 for the actual root cause and fix.",
"tags": [
"not-a-bug",
"design-decision",
"scheduler",
"sim-replay",
"recorded-decision-reconciliation",
"reverted-fix"
],
"related_bugs": [
"bug-1739",
"bug-1804",
"bug-1778"
],
"occurrences": 1,
"last_seen": "2026-09-15T07:40:00Z"
},
{
"id": "bug-1812",
"timestamp": "2026-09-15T08:05:00Z",
"error_message": "User-reported (interactive sim session=34->35 stepped live, AND independently confirmed via a real production scheduler.log from rshen@pc13v /t/xce-ha-deploy): job 0-14-1789369200-0[21] holds allocs=1 (state=RUN) at the end of session 34; stepping into session 35 replays a JOBS_PUSH resync (njobs=60, including all of array job 0-14-1789369200-0[0-39]) followed by a JOB_UPDATE and a redundant JOB_STATE_CHANGE(RUN) for [21]; immediately after, `info job [21]` shows state=RUN but allocs=0. The real scheduler.log shows the identical mechanism live: `sched_job_state_change ... state 2` (schedStartJob confirmation) at 15:00:05.196, followed 16ms later by `JOBS_PUSH: sched_add_job failed for jobid ...[21] (rc=-17), merging state into existing tracked job instead of discarding` -- repeating in bursts on later reconnects (six WARNs in ~2s at 15:01:51-53).",
"file": "src/scheduler/src/core_events.c (XCE_CORE_EVENT_JOBS_PUSH duplicate-jobid merge, bug-1734/bug-1739's own code)",
"root_cause": "CONFIRMED, via direct code reading plus independent live evidence from both a sim replay and a real production scheduler.log. xce_core_push_events()'s JOBS_PUSH handler, on a duplicate jobid (sched_add_job() -EEXIST, a resync redelivery of an already-tracked job), branches three ways once job_state_regresses_terminal() is ruled out: (1) resync claims RUN with real allocation data -> transfer it in (bug-1739's fix); (2) resync claims RUN with NO allocation data -> refuse, keep existing (bug-1739's own companion guard); (3) resync claims anything else (the bare `else` branch, bug-1734's original fix) -> UNCONDITIONALLY frees existing->accounts/existing->allocs and overwrites existing->state with the resync's own claim, with NO check for whether `existing` already holds a REAL, already-captured allocation the resync claim doesn't support. on_scheduler_join()'s full resync (xcesched_push_jobs(), src/controller/sched.c:214, fired on every scheduler (re)connect via join.c) is built from collect_jobs_committed() -- a committed-data snapshot gated only on 'job submission is committed', NOT on whether this specific job's DISPATCH transition has been committed yet. When a scheduler reconnects (or the connection blips) around the same moment it makes/receives confirmation of its own fresh dispatch decision for a job, the resync's committed-data snapshot can legitimately still show that job as PEND (its own dispatch commit hasn't propagated into the snapshot query yet) even though the scheduler's own live state (already set via sched_commit_decision() or the schedStartJob/JOB_STATE_CHANGE confirmation path) correctly has it RUN with a real allocation. Branch (3) blindly trusts the resync's stale PEND claim over the scheduler's own more-current, already-allocated state, freeing the real allocation and (in the observed cases) leaving state briefly PEND until a later, in-flight JOB_UPDATE/JOB_STATE_CHANGE for the SAME jobid (already queued in the same event batch, confirming the SAME dispatch) re-sets state=RUN -- but with no mechanism to restore the allocation branch (3) just destroyed. This exactly matches both the sim replay symptom (info jobs shows RUN, info worker's allocs-driven assigned-jobs listing omits it) and the production log's precise 16ms collision. Not the same mechanism as bug-1811's original (correct, but incomplete) 'Recorded-Decision Reconciliation transient window' explanation -- that window is real and self-healing, but this is a genuinely separate, permanent-until-this-fix data-loss bug triggered by resync/dispatch timing, not the reconciliation mechanism itself.",
"fix": "FIXED (2026-09-15), TDD RED->GREEN. Added a guard in JOBS_PUSH's duplicate-jobid merge (core_events.c), inserted between bug-1739's `else if (JOB_IS_RUN(job->state))` branch and the original bare `else`: `else if ((existing->allocs != NULL) && !JOB_IS_DONE(job->state) && !JOB_IS_EXIT(job->state))` -- refuses the resync claim (WARNs, keeps existing->state/allocs/accounts untouched) whenever `existing` already holds a real allocation AND the resync's claimed state is not genuinely terminal. A terminal claim (DONE/EXIT) still falls through to the original branch, correctly releasing the allocation -- this is bug-1734's own legitimate 'job finished while disconnected' case and must not be blocked. Symmetric in spirit to bug-1739's own phantom-RUN guard (refuse to fabricate an unsupported RUN; now also refuse to destroy a supported RUN). TDD: added test_jobs_push_duplicate_jobid_run_with_allocation_survives_stale_nonrun_resync (test_core_events.c) -- job tracked RUN with a real allocation, redelivered via JOBS_PUSH as PEND with no allocation, must keep its RUN state and real allocation intact; and its companion test_jobs_push_duplicate_jobid_terminal_resync_releases_real_allocation -- the SAME RUN-with-real-allocation job, redelivered as DONE, must still release the allocation and transition (proves the new guard doesn't block legitimate terminal resyncs). RED confirmed (`xce-build.sh scheduler asan xce.build --incremental`): exactly the new stale-resync test failed (Expected 2 [RUN] Was 1 [PEND]), the terminal-resync companion already passed (unaffected by the unfixed code), 75 other tests green. GREEN confirmed after the fix: full scheduler suite 77/77 Ok, zero regressions -- including test_sim_e2e_original_motivation.c (the Recorded-Decision Reconciliation live-verify a FIRST, incorrect fix attempt this same session had broken and had to be reverted) and test_driver_translate.c, both untouched by this fix and green throughout.",
"tags": [
"scheduler",
"resync",
"jobs-push",
"allocation-loss",
"race-condition",
"fixed"
],
"related_bugs": [
"bug-1734",
"bug-1738",
"bug-1739",
"bug-1811"
],
"occurrences": 1,
"last_seen": "2026-09-15T08:05:00Z"
},
{
"id": "bug-1813",
"timestamp": "2026-09-15T08:35:00Z",
"error_message": "Follow-up to bug-1812's investigation (user-directed architectural fix, not a live-reproduced crash): while investigating why a scheduler reconnect/resync storm collides with in-flight array-job dispatches (bug-1812's own root cause), user asked to investigate the connection-loss trigger itself. Traced the scheduler's array-job receive path (handler.c's sched_on_job_array(), reached via scheduler_req_handler() on the shared vtaskmgr thread pool, default 8 threads) and found it does the FULL per-element expansion (dedup check via sched_get_job() + N independent deep clones via clone_job_submit_data()/clone_job_startinfo()) inline, synchronously, on whatever vtask thread handles the incoming push -- the exact 'long/unbounded work on a pool that must stay free for liveness-critical messages' anti-pattern .kiro/steering/ha-async-io.md's Thread-role contract warns about (the same pool is shared with the scheduler's own controller connection traffic). A prior, independent finding (2026-09-04 discovery, this project) already showed a burst of 6x50-element array pushes (300 job events, landing within ~2ms of each other) caused the scheduler to lose its connection to the primary controller within 245ms and enter a failover/reconnect loop -- directly consistent with this mechanism. User proposed the fix directly: the request-handling thread should just requeue the raw request, with the actual (expensive) work deferred to run alongside 'begin session' -- i.e. the round-processing thread's own xce_sched_driver_translate_pending_events()/xce_core_begin_session() sequence (main.c's xce_sched_driver_run_normal_mode_round(), driver.c), which carries no connection-liveness responsibility and already establishes this exact deferred-dispatch pattern for XCE_SCHED_OP_ADMIN_RECORD.",
"file": "src/scheduler/include/scheduler.h, src/scheduler/src/handler.c, src/scheduler/src/driver.c, src/scheduler/testsuite/test_job_array_expand.c, src/scheduler/testsuite/test_job_array_wire_e2e.c, src/scheduler/testsuite/test_driver_translate.c",
"root_cause": "CONFIRMED via direct code reading (not yet live-reproduced end-to-end with gdb/ASAN under a real burst -- this fix is the same corrective direction bug-1714's own still-open investigation was heading, applied proactively here rather than waiting for a full live repro). handler.c's sched_on_job_array() (the XceData_data_jobArray case's handler, invoked by scheduler_req_handler() on a vtaskmgr thread) did the ENTIRE per-element expansion synchronously and inline: for each of up to array_size elements, a sched_get_job() dedup lookup plus a full deep clone (clone_job_submit_data()/clone_job_startinfo(), several calloc()+strdup() calls per element) before ever enqueueing anything. For a large concurrent burst (6 arrays x 50 elements landing within ~2ms, the 2026-09-04 finding), this ties up several of the vtaskmgr pool's 8 threads for the duration of 300 elements' worth of cloning -- the same pool the scheduler's own controller connection traffic depends on, a direct instance of ha-async-io.md's documented anti-pattern (a project-wide DESIGN INVARIANT, 'applies to ALL components: controller, worker, scheduler').",
"fix": "FIXED (2026-09-15), TDD RED->GREEN, user-directed design. Added XCE_SCHED_OP_UPDATE_JOB_ARRAY (scheduler.h) mirroring XCE_SCHED_OP_ADMIN_RECORD's own established deferred-dispatch pattern, with a new xce_sched_update_request_t.u.jobarrayupdate union member carrying the raw, still-unexpanded xce_job_submit_t* descriptor plus (post-expansion) expanded_njobs/expanded_jobs for release_translated_payload()'s later cleanup. handler.c's sched_on_job_array() now does O(1) work only: validate the descriptor shape, wrap it in a request, enq_pend_update() -- no loop, no clone, no dedup check, ownership of the descriptor transferred to the queue on success (sched_handle_dataset()'s XceData_data_jobArray case updated to null dataset->rows on success instead of unconditionally freeing it). The full per-element expansion logic (clone_job_startinfo/clone_job_submit_data/make_array_element, moved verbatim) plus its dedup check now live in driver.c's translate_one_pending_update(), a new XCE_SCHED_OP_UPDATE_JOB_ARRAY case producing the same XCE_CORE_EVENT_JOBS_PUSH shape the old handler.c code did -- this runs on the round-processing thread (main.c's xce_sched_driver_run_normal_mode_round(), invoked before xce_core_begin_session() every round), which carries no connection-liveness responsibility. Side benefit: the dedup check's own pre-existing TOCTOU race (documented in the original code as bug-1714 follow-up 'Bug B', a genuine cross-thread race between the vtask thread's check and the driver thread's own core->jobs mutations) is now eliminated for the single-request case, since only the round-processing thread ever runs this check -- the remaining same-batch-double-delivery edge case is unchanged and was already documented as safe-by-design (sched_add_job()'s key_add() plus the JOBS_PUSH duplicate-merge logic, core_events.c, is the actual correctness boundary, untouched by this fix). release_translated_payload() (driver.c) gained a matching cleanup case: frees the original job_array descriptor unconditionally (never itself pushed into core) plus the N expanded clones (same unconditional free XCE_SCHED_OP_UPDATE_JOB's own jobupdate.jobs already gets). TDD: 4 new unit tests in test_driver_translate.c directly on translate_one_pending_update() (expands into N events with independent clones; skips already-existing elements against a REAL core->jobs kvmgr; all-duplicate redelivery returns 0/no-op; malformed payload returns 0) plus 1 new full drain-push-release end-to-end test (test_drain_translates_and_pushes_job_array_op) proving ownership correctness under real ASAN instrumentation. test_job_array_expand.c's and test_job_array_wire_e2e.c's own tests (which previously asserted the OLD inline-expansion behavior, including test_job_array_wire_e2e.c's tlog early-print-callback log-count capture, since the confirmation log line moved to driver.c) were revised to assert the NEW deferred-enqueue behavior instead -- redelivery dedup and log-firing-count coverage moved to test_driver_translate.c's own tests (log-firing-count itself was not re-added at the new location, a deliberate small scope trim, noted here rather than silently dropped). RED confirmed at each stage (xce-build.sh scheduler asan xce.build --incremental): first pass caught the 4 new driver-level unit test failures as expected plus 2 pre-existing test_job_array_wire_e2e.c failures this refactor had NOT yet been applied to (a test file this investigation initially missed, found only via the full suite run -- both asserted 'Expected 2 Was 11', i.e. the old handler-level XCE_SCHED_OP_UPDATE_JOB=2 check now correctly seeing the new XCE_SCHED_OP_UPDATE_JOB_ARRAY=11). GREEN confirmed after revising that file too: full scheduler suite 77/77 Ok, zero regressions, including test_sim_e2e_original_motivation.c (verified explicitly, given this same session's earlier unrelated mistake on bug-1811/1812's own investigation broke it once already) and all 10 new/revised job-array tests individually confirmed PASS via the raw test log.",
"tags": [
"scheduler",
"vtaskmgr",
"async-io",
"connection-stability",
"job-array",
"architecture",
"fixed",
"user-directed"
],
"related_bugs": [
"bug-1812",
"bug-1734",
"bug-1739",
"bug-1714"
],
"occurrences": 1,
"last_seen": "2026-09-15T08:35:00Z"
}
]
}

39
.wolf/memory.md
View file

@ -4744,3 +4744,42 @@
| 20:42 | Edited src/lib/mylock.c | added 1 condition(s) | ~303 |
| 20:56 | Edited src/lib/vtaskmgr.c | 11→13 lines | ~236 |
| 20:56 | Edited src/lib/vtaskmgr.c | 5→4 lines | ~74 |
| 23:59 | Session end: 17 writes across 5 files (test_vtaskmgr_teardown.c, test.c, meson.build, vtaskmgr.c, mylock.c) | 9 reads | ~10856 tok |
## Session: 2026-09-15 23:59
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
| 00:15 | Edited src/scheduler/testsuite/test_core_events.c | modified 1811() | ~1088 |
| 00:16 | Edited src/scheduler/testsuite/test_core_events.c | modified 1811() | ~1131 |
| 00:16 | Edited src/scheduler/testsuite/test_core_events.c | 2→6 lines | ~115 |
| 00:23 | Edited src/scheduler/src/core_events.c | added 1 condition(s) | ~686 |
| 00:24 | Edited src/scheduler/src/core_events.c | added 2 condition(s) | ~872 |
| 00:32 | Edited src/scheduler/testsuite/test_core_events.c | modified test_push_events_fires_each_hook_once_per_event_with_picker_idle() | ~865 |
| 00:35 | Edited src/scheduler/testsuite/test_driver_translate.c | 4→5 lines | ~31 |
| 00:35 | Edited src/scheduler/testsuite/test_driver_translate.c | expanded (+19 lines) | ~391 |
| 00:55 | Edited src/scheduler/testsuite/test_core_events.c | modified 1812() | ~1505 |
| 00:55 | Edited src/scheduler/testsuite/test_core_events.c | 1→3 lines | ~69 |
| 01:01 | Edited src/scheduler/src/core_events.c | added 1 condition(s) | ~896 |
| 01:27 | Edited src/scheduler/include/scheduler.h | expanded (+21 lines) | ~428 |
| 01:27 | Edited src/scheduler/include/scheduler.h | expanded (+16 lines) | ~266 |
| 01:28 | Edited src/scheduler/src/handler.c | modified REVISED() | ~514 |
| 01:28 | Edited src/scheduler/src/handler.c | added 1 condition(s) | ~252 |
| 01:29 | Edited src/scheduler/src/driver.c | 2→5 lines | ~76 |
| 01:30 | Edited src/scheduler/src/driver.c | added 12 condition(s) | ~1344 |
| 01:30 | Edited src/scheduler/src/driver.c | added 5 condition(s) | ~1031 |
| 01:30 | Edited src/scheduler/src/driver.c | modified for() | ~322 |
| 01:31 | Edited src/scheduler/testsuite/test_job_array_expand.c | modified REVISED() | ~340 |
| 01:31 | Edited src/scheduler/testsuite/test_job_array_expand.c | modified pop_job_array_update() | ~239 |
| 01:31 | Edited src/scheduler/testsuite/test_job_array_expand.c | modified test_job_array_push_enqueues_raw_descriptor_unexpanded() | ~351 |
| 01:32 | Edited src/scheduler/testsuite/test_job_array_expand.c | reduced (-7 lines) | ~21 |
| 01:33 | Edited src/scheduler/testsuite/test_job_array_expand.c | modified test_wire_dispatch_chain_delivers_job_array_descriptor() | ~1152 |
| 01:33 | Edited src/scheduler/testsuite/test_job_array_expand.c | inline fix | ~20 |
| 01:35 | Edited src/scheduler/testsuite/test_driver_translate.c | modified make_wire_job_array() | ~306 |
| 01:36 | Edited src/scheduler/testsuite/test_driver_translate.c | modified follows() | ~1396 |
| 01:36 | Edited src/scheduler/testsuite/test_driver_translate.c | 2→6 lines | ~102 |
| 01:38 | Edited src/scheduler/testsuite/test_driver_translate.c | modified test_drain_translates_and_pushes_job_array_op() | ~472 |
| 01:38 | Edited src/scheduler/testsuite/test_driver_translate.c | 1→2 lines | ~34 |
| 01:46 | Edited src/scheduler/testsuite/test_job_array_wire_e2e.c | modified vtask_entry() | ~1067 |
| 01:47 | Edited src/scheduler/testsuite/test_job_array_wire_e2e.c | modified pop_job_array_update() | ~216 |
| 01:47 | Edited src/scheduler/testsuite/test_job_array_wire_e2e.c | modified test_one_push_enqueues_job_array_descriptor() | ~1041 |

37
src/scheduler/include/scheduler.h
View file

@ -40,6 +40,27 @@ typedef enum {
* XCE_SCHED_OP_UPDATE_JOB (the dataset bulk-create path) so a notify
* update to an existing job mutates it in place and never creates a job. */
XCE_SCHED_OP_UPDATE_JOB_FIELDS,
/* Compact job-array push (job-array-single-wal-record's xcesched_
* push_jobarray() wire shape: ONE JobSubmit{jobid, arraySize, submit}
* carrying an array's base jobid, element count, and one shared job
* spec) -- deferred from the request-handling thread (handler.c's
* scheduler_req_handler()) to the driver thread, the same pattern
* XCE_SCHED_OP_ADMIN_RECORD below already establishes. Per-element
* expansion (N independent deep clones via clone_job_submit_data()/
* clone_job_startinfo(), driver.c) used to run inline in
* sched_on_job_array() on whatever vtaskmgr thread handled the incoming
* push -- for a large concurrent burst (live-observed: 6x50-element
* array pushes landing within ~2ms of each other) this occupies several
* of the vtaskmgr pool's threads for the duration of the expansion, the
* exact "long/unbounded work on a pool that must stay free for
* liveness-critical messages" anti-pattern ha-async-io.md's Thread-role
* contract warns about (the pool is shared with the scheduler's own
* controller connection traffic). The vtask-thread handler now only
* takes ownership of the compact descriptor and enqueues it -- O(1),
* bounded -- translate_one_pending_update() (driver.c) does the actual
* per-element expansion on the dedicated round-processing thread
* instead, which carries no connection liveness responsibility. */
XCE_SCHED_OP_UPDATE_JOB_ARRAY,
/* Scheduler admin recording control (scheduler-core spec, task 9.4, Req
* 5 AC 10): a `recordStart`/`recordStop` SchedAdminRequest, deferred
* from the request-handling thread (handler.c's scheduler_req_handler())
@ -69,6 +90,22 @@ typedef struct {
struct {
xce_job_t* job;
} jobfieldsupdate;
struct {
/* Input: raw compact array descriptor -- ownership transferred
* here from sched_handle_dataset()'s XceData_data_jobArray
* case. See XCE_SCHED_OP_UPDATE_JOB_ARRAY's own comment above. */
xce_job_submit_t* job_array;
/* Output: populated by translate_one_pending_update()'s
* expansion of job_array above -- the N per-element clones,
* the SAME pointers aliased into the translated
* XCE_CORE_EVENT_JOBS_PUSH event's jobs_push.jobs, freed by
* release_translated_payload() the same way
* XCE_SCHED_OP_UPDATE_JOB's own jobupdate.jobs is freed. Zero/
* NULL until expansion runs (or if it produced nothing to
* push -- an all-duplicate redelivery). */
int expanded_njobs;
xce_job_t** expanded_jobs;
} jobarrayupdate;
struct {
int nworkers;
xce_worker_t** workers;

248
src/scheduler/src/driver.c
View file

@ -31,6 +31,9 @@
* XCE_SCHED_OP_UPDATE_RESVALS_FULL -> XCE_CORE_EVENT_RESVALS_FULL
* XCE_SCHED_OP_UPDATE_RESVALS -> XCE_CORE_EVENT_RESVALS_DELTA
* XCE_SCHED_OP_JOB_RUSAGE -> XCE_CORE_EVENT_JOB_RUSAGE
* XCE_SCHED_OP_UPDATE_JOB_ARRAY -> XCE_CORE_EVENT_JOBS_PUSH (N-way
* expansion -- see XCE_SCHED_OP_UPDATE_JOB_ARRAY's own comment,
* scheduler.h, and this file's job-array expansion helpers below)
*
* Two documented judgment calls on fields the legacy
* `xce_sched_update_request_t` payload simply does not carry (handler.c is
@ -176,11 +179,169 @@ static void release_translated_payload(xce_sched_core_t* core,
break;
}
case XCE_SCHED_OP_UPDATE_JOB_ARRAY: {
int i;
/* The original, still-unexpanded descriptor -- never itself pushed
* into core (only the clones below are), so freed unconditionally,
* same as XCE_SCHED_OP_UPDATE_JOB_FIELDS's job above. */
free_JobSubmit_ptr(&(req->u.jobarrayupdate.job_array)); /* NULL-safe */
/* The N per-element clones translate_one_pending_update() built --
* same unconditional free XCE_SCHED_OP_UPDATE_JOB's jobupdate.jobs
* gets above (core_build_job() deep-copies what it needs out of
* each wire xce_job_t; nothing here is retained by core->jobs
* itself). NULL/0 if translation never reached expansion (an
* invalid payload, an all-duplicate redelivery, or an OOM mid-loop
* that already cleaned up after itself). */
for (i = 0; i < req->u.jobarrayupdate.expanded_njobs; i++) {
free_Job_ptr(&(req->u.jobarrayupdate.expanded_jobs[i]));
}
free(req->u.jobarrayupdate.expanded_jobs);
break;
}
default:
break;
}
}
/*
* clone_job_startinfo/clone_job_submit_data/make_array_element - job-array
* expansion helpers, MOVED HERE from handler.c (connection-loss-under-burst
* fix -- see XCE_SCHED_OP_UPDATE_JOB_ARRAY's own comment, scheduler.h) so
* the per-element deep-clone work they do runs on this file's
* round-processing thread instead of on whatever vtaskmgr thread handled
* the incoming array push. Logic unchanged from the original handler.c
* versions (job-array-single-wal-record task 5.3).
*
* Each expanded element gets its OWN independent deep copy of the shared
* submit spec, never a shared pointer: the existing per-job pipeline this
* feeds into (XCE_CORE_EVENT_JOBS_PUSH -> core_build_job(), and this file's
* own release_translated_payload()) calls free_Job_ptr() on every element
* independently, which would double-free a submit pointer shared across
* more than one xce_job_t.
*/
static job_startinfo_t* clone_job_startinfo(const job_startinfo_t* src)
{
job_startinfo_t* d;
if (src == NULL) {
return NULL;
}
d = (job_startinfo_t*)calloc(1, sizeof(*d));
if (d == NULL) {
return NULL;
}
#define CLONE_STR_FIELD(f) \
do { \
if (src->f != NULL) { \
d->f = strdup(src->f); \
if (d->f == NULL) { \
goto fail; \
} \
} \
} while (0)
CLONE_STR_FIELD(exechome);
CLONE_STR_FIELD(execcwd);
CLONE_STR_FIELD(execuser);
CLONE_STR_FIELD(cmdline);
CLONE_STR_FIELD(exdata);
CLONE_STR_FIELD(launch_type);
CLONE_STR_FIELD(q_pre_exec);
CLONE_STR_FIELD(q_post_exec);
CLONE_STR_FIELD(job_pre_exec);
CLONE_STR_FIELD(job_post_exec);
#undef CLONE_STR_FIELD
return d;
fail:
free_JobStartInfo_ptr(&d);
return NULL;
}
static job_submit_data_t* clone_job_submit_data(const job_submit_data_t* src)
{
job_submit_data_t* d;
if (src == NULL) {
return NULL;
}
d = (job_submit_data_t*)calloc(1, sizeof(*d));
if (d == NULL) {
return NULL;
}
d->mintasks = src->mintasks;
d->maxtasks = src->maxtasks;
#define CLONE_STR_FIELD(f) \
do { \
if (src->f != NULL) { \
d->f = strdup(src->f); \
if (d->f == NULL) { \
goto fail; \
} \
} \
} while (0)
CLONE_STR_FIELD(queue);
CLONE_STR_FIELD(user);
CLONE_STR_FIELD(job_name);
CLONE_STR_FIELD(project_name);
CLONE_STR_FIELD(user_group);
CLONE_STR_FIELD(job_description);
CLONE_STR_FIELD(resreq);
#undef CLONE_STR_FIELD
if (src->start_info != NULL) {
d->start_info = clone_job_startinfo(src->start_info);
if (d->start_info == NULL) {
goto fail;
}
}
return d;
fail:
free_JobSubmitData_ptr(&d);
return NULL;
}
/* One array element, freshly-submitted: JOB_STAT_PEND / receive-time
* sub_time, mirroring the state a genuinely-new single-job push arrives
* with. The compact wire descriptor (Req 1.1/6.1) carries only (jobid,
* array_size, submit) by design -- no per-element timestamps -- so there is
* no "real" sub_time to recover here. */
static xce_job_t* make_array_element(const xce_job_submit_t* job_array,
uint64_t jobid)
{
xce_job_t* j = (xce_job_t*)calloc(1, sizeof(*j));
if (j == NULL) {
return NULL;
}
j->jobid = jobid;
j->state = JOB_STAT_PEND;
j->sub_time = (uint64_t)time(NULL);
j->submit = clone_job_submit_data(job_array->submit);
if (j->submit == NULL) {
free(j);
return NULL;
}
return j;
}
/*
* translate_one_pending_update - translates one dequeued
* xce_sched_update_request_t into *ev per this file's op-type table.
@ -276,6 +437,93 @@ static int translate_one_pending_update(xce_sched_core_t* core,
ev->u.job_rusage.rusages = req->u.job_rusage.rusages;
return 1;
case XCE_SCHED_OP_UPDATE_JOB_ARRAY: {
/* Connection-loss-under-burst fix -- see XCE_SCHED_OP_UPDATE_JOB_
* ARRAY's own comment (scheduler.h). Per-element expansion moved
* here, verbatim, from handler.c's former sched_on_job_array() --
* same base_jobid + i arithmetic, same best-effort dedup rationale
* (bug-1714 follow-up Bug B's own comment applies unchanged: this is
* now race-free WITHIN one drain batch's translation pass since
* only this one thread ever runs it, but a duplicate delivered
* within the SAME batch still isn't caught here -- core->jobs isn't
* updated until xce_core_push_events() runs once for the whole
* batch, after this pass completes -- the actual, unaffected
* correctness boundary remains sched_add_job()'s key_add() plus the
* JOBS_PUSH duplicate-merge handling, core_events.c). */
xce_job_submit_t* job_array = req->u.jobarrayupdate.job_array;
xce_job_t** jobs;
uint64_t base_jobid, array_size, i;
int n_new = 0;
char jobidString[128];
if ((job_array == NULL) || (job_array->submit == NULL)
|| (job_array->array_size == 0)) {
WARN("%s UPDATE_JOB_ARRAY: invalid job_array payload, skipping",
__func__);
return 0;
}
base_jobid = job_array->jobid;
array_size = job_array->array_size;
jobs = (xce_job_t**)calloc((size_t)array_size, sizeof(xce_job_t*));
if (jobs == NULL) {
ERROR("%s UPDATE_JOB_ARRAY: out of memory allocating a "
"%llu-element expansion array",
__func__, (unsigned long long)array_size);
return 0;
}
for (i = 0; i < array_size; i++) {
uint64_t jobid = base_jobid + i;
if (sched_get_job(core, jobid) != NULL) {
continue;
}
jobs[n_new] = make_array_element(job_array, jobid);
if (jobs[n_new] == NULL) {
int j;
for (j = 0; j < n_new; j++) {
free_Job_ptr(&jobs[j]);
}
free(jobs);
ERROR("%s UPDATE_JOB_ARRAY: out of memory cloning element "
"%llu",
__func__, (unsigned long long)jobid);
return 0;
}
n_new++;
}
if (n_new == 0) {
/* Every index already exists: a fully-duplicate redelivery.
* Nothing new to schedule -- same "no-op, not a fresh
* expansion" Observable the original handler.c version
* documented (Req 8.4). */
free(jobs);
return 0;
}
fmt_jobid(jobidString, sizeof(jobidString) - 1, base_jobid);
INFO("job array push received and expanded: base_jobid=%s "
"array_size=%llu new_tasks=%d",
jobidString, (unsigned long long)array_size, n_new);
/* Stash the expansion result back on req so release_translated_
* payload() can free it later -- these are freshly-allocated clones
* distinct from job_array (the original, still-unexpanded
* descriptor, freed separately -- see that function's own case). */
req->u.jobarrayupdate.expanded_njobs = n_new;
req->u.jobarrayupdate.expanded_jobs = jobs;
ev->type = XCE_CORE_EVENT_JOBS_PUSH;
ev->u.jobs_push.njobs = n_new;
ev->u.jobs_push.jobs = jobs;
return 1;
}
default:
WARN("%s unrecognized pending-update op %d, skipping", __func__,
(int)req->op);

268
src/scheduler/src/handler.c
View file

@ -42,242 +42,45 @@ static int sched_on_jobs(xce_scheduler_t* sched, int njobs, xce_job_t** jobs)
/* job-array-single-wal-record task 5.3 (design.md "Compact Scheduler/
* API-Server Push", File Structure Plan handler.c bullet; Req 6.3, 6.4, 6.5,
* 8.4): sibling case to XceData_data_schedjobs (sched_handle_dataset, below)
* decoding the compact Data.jobArray descriptor -- task 5.1's
* xcesched_push_jobarray() wire shape, one JobSubmit{jobid, arraySize,
* submit} carrying the array's base jobid, element count and ONE shared job
* spec -- and expanding it locally into N ordinary schedulable tasks via the
* same base_jobid + i arithmetic the controller's own restore_job_array()
* (src/controller/job.c) uses to derive each element's identity.
* 8.4), REVISED (connection-loss-under-burst fix, see
* XCE_SCHED_OP_UPDATE_JOB_ARRAY's own comment in scheduler.h): decodes the
* compact Data.jobArray descriptor -- task 5.1's xcesched_push_jobarray()
* wire shape, one JobSubmit{jobid, arraySize, submit} carrying the array's
* base jobid, element count and ONE shared job spec -- and hands it off to
* the driver thread UNEXPANDED. The per-element expansion into N ordinary
* schedulable tasks (clone_job_submit_data()/clone_job_startinfo()/
* make_array_element(), base_jobid + i arithmetic) now lives in driver.c's
* translate_one_pending_update(), which runs on the round-processing thread,
* not here on whatever vtaskmgr thread handled the incoming push -- a large
* concurrent burst of array pushes no longer ties up several vtask threads
* for the duration of N deep clones.
*
* Each expanded element gets its OWN independent deep copy of the shared
* submit spec (clone_job_submit_data/clone_job_startinfo below), never a
* shared pointer: the existing per-job pipeline this feeds into
* (sched_on_jobs -> XCE_SCHED_OP_UPDATE_JOB -> sched_pend_update_killer)
* calls free_Job_ptr() on every element independently, which would
* double-free a submit pointer shared across more than one xce_job_t. */
static job_startinfo_t* clone_job_startinfo(const job_startinfo_t* src)
{
job_startinfo_t* d;
if (src == NULL) {
return NULL;
}
d = (job_startinfo_t*)calloc(1, sizeof(*d));
if (d == NULL) {
return NULL;
}
#define CLONE_STR_FIELD(f) \
do { \
if (src->f != NULL) { \
d->f = strdup(src->f); \
if (d->f == NULL) { \
goto fail; \
} \
} \
} while (0)
CLONE_STR_FIELD(exechome);
CLONE_STR_FIELD(execcwd);
CLONE_STR_FIELD(execuser);
CLONE_STR_FIELD(cmdline);
CLONE_STR_FIELD(exdata);
CLONE_STR_FIELD(launch_type);
CLONE_STR_FIELD(q_pre_exec);
CLONE_STR_FIELD(q_post_exec);
CLONE_STR_FIELD(job_pre_exec);
CLONE_STR_FIELD(job_post_exec);
#undef CLONE_STR_FIELD
return d;
fail:
free_JobStartInfo_ptr(&d);
return NULL;
}
static job_submit_data_t* clone_job_submit_data(const job_submit_data_t* src)
{
job_submit_data_t* d;
if (src == NULL) {
return NULL;
}
d = (job_submit_data_t*)calloc(1, sizeof(*d));
if (d == NULL) {
return NULL;
}
d->mintasks = src->mintasks;
d->maxtasks = src->maxtasks;
#define CLONE_STR_FIELD(f) \
do { \
if (src->f != NULL) { \
d->f = strdup(src->f); \
if (d->f == NULL) { \
goto fail; \
} \
} \
} while (0)
CLONE_STR_FIELD(queue);
CLONE_STR_FIELD(user);
CLONE_STR_FIELD(job_name);
CLONE_STR_FIELD(project_name);
CLONE_STR_FIELD(user_group);
CLONE_STR_FIELD(job_description);
CLONE_STR_FIELD(resreq);
#undef CLONE_STR_FIELD
if (src->start_info != NULL) {
d->start_info = clone_job_startinfo(src->start_info);
if (d->start_info == NULL) {
goto fail;
}
}
return d;
fail:
free_JobSubmitData_ptr(&d);
return NULL;
}
/* One array element, freshly-submitted: JOB_STAT_PEND / receive-time
* sub_time, mirroring the state a genuinely-new single-job push arrives
* with. The compact wire descriptor (Req 1.1/6.1) carries only (jobid,
* array_size, submit) by design -- no per-element timestamps -- so there is
* no "real" sub_time to recover here. */
static xce_job_t* make_array_element(const xce_job_submit_t* job_array,
uint64_t jobid)
{
xce_job_t* j = (xce_job_t*)calloc(1, sizeof(*j));
if (j == NULL) {
return NULL;
}
j->jobid = jobid;
j->state = JOB_STAT_PEND;
j->sub_time = (uint64_t)time(NULL);
j->submit = clone_job_submit_data(job_array->submit);
if (j->submit == NULL) {
free(j);
return NULL;
}
return j;
}
* `job_array` ownership transfers to the enqueued request on success (the
* caller, sched_handle_dataset() below, must not free it in that case);
* on failure (-EINVAL/-ENOMEM) ownership is NOT taken -- the caller still
* owns and must free it, same convention every other sched_on_*() helper in
* this file already follows. */
static int sched_on_job_array(xce_scheduler_t* sched,
xce_job_submit_t* job_array)
{
xce_job_t** jobs;
uint64_t base_jobid;
uint64_t array_size;
uint64_t i;
int n_new = 0;
int rc;
char jobidString[128];
xce_sched_update_request_t* req;
if ((sched == NULL) || (job_array == NULL) || (job_array->submit == NULL)
|| (job_array->array_size == 0)) {
return -EINVAL;
}
base_jobid = job_array->jobid;
array_size = job_array->array_size;
jobs = (xce_job_t**)calloc((size_t)array_size, sizeof(xce_job_t*));
if (jobs == NULL) {
req = (xce_sched_update_request_t*)calloc(1, sizeof(*req));
if (req == NULL) {
return -ENOMEM;
}
for (i = 0; i < array_size; i++) {
uint64_t jobid = base_jobid + i;
req->op = XCE_SCHED_OP_UPDATE_JOB_ARRAY;
req->u.jobarrayupdate.job_array = job_array;
/* Req 6.5/8.4: deduplicate a redelivered push by element identity --
* reuse the SAME jobid-keyed exists-check (sched_get_job against
* sched->jobs) the ordinary single-job path relies on downstream
* (sched_add_job -> key_add -> -EEXIST), applied here per-element so
* an already-materialized element is never re-queued. Not a new
* dedup mechanism.
*
* bug-1714 follow-up (Bug B): this check has a TOCTOU race window,
* and is a best-effort optimization for the common case, NOT an
* atomic correctness boundary. This function runs on whatever
* thread handles the incoming network request; a job is only
* actually inserted into sched->core.jobs later, when the separate
* DRIVER thread drains the pending-update queue this function
* enqueues into (via sched_on_jobs()/enq_pend_update()) and calls
* xce_core_push_events() -> the JOBS_PUSH case (core_events.c) ->
* sched_add_job(). Between this function enqueueing a batch and the
* driver thread actually processing it, sched_get_job() cannot see
* those jobids yet -- so a redelivered (e.g. network-retried) copy
* of the same array push arriving on the request thread during that
* window is NOT caught here, and a second, independent set of
* xce_job_t/job_submit_data_t clones for the same jobids gets
* constructed and enqueued too. This is safe, not just tolerated:
* the redelivered duplicate still gets fully constructed and pushed
* through the pipeline, but is correctly and safely rejected at the
* ACTUAL authoritative uniqueness boundary -- sched_add_job()'s
* key_add() in core.c, whose failure return is now checked by
* core_events.c's JOBS_PUSH handler (Bug A fix) -- which frees the
* orphaned duplicate and never hands it to the picker. Do not treat
* this early check as a correctness guarantee, and do not "fix" the
* race here (e.g. via a new synchronized pending-jobids set) without
* first re-deriving why that's unnecessary now that Bug A closes the
* actual hazard. */
if (sched_get_job(&(sched->core), jobid) != NULL) {
continue;
}
enq_pend_update(sched, req);
jobs[n_new] = make_array_element(job_array, jobid);
if (jobs[n_new] == NULL) {
int j;
for (j = 0; j < n_new; j++) {
free_Job_ptr(&jobs[j]);
}
free(jobs);
return -ENOMEM;
}
n_new++;
}
if (n_new == 0) {
/* Every index already exists: a fully-duplicate redelivery. Nothing
* new to schedule, and nothing to confirm -- Req 8.4 ties the
* confirmation log to a successful *expansion*; a redelivery that
* expands zero new elements is a no-op, not a fresh expansion
* (task 5.3 Observable note). */
free(jobs);
return 0;
}
fmt_jobid(jobidString, sizeof(jobidString) - 1, base_jobid);
INFO("job array push received and expanded: base_jobid=%s "
"array_size=%llu new_tasks=%d",
jobidString, (unsigned long long)array_size, n_new);
rc = sched_on_jobs(sched, n_new, jobs);
if (rc < 0) {
for (i = 0; i < (uint64_t)n_new; i++) {
free_Job_ptr(&jobs[i]);
}
free(jobs);
}
/* On success sched_on_jobs()/the pend-update killer now owns jobs[] and
* every element in it (mirrors the existing XceData_data_schedjobs case
* below, which hands dataset->rows to sched_on_jobs the same way). */
return rc;
return 0;
}
static int sched_on_workers(xce_scheduler_t* sched, int nworkers,
@ -388,15 +191,20 @@ static int sched_handle_dataset(xce_scheduler_t* sched, dataset_t* dataset)
break;
case XceData_data_jobArray:
rc = sched_on_job_array(sched, (xce_job_submit_t*)dataset->rows);
/* Unlike the list-shaped cases above, we never alias dataset->rows
* into the pend-update queue -- every array element is its own
* independent clone (see make_array_element/clone_job_submit_data
* above) -- so the original (jobid, arraySize, submit) descriptor
* is never consumed downstream. free_Dataset()/free_Dataset_ptr()
* are a no-op for every table type, so this is the one call site
* responsible for releasing it, on both the success and error
* path. */
free_JobSubmit_ptr((xce_job_submit_t**)&dataset->rows);
/* REVISED (connection-loss-under-burst fix): sched_on_job_array()
* now enqueues the raw descriptor itself (XCE_SCHED_OP_UPDATE_JOB_
* ARRAY) instead of expanding it inline -- on success, ownership of
* dataset->rows transfers to that queued request (freed later by
* driver.c's release_translated_payload(), same as every other
* dataq-owned payload). free_Dataset()/free_Dataset_ptr() are a
* no-op for every table type, so this is still the one call site
* responsible for releasing it on the FAILURE path (-EINVAL/-ENOMEM,
* ownership never transferred). */
if (rc >= 0) {
dataset->rows = NULL;
} else {
free_JobSubmit_ptr((xce_job_submit_t**)&dataset->rows);
}
dataset->nrows = 0;
break;
}

187
src/scheduler/testsuite/test_driver_translate.c
View file

@ -353,6 +353,29 @@ static xce_job_t* make_wire_job(uint64_t jobid, uint32_t state,
return j;
}
/* Compact job-array descriptor, same shape init_job_array() (test_job_
* array_expand.c) builds -- heap-owned strings, matching what a real
* XCE_SCHED_OP_UPDATE_JOB_ARRAY request now carries unexpanded (connection-
* loss-under-burst fix). */
static xce_job_submit_t* make_wire_job_array(uint64_t base_jobid,
uint64_t array_size)
{
job_submit_data_t* submit = (job_submit_data_t*)calloc(1, sizeof(*submit));
xce_job_submit_t* ja = (xce_job_submit_t*)calloc(1, sizeof(*ja));
submit->queue = strdup("normal");
submit->user = strdup("tester");
submit->mintasks = 1;
submit->maxtasks = 1;
submit->resreq = strdup("type == local");
ja->jobid = base_jobid;
ja->array_size = array_size;
ja->submit = submit;
return ja;
}
/* ---------------------------------------------------------------------- */
/* Per-op unit tests of translate_one_pending_update() (white-box static */
/* call) -- one per op type, asserting event type + every field matches. */
@ -633,6 +656,132 @@ static void test_translate_job_rusage_null_rusages_skips(void)
TEST_ASSERT_EQUAL_INT(0, rc);
}
/* Connection-loss-under-burst fix (XCE_SCHED_OP_UPDATE_JOB_ARRAY, moved
* from handler.c's former sched_on_job_array()): a compact Data.jobArray
* push expands into N ordinary schedulable tasks via base_jobid + i
* arithmetic, HERE on the round-processing thread instead of the
* request-handling thread. */
static void test_translate_job_array_expands_into_n_events(void)
{
xce_sched_core_t core;
xce_job_submit_t* ja = make_wire_job_array(9000, 3);
xce_sched_update_request_t* req = make_req(XCE_SCHED_OP_UPDATE_JOB_ARRAY);
xce_core_event_t ev;
int rc;
fixture_init(&core);
req->u.jobarrayupdate.job_array = ja;
rc = translate_one_pending_update(&core, req, 42, &ev);
TEST_ASSERT_EQUAL_INT(1, rc);
TEST_ASSERT_EQUAL_INT(XCE_CORE_EVENT_JOBS_PUSH, ev.type);
TEST_ASSERT_EQUAL_UINT64(42, ev.ts);
TEST_ASSERT_EQUAL_INT(3, ev.u.jobs_push.njobs);
TEST_ASSERT_EQUAL_UINT64(9000, ev.u.jobs_push.jobs[0]->jobid);
TEST_ASSERT_EQUAL_UINT64(9001, ev.u.jobs_push.jobs[1]->jobid);
TEST_ASSERT_EQUAL_UINT64(9002, ev.u.jobs_push.jobs[2]->jobid);
TEST_ASSERT_EQUAL_STRING("normal", ev.u.jobs_push.jobs[0]->submit->queue);
TEST_ASSERT_EQUAL_STRING("normal", ev.u.jobs_push.jobs[2]->submit->queue);
/* every element gets its OWN independent clone, not a shared pointer --
* required so each is independently free_Job_ptr()-able without a
* double-free. */
TEST_ASSERT_TRUE(ev.u.jobs_push.jobs[0]->submit
!= ev.u.jobs_push.jobs[1]->submit);
TEST_ASSERT_TRUE(ev.u.jobs_push.jobs[1]->submit
!= ev.u.jobs_push.jobs[2]->submit);
/* Expansion result stashed back on req (same pointers as ev.u.jobs_push
* above) so release_translated_payload() can free it later -- see that
* function's own new case. */
TEST_ASSERT_EQUAL_INT(3, req->u.jobarrayupdate.expanded_njobs);
TEST_ASSERT_EQUAL_PTR(ev.u.jobs_push.jobs,
req->u.jobarrayupdate.expanded_jobs);
}
/* Req 6.5/8.4 (moved from handler.c, same rationale): deduplicate a
* redelivered push by element identity -- only genuinely-new indices are
* expanded. Seeds 2 of 3 indices into the REAL core->jobs kvmgr this file's
* fixture_init() sets up (via the real xce_sched_core_init()), simulating
* "already materialized from an earlier round". */
static void test_translate_job_array_skips_already_existing_elements(void)
{
xce_sched_core_t core;
xce_job_submit_t* ja = make_wire_job_array(9100, 3);
xce_sched_update_request_t* req = make_req(XCE_SCHED_OP_UPDATE_JOB_ARRAY);
xce_core_event_t ev;
xce_sched_job_t existing0;
xce_sched_job_t existing1;
int rc;
fixture_init(&core);
req->u.jobarrayupdate.job_array = ja;
memset(&existing0, 0, sizeof(existing0));
existing0.jobid = 9100;
sched_add_job(&core, &existing0);
memset(&existing1, 0, sizeof(existing1));
existing1.jobid = 9101;
sched_add_job(&core, &existing1);
rc = translate_one_pending_update(&core, req, 1, &ev);
TEST_ASSERT_EQUAL_INT(1, rc);
TEST_ASSERT_EQUAL_INT(XCE_CORE_EVENT_JOBS_PUSH, ev.type);
TEST_ASSERT_EQUAL_INT(1, ev.u.jobs_push.njobs);
TEST_ASSERT_EQUAL_UINT64(9102, ev.u.jobs_push.jobs[0]->jobid);
}
/* Every index already exists: a fully-duplicate redelivery -- nothing new
* to schedule, translate reports "nothing to push" (rc=0), same as any
* other skip case, not an error. */
static void test_translate_job_array_all_duplicates_returns_zero_noop(void)
{
xce_sched_core_t core;
xce_job_submit_t* ja = make_wire_job_array(9200, 2);
xce_sched_update_request_t* req = make_req(XCE_SCHED_OP_UPDATE_JOB_ARRAY);
xce_core_event_t ev;
xce_sched_job_t existing0;
xce_sched_job_t existing1;
int rc;
fixture_init(&core);
req->u.jobarrayupdate.job_array = ja;
memset(&existing0, 0, sizeof(existing0));
existing0.jobid = 9200;
sched_add_job(&core, &existing0);
memset(&existing1, 0, sizeof(existing1));
existing1.jobid = 9201;
sched_add_job(&core, &existing1);
rc = translate_one_pending_update(&core, req, 1, &ev);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_INT(0, req->u.jobarrayupdate.expanded_njobs);
TEST_ASSERT_NULL(req->u.jobarrayupdate.expanded_jobs);
}
/* A never-decoded/malformed descriptor (NULL job_array, the enqueue side
* should never produce this, guarded defensively here anyway, same
* discipline every other case in this function follows) skips cleanly. */
static void test_translate_job_array_null_payload_returns_zero(void)
{
xce_sched_core_t core;
xce_sched_update_request_t* req = make_req(XCE_SCHED_OP_UPDATE_JOB_ARRAY);
xce_core_event_t ev;
int rc;
fixture_init(&core);
req->u.jobarrayupdate.job_array = NULL;
rc = translate_one_pending_update(&core, req, 1, &ev);
TEST_ASSERT_EQUAL_INT(0, rc);
}
static void test_translate_unknown_op_skips(void)
{
xce_sched_core_t core;
@ -854,6 +1003,39 @@ static void test_drain_translates_and_pushes_all_ten_op_types(void)
TEST_ASSERT_EQUAL_PTR(config, core.config);
}
/* Connection-loss-under-burst fix, full drain-push-release cycle: proves
* XCE_SCHED_OP_UPDATE_JOB_ARRAY's ownership handling is correct end to end
* under a REAL ASAN-instrumented run -- both the original job_array
* descriptor (release_translated_payload()'s unconditional free) and the N
* freshly-expanded clones (adopted by core_build_job(), so never
* double-freed) survive the real xce_sched_driver_translate_pending_events()
* -> xce_core_push_events() -> release_translated_payload() sequence
* cleanly -- the same proof test_drain_translates_and_pushes_all_ten_op_types
* gives every other op type, dedicated here since job-array's cleanup path
* (two distinct owned allocations per request, not one) is unique to it. */
static void test_drain_translates_and_pushes_job_array_op(void)
{
xce_sched_core_t core;
xce_job_submit_t* ja;
xce_sched_update_request_t* req;
int rc;
fixture_init(&core);
ja = make_wire_job_array(9300, 3);
req = make_req(XCE_SCHED_OP_UPDATE_JOB_ARRAY);
req->u.jobarrayupdate.job_array = ja;
enqueue(&core, req);
rc = xce_sched_driver_translate_pending_events(&core, NULL);
TEST_ASSERT_EQUAL_INT(1, rc);
TEST_ASSERT_NOT_NULL(sched_get_job(&core, 9300));
TEST_ASSERT_NOT_NULL(sched_get_job(&core, 9301));
TEST_ASSERT_NOT_NULL(sched_get_job(&core, 9302));
TEST_ASSERT_EQUAL_UINT32(JOB_STAT_PEND, sched_get_job(&core, 9300)->state);
}
/* ---------------------------------------------------------------------- */
/* Buffering discipline (Req 5 AC 8): events accumulate between drains; */
/* an empty drain is a valid no-op returning 0, not an error. */
@ -932,8 +1114,13 @@ int main(void)
RUN_TEST(test_translate_resvals_delta);
RUN_TEST(test_translate_job_rusage);
RUN_TEST(test_translate_job_rusage_null_rusages_skips);
RUN_TEST(test_translate_job_array_expands_into_n_events);
RUN_TEST(test_translate_job_array_skips_already_existing_elements);
RUN_TEST(test_translate_job_array_all_duplicates_returns_zero_noop);
RUN_TEST(test_translate_job_array_null_payload_returns_zero);
RUN_TEST(test_translate_unknown_op_skips);
RUN_TEST(test_drain_translates_and_pushes_all_ten_op_types);
RUN_TEST(test_drain_translates_and_pushes_job_array_op);
RUN_TEST(test_accumulates_between_drains_and_empty_drain_is_noop);
RUN_TEST(test_null_core_returns_einval);
RUN_TEST(test_uninitialized_pend_updates_returns_einval);

182
src/scheduler/testsuite/test_job_array_expand.c
View file

@ -3,12 +3,22 @@
* job-array push receive path (job-array-single-wal-record task 5.3, Req
* 6.3, 6.4, 6.5, 8.4).
*
* REVISED (connection-loss-under-burst fix, see XCE_SCHED_OP_UPDATE_JOB_
* ARRAY's own comment, scheduler.h): sched_on_job_array() no longer expands
* the compact descriptor into N tasks itself -- it just takes ownership and
* enqueues it as a single XCE_SCHED_OP_UPDATE_JOB_ARRAY request. The actual
* per-element expansion (and its dedup check) now runs in driver.c's
* translate_one_pending_update(), covered by test_driver_translate.c's own
* UPDATE_JOB_ARRAY tests. This file now covers only: (a) sched_on_job_array()
* /sched_handle_dataset()'s own input validation and ownership-transfer
* bookkeeping, and (b) that the wire-decode -> handler -> queue chain
* delivers the descriptor's fields correctly, unmodified.
*
* Strategy: pull the real (static) sched_handle_dataset / sched_on_job_array
* into this TU via #include "../src/handler.c" (same white-box pattern as
* test_handlers.c), driven against a REAL sched->jobs kvmgr so the
* per-element dedup check (sched_get_job) is exercised for real rather than
* faked. sched_add_job/sched_get_job below are the SAME kvmgr key_add/
* key_find path production uses (mirrors test_job_update.c's precedent).
* test_handlers.c). sched_add_job/sched_get_job stubs below are unused by
* anything this file still exercises (kept, harmless, for parity with other
* handler.c white-box test files that DO need them).
*/
#define UNIT_TEST 1
#include <errno.h>
@ -94,13 +104,6 @@ static void fixture_init(xce_scheduler_t* sched)
sched->core.jobs = makeInstance(&xkvmgr_type, &jobs_params);
}
static void seed_existing_job(xce_scheduler_t* sched, uint64_t jobid)
{
xce_sched_job_t* job = (xce_sched_job_t*)calloc(1, sizeof(*job));
job->jobid = jobid;
sched_add_job(&(sched->core), job);
}
/* Builds a heap-allocated, heap-owned-strings (jobArray, submit) pair -- the
* same shape decode_JobSubmit_ptr() produces on the real wire-decode path
* (controller.capnp.c), not a stack/string-literal stand-in. Task 7.3 made
@ -128,41 +131,42 @@ static xce_job_submit_t* init_job_array(uint64_t base_jobid,
return ja;
}
/* Pop one pending update request, asserting it is an UPDATE_JOB op; hands
* back njobs/jobs (caller now owns them -- free via free_Job_ptr + free() to
* mirror sched_pend_update_killer). Returns -1 if the queue is empty. */
static int pop_job_update(xce_scheduler_t* sched, xce_job_t*** jobs_out)
/* Pop one pending update request, asserting it is an UPDATE_JOB_ARRAY op;
* hands back the raw job_array descriptor (caller now owns it -- free via
* free_JobSubmit_ptr to mirror release_translated_payload()'s own cleanup on
* the failure/never-translated path). Returns NULL if the queue is empty. */
static xce_job_submit_t* pop_job_array_update(xce_scheduler_t* sched)
{
data_item_t* it = dataq_deq(sched->core.pend_updates[0], NULL);
xce_sched_update_request_t* req;
int njobs;
xce_job_submit_t* job_array;
if (it == NULL) {
return -1;
return NULL;
}
req = ITPTR_TO_UPDATE_REQ(it);
TEST_ASSERT_EQUAL_INT(XCE_SCHED_OP_UPDATE_JOB, req->op);
TEST_ASSERT_EQUAL_INT(XCE_SCHED_OP_UPDATE_JOB_ARRAY, req->op);
njobs = req->u.jobupdate.njobs;
if (jobs_out != NULL) {
*jobs_out = req->u.jobupdate.jobs;
}
job_array = req->u.jobarrayupdate.job_array;
TEST_ASSERT_EQUAL_INT(0, req->u.jobarrayupdate.expanded_njobs);
TEST_ASSERT_NULL(req->u.jobarrayupdate.expanded_jobs);
free(req);
return njobs;
return job_array;
}
/* ── Req 6.3/6.4: a compact Data.jobArray push expands locally into N
* ordinary schedulable tasks via base_jobid + i arithmetic ── */
/* ── Connection-loss-under-burst fix: a compact Data.jobArray push is now
* queued UNEXPANDED -- ownership of the raw descriptor transfers to the
* XCE_SCHED_OP_UPDATE_JOB_ARRAY request as-is, no per-element work happens
* on this (request-handling) thread. Expansion into N tasks is
* test_driver_translate.c's concern now (translate_one_pending_update()). ── */
static void test_job_array_push_expands_into_n_tasks(void)
static void test_job_array_push_enqueues_raw_descriptor_unexpanded(void)
{
xce_scheduler_t sched;
xce_job_submit_t* ja;
dataset_t dataset;
xce_job_t** jobs = NULL;
int njobs;
int i;
xce_job_submit_t* popped;
fixture_init(&sched);
ja = init_job_array(5000, 3);
@ -171,71 +175,19 @@ static void test_job_array_push_expands_into_n_tasks(void)
dataset.table = XceData_data_jobArray;
dataset.rows = (void*)ja;
/* sched_handle_dataset() owns and frees dataset.rows on return (task
* 7.3) -- do not touch ja/dataset.rows again after this call. */
TEST_ASSERT_TRUE(sched_handle_dataset(&sched, &dataset) >= 0);
njobs = pop_job_update(&sched, &jobs);
TEST_ASSERT_EQUAL_INT(3, njobs);
TEST_ASSERT_EQUAL_UINT64(5000, jobs[0]->jobid);
TEST_ASSERT_EQUAL_UINT64(5001, jobs[1]->jobid);
TEST_ASSERT_EQUAL_UINT64(5002, jobs[2]->jobid);
TEST_ASSERT_EQUAL_STRING("normal", jobs[0]->submit->queue);
TEST_ASSERT_EQUAL_STRING("normal", jobs[2]->submit->queue);
/* every element gets its OWN independent clone, not a shared pointer --
* required so each is independently free_Job_ptr()-able downstream
* (sched_pend_update_killer) without a double-free. */
TEST_ASSERT_TRUE(jobs[0]->submit != jobs[1]->submit);
TEST_ASSERT_TRUE(jobs[1]->submit != jobs[2]->submit);
/* Ownership transferred to the queued request -- sched_handle_dataset()
* must not have freed it and must have nulled dataset.rows out. */
TEST_ASSERT_NULL(dataset.rows);
for (i = 0; i < njobs; i++) {
free_Job_ptr(&jobs[i]);
}
free(jobs);
}
popped = pop_job_array_update(&sched);
TEST_ASSERT_EQUAL_PTR(ja, popped); /* same descriptor, not a clone */
TEST_ASSERT_EQUAL_UINT64(5000, popped->jobid);
TEST_ASSERT_EQUAL_UINT64(3, popped->array_size);
TEST_ASSERT_EQUAL_STRING("normal", popped->submit->queue);
/* ── Req 6.5/8.4: a redelivered push (e.g. after a scheduler reconnect)
* must not create duplicate tasks -- reuse the jobid-keyed exists-check
* (sched_get_job) the ordinary single-job path relies on ── */
static void test_job_array_redelivery_does_not_duplicate(void)
{
xce_scheduler_t sched;
dataset_t dataset;
xce_job_t** jobs = NULL;
int njobs;
int i;
fixture_init(&sched);
memset(&dataset, 0, sizeof(dataset));
dataset.table = XceData_data_jobArray;
dataset.rows = (void*)init_job_array(6000, 2);
/* first delivery: 2 new tasks enqueued. sched_handle_dataset() owns and
* frees dataset.rows on return (task 7.3). */
TEST_ASSERT_TRUE(sched_handle_dataset(&sched, &dataset) >= 0);
njobs = pop_job_update(&sched, &jobs);
TEST_ASSERT_EQUAL_INT(2, njobs);
for (i = 0; i < njobs; i++) {
/* simulate the async consumer (scheduler.c's sched_update_jobs ->
* sched_add_job, not linked into this white-box TU) materializing
* the element into sched->jobs, so the redelivery's dedup check
* sees them as already existing -- exactly what a real reconnect
* scenario looks like from the handler's point of view. */
seed_existing_job(&sched, jobs[i]->jobid);
free_Job_ptr(&jobs[i]);
}
free(jobs);
/* redelivery of the IDENTICAL push: a genuinely new decode of the
* retransmitted wire message (same jobid/array_size/submit content, but
* its own freshly-allocated descriptor -- exactly what a real reconnect
* redelivers; the first descriptor is already freed by the call above).
* Every index already exists. */
dataset.rows = (void*)init_job_array(6000, 2);
TEST_ASSERT_EQUAL_INT(0, sched_handle_dataset(&sched, &dataset));
TEST_ASSERT_EQUAL_INT(-1, pop_job_update(&sched, NULL)); /* nothing new */
free_JobSubmit_ptr(&popped);
}
/* ── Guard-clause coverage on the decode step itself ── */
@ -279,7 +231,8 @@ static void test_job_array_null_descriptor_returns_einval(void)
TEST_ASSERT_EQUAL_INT(-EINVAL, sched_on_job_array(&sched, NULL));
}
/* ── job-array-single-wal-record task 5.3b: full wire-dispatch chain ──────
/* ── job-array-single-wal-record task 5.3b: full wire-dispatch chain
* (REVISED for the connection-loss-under-burst fix) ──────────────────────
* Every test above feeds a stack-built xce_job_submit_t straight into
* sched_handle_dataset(), bypassing the transport layer entirely -- the same
* white-box convention every sibling handler.c test uses (task 5.3's own
@ -290,10 +243,11 @@ static void test_job_array_null_descriptor_returns_einval(void)
* sends -- OPCode_push / Request_body_pushmsg carrying one Data.jobArray
* descriptor -- then hands extract_data()'s reassembled dataset_t to
* sched_handle_dataset(), proving the sender-payload -> wire-dispatch ->
* receiver-expand chain is intact end to end, not just sched_on_job_array()
* in isolation. */
* receiver-enqueue chain delivers the descriptor's fields correctly, intact
* end to end -- expansion itself is no longer this chain's job (driver.c's
* concern now), so this only proves the queued descriptor. */
static void test_wire_dispatch_chain_delivers_n_tasks(void)
static void test_wire_dispatch_chain_delivers_job_array_descriptor(void)
{
xce_scheduler_t sched;
job_submit_data_t submit;
@ -303,9 +257,7 @@ static void test_wire_dispatch_chain_delivers_n_tasks(void)
xce_request_t decoded;
io_packet_t* pkt;
dataset_t dataset;
xce_job_t** jobs = NULL;
int njobs;
int i;
xce_job_submit_t* popped;
fixture_init(&sched);
@ -350,27 +302,22 @@ static void test_wire_dispatch_chain_delivers_n_tasks(void)
* handler (task 5.3, unmodified) -- proving the two layers compose. */
TEST_ASSERT_TRUE(sched_handle_dataset(&sched, &dataset) >= 0);
/* Assert: N schedulable tasks, correctly addressed via base_jobid + i. */
njobs = pop_job_update(&sched, &jobs);
TEST_ASSERT_EQUAL_INT(4, njobs);
TEST_ASSERT_EQUAL_UINT64(8000, jobs[0]->jobid);
TEST_ASSERT_EQUAL_UINT64(8001, jobs[1]->jobid);
TEST_ASSERT_EQUAL_UINT64(8002, jobs[2]->jobid);
TEST_ASSERT_EQUAL_UINT64(8003, jobs[3]->jobid);
TEST_ASSERT_EQUAL_STRING("normal", jobs[0]->submit->queue);
TEST_ASSERT_EQUAL_STRING("normal", jobs[3]->submit->queue);
for (i = 0; i < njobs; i++) {
free_Job_ptr(&jobs[i]);
}
free(jobs);
/* Assert: the decoded descriptor was queued unexpanded, fields intact,
* correctly addressed via base_jobid + array_size. */
popped = pop_job_array_update(&sched);
TEST_ASSERT_NOT_NULL(popped);
TEST_ASSERT_EQUAL_UINT64(8000, popped->jobid);
TEST_ASSERT_EQUAL_UINT64(4, popped->array_size);
TEST_ASSERT_EQUAL_STRING("normal", popped->submit->queue);
free_JobSubmit_ptr(&popped);
/* Cleanup: extract_data() transferred ownership of the decoded
* xce_job_submit_t into dataset.rows (nulling the source pointer inside
* decoded.body.pushmsg); sched_handle_dataset()'s jobArray case now
* frees dataset.rows itself and nulls it out (task 7.3), so there is
* nothing left for this test to free there. decoded.body.pushmsg is a
* hollow container, safe to free via the normal request teardown. */
* decoded.body.pushmsg); sched_handle_dataset()'s jobArray case
* transfers that ownership again, into the queued request, and nulls
* dataset.rows out -- nothing left for this test to free there.
* decoded.body.pushmsg is a hollow container, safe to free via the
* normal request teardown. */
TEST_ASSERT_NULL(dataset.rows);
xce_free_request(&decoded);
}
@ -380,11 +327,10 @@ int main(int argc, char* argv[])
(void)argc;
(void)argv;
UNITY_BEGIN();
RUN_TEST(test_job_array_push_expands_into_n_tasks, 0);
RUN_TEST(test_job_array_redelivery_does_not_duplicate, 0);
RUN_TEST(test_job_array_push_enqueues_raw_descriptor_unexpanded, 0);
RUN_TEST(test_job_array_null_submit_returns_einval, 0);
RUN_TEST(test_job_array_zero_size_returns_einval, 0);
RUN_TEST(test_job_array_null_descriptor_returns_einval, 0);
RUN_TEST(test_wire_dispatch_chain_delivers_n_tasks, 0);
RUN_TEST(test_wire_dispatch_chain_delivers_job_array_descriptor, 0);
return UNITY_END();
}

221
src/scheduler/testsuite/test_job_array_wire_e2e.c
View file

@ -16,12 +16,23 @@
* -> sched_handle_dataset()
* -> sched_on_job_array() (task 5.3)
*
* ...delivers exactly N correctly-addressed scheduler tasks and fires task
* 5.3's confirmation log line exactly once (Req 6.3, 6.4, 8.4); and that an
* identical byte-for-byte SECOND delivery (independently re-decoded from the
* same encoded bytes -- the wire-level equivalent of "the same push arrives
* twice", e.g. after a reconnect) creates zero new tasks and logs zero
* additional confirmation lines (Req 6.5, 8.4).
* ...delivers the compact descriptor into the pend-updates queue intact and
* unexpanded (Req 6.3, 6.4), correctly addressed by base jobid/array_size.
*
* REVISED (connection-loss-under-burst fix, see XCE_SCHED_OP_UPDATE_JOB_
* ARRAY's own comment, scheduler.h): sched_on_job_array() no longer expands
* the descriptor into N tasks, deduplicates a redelivery, or logs a
* confirmation line itself -- all three now happen in driver.c's
* translate_one_pending_update(), on the round-processing thread, covered by
* test_driver_translate.c's own UPDATE_JOB_ARRAY tests
* (test_translate_job_array_expands_into_n_events,
* test_translate_job_array_skips_already_existing_elements,
* test_translate_job_array_all_duplicates_returns_zero_noop). This file's
* own two tests were revised accordingly: a redelivery of byte-identical
* wire bytes through THIS layer alone is correctly and safely a second,
* independent, unexpanded descriptor -- the wire/handler layer has no
* dedup check of its own to lose; the actual dedup coverage lives in
* test_driver_translate.c now.
*
* Existing coverage this test does NOT duplicate:
* - test_job_array_sched_push.c (task 5.1) already proves the CONTROLLER
@ -29,14 +40,13 @@
* - test_request.c's test_extract_data_job_array_passes_through_intact
* (task 5.3b) already proves extract_data() itself decodes the payload
* correctly via a real encode/decode round trip.
* - test_job_array_expand.c's test_wire_dispatch_chain_delivers_n_tasks
* (task 5.3b) already proves extract_data() -> sched_handle_dataset()
* compose, but stops one layer below the real registered handler and
* never exercises redelivery through the wire path or the log line.
* - test_job_array_expand.c's test_wire_dispatch_chain_delivers_job_array_
* descriptor (task 5.3b) already proves extract_data() ->
* sched_handle_dataset() compose, but stops one layer below the real
* registered handler.
* This test's unique contribution: the REAL top-level dispatcher
* (scheduler_req_handler(), not sched_handle_dataset() directly), the
* redelivery-under-real-wire-bytes case, and the confirmation log line's
* actual emission count.
* (scheduler_req_handler(), not sched_handle_dataset() directly) and the
* redelivery-under-real-wire-bytes case.
*
* vtask_entry() (xcectx.c) is `static` and not exported for testing -- it is
* also the trigger normally driven by a live socket + iomgr epoll loop, which
@ -51,13 +61,11 @@
* Bug-catching demonstration (this task's "RED phase"): temporarily
* reverting extract_data()'s `case XceData_data_jobArray:` (task 5.3b's fix)
* back to falling through to `default: rc = -EINVAL` makes this test's first
* assertion (`callnext != 0` while dataset->rows stays NULL) fail loudly,
* and the subsequent job-count assertions fail too (zero tasks created) --
* assertion (`callnext != 0` while dataset->rows stays NULL) fail loudly --
* confirming this test has teeth against the exact class of bug 5.3b fixed.
*/
#define UNIT_TEST 1
#include <errno.h>
#include <stdarg.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
@ -139,96 +147,28 @@ static void fixture_init(xce_scheduler_t* sched)
sched->core.jobs = makeInstance(&xkvmgr_type, &jobs_params);
}
/* Pop one pending update request, asserting it is an UPDATE_JOB op; hands
* back njobs/jobs (caller now owns them -- free via free_Job_ptr + free() to
* mirror sched_pend_update_killer). Returns -1 if the queue is empty. Mirrors
/* Pop one pending update request, asserting it is an UPDATE_JOB_ARRAY op;
* hands back the raw job_array descriptor (caller now owns it -- free via
* free_JobSubmit_ptr). Returns NULL if the queue is empty. Mirrors
* test_job_array_expand.c's identical helper. */
static int pop_job_update(xce_scheduler_t* sched, xce_job_t*** jobs_out)
static xce_job_submit_t* pop_job_array_update(xce_scheduler_t* sched)
{
data_item_t* it = dataq_deq(sched->core.pend_updates[0], NULL);
xce_sched_update_request_t* req;
int njobs;
xce_sched_update_request_t* req;
xce_job_submit_t* job_array;
if (it == NULL) {
return -1;
return NULL;
}
req = ITPTR_TO_UPDATE_REQ(it);
TEST_ASSERT_EQUAL_INT(XCE_SCHED_OP_UPDATE_JOB, req->op);
TEST_ASSERT_EQUAL_INT(XCE_SCHED_OP_UPDATE_JOB_ARRAY, req->op);
njobs = req->u.jobupdate.njobs;
if (jobs_out != NULL) {
*jobs_out = req->u.jobupdate.jobs;
}
job_array = req->u.jobarrayupdate.job_array;
free(req);
return njobs;
return job_array;
}
/* Materializes a popped job array into sched->jobs (mirroring what the
* async consumer, scheduler.c's sched_update_jobs -> sched_add_job, not
* linked into this white-box TU, would do) -- so a SECOND, redelivered
* push's dedup check sees the first delivery's elements as already
* existing. Frees each popped xce_job_t and the array itself, same
* simulated-reconnect technique test_job_array_expand.c's
* test_job_array_redelivery_does_not_duplicate uses (there via
* seed_existing_job, here folded into one step since njobs is already
* known). */
static void materialize_popped_jobs(xce_scheduler_t* sched, xce_job_t** jobs,
int njobs)
{
int i;
for (i = 0; i < njobs; i++) {
xce_sched_job_t* job = (xce_sched_job_t*)calloc(1, sizeof(*job));
job->jobid = jobs[i]->jobid;
sched_add_job(&(sched->core), job);
free_Job_ptr(&jobs[i]);
}
free(jobs);
}
/* ---------------------------------------------------------------------- */
/* Early-print log capture: sched_on_job_array()'s confirmation line uses */
/* INFO() (tlog_info -> TLOG_INFO). No test binary in this codebase calls */
/* tlog_init(), so every INFO() call routes through tlog's pre-init */
/* "early print" fallback (tlog.c's _tlog_early_print(), confirmed via the */
/* vendored tlog source). tlog_reg_early_printf_callback() lets a test */
/* intercept that fallback synchronously, on the calling thread -- no */
/* tlog_init()/tlog_exit() lifecycle, so it cannot disturb any other test */
/* sharing this process. va_copy() is used so the caller's own va_list is */
/* never consumed, in case tlog's own early-print path also reads it. */
/* ---------------------------------------------------------------------- */
static int g_confirm_log_count = 0;
static void wire_e2e_early_print_cb(struct tlog_loginfo* loginfo,
const char* format, va_list ap)
{
char buf[512];
va_list ap_copy;
if (loginfo == NULL || loginfo->level != TLOG_INFO) {
return;
}
va_copy(ap_copy, ap);
vsnprintf(buf, sizeof(buf), format, ap_copy);
va_end(ap_copy);
if (strstr(buf, "job array push received and expanded") != NULL) {
g_confirm_log_count++;
}
}
static void start_log_capture(void)
{
g_confirm_log_count = 0;
tlog_set_early_printf(1, 1, 0);
tlog_reg_early_printf_callback(wire_e2e_early_print_cb);
}
static void stop_log_capture(void) { tlog_reg_early_printf_callback(NULL); }
/* ---------------------------------------------------------------------- */
/* Real wire construction: the exact message xcesched_push_jobarray() */
/* (sched.c) sends -- OPCode_push / Request_body_pushmsg carrying one */
@ -328,52 +268,48 @@ static int deliver_via_real_dispatch_chain(io_packet_t* pkt,
return rc;
}
/* ── Test 1: one real wire push -> N correctly-addressed tasks, log fires */
/* exactly once (Req 6.3, 6.4, 8.4) */
/* ── Test 1: one real wire push -> the compact descriptor is queued intact */
/* and unexpanded (Req 6.3, 6.4) -- expansion itself is driver.c's concern */
/* now, covered by test_driver_translate.c. */
static void test_one_push_produces_n_tasks_and_logs_once(void)
static void test_one_push_enqueues_job_array_descriptor(void)
{
xce_scheduler_t sched;
job_submit_data_t submit_storage;
io_packet_t* pkt = NULL;
dataset_t* ds = NULL;
xce_job_t** jobs = NULL;
int njobs;
int i;
xce_job_submit_t* popped;
fixture_init(&sched);
start_log_capture();
build_encoded_push(&pkt, 42000, 6, &submit_storage);
TEST_ASSERT_TRUE(
deliver_via_real_dispatch_chain(pkt, &sched, &ds) >= 0);
njobs = pop_job_update(&sched, &jobs);
TEST_ASSERT_EQUAL_INT(6, njobs);
for (i = 0; i < njobs; i++) {
TEST_ASSERT_EQUAL_UINT64(42000 + (uint64_t)i, jobs[i]->jobid);
TEST_ASSERT_EQUAL_STRING("normal", jobs[i]->submit->queue);
free_Job_ptr(&jobs[i]);
}
free(jobs);
popped = pop_job_array_update(&sched);
TEST_ASSERT_NOT_NULL(popped);
TEST_ASSERT_EQUAL_UINT64(42000, popped->jobid);
TEST_ASSERT_EQUAL_UINT64(6, popped->array_size);
TEST_ASSERT_EQUAL_STRING("normal", popped->submit->queue);
free_JobSubmit_ptr(&popped);
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, g_confirm_log_count,
"task 5.3's confirmation log line must fire exactly once for one "
"successful array expansion");
stop_log_capture();
free_JobSubmit_ptr((xce_job_submit_t**)&ds->rows);
free(ds);
printf("test_one_push_produces_n_tasks_and_logs_once: PASS\n");
printf("test_one_push_enqueues_job_array_descriptor: PASS\n");
}
/* ── Test 2: redelivery of the IDENTICAL encoded bytes (independently */
/* re-decoded, simulating a reconnect replay) creates zero new tasks and */
/* zero additional log lines (Req 6.5, 8.4) */
/* re-decoded, simulating a reconnect replay) -- REVISED (connection-loss- */
/* under-burst fix): the wire/handler layer no longer deduplicates at all */
/* (see XCE_SCHED_OP_UPDATE_JOB_ARRAY's own comment, scheduler.h), so a */
/* redelivery through THIS layer alone correctly and safely produces a */
/* second, independent, unexpanded descriptor -- dedup coverage now lives */
/* in test_driver_translate.c's own */
/* test_translate_job_array_skips_already_existing_elements / */
/* test_translate_job_array_all_duplicates_returns_zero_noop (Req 6.5). */
static void test_redelivery_creates_no_new_tasks_and_logs_nothing(void)
static void test_redelivery_enqueues_independent_second_descriptor(void)
{
xce_scheduler_t sched;
job_submit_data_t submit_storage;
@ -381,8 +317,8 @@ static void test_redelivery_creates_no_new_tasks_and_logs_nothing(void)
io_packet_t* pkt2 = NULL;
dataset_t* ds1 = NULL;
dataset_t* ds2 = NULL;
xce_job_t** jobs = NULL;
int njobs;
xce_job_submit_t* popped1;
xce_job_submit_t* popped2;
fixture_init(&sched);
@ -392,34 +328,31 @@ static void test_redelivery_creates_no_new_tasks_and_logs_nothing(void)
build_encoded_push(&pkt1, 43000, 4, &submit_storage);
build_encoded_push(&pkt2, 43000, 4, &submit_storage);
start_log_capture();
/* First delivery: 4 new tasks. */
/* First delivery. */
TEST_ASSERT_TRUE(deliver_via_real_dispatch_chain(pkt1, &sched, &ds1) >= 0);
njobs = pop_job_update(&sched, &jobs);
TEST_ASSERT_EQUAL_INT(4, njobs);
materialize_popped_jobs(&sched, jobs, njobs);
TEST_ASSERT_EQUAL_INT(1, g_confirm_log_count);
popped1 = pop_job_array_update(&sched);
TEST_ASSERT_NOT_NULL(popped1);
TEST_ASSERT_EQUAL_UINT64(43000, popped1->jobid);
TEST_ASSERT_EQUAL_UINT64(4, popped1->array_size);
free_JobSubmit_ptr(&popped1);
free_JobSubmit_ptr((xce_job_submit_t**)&ds1->rows);
free(ds1);
/* Second, byte-identical delivery (simulated redelivery/reconnect):
* every index (43000..43003) already exists in sched->jobs -- must be a
* pure no-op, and must NOT log a second confirmation line. */
TEST_ASSERT_EQUAL_INT(
0, deliver_via_real_dispatch_chain(pkt2, &sched, &ds2));
TEST_ASSERT_EQUAL_INT_MESSAGE(
-1, pop_job_update(&sched, &jobs),
"redelivery must enqueue nothing new");
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, g_confirm_log_count,
"redelivery must not add a second confirmation log line -- still "
"exactly 1 total, from the first delivery only");
/* Second, byte-identical delivery (simulated redelivery/reconnect): a
* second, independent descriptor -- correct and safe by design (the
* downstream dedup check, now in driver.c, runs against core->jobs,
* which this white-box TU never populates from a queued-but-not-yet-
* translated request). */
TEST_ASSERT_TRUE(deliver_via_real_dispatch_chain(pkt2, &sched, &ds2) >= 0);
popped2 = pop_job_array_update(&sched);
TEST_ASSERT_NOT_NULL(popped2);
TEST_ASSERT_EQUAL_UINT64(43000, popped2->jobid);
TEST_ASSERT_EQUAL_UINT64(4, popped2->array_size);
free_JobSubmit_ptr(&popped2);
stop_log_capture();
free_JobSubmit_ptr((xce_job_submit_t**)&ds2->rows);
free(ds2);
printf("test_redelivery_creates_no_new_tasks_and_logs_nothing: PASS\n");
printf("test_redelivery_enqueues_independent_second_descriptor: PASS\n");
}
int main(int argc, char* argv[])
@ -427,7 +360,7 @@ int main(int argc, char* argv[])
(void)argc;
(void)argv;
UNITY_BEGIN();
RUN_TEST(test_one_push_produces_n_tasks_and_logs_once, 0);
RUN_TEST(test_redelivery_creates_no_new_tasks_and_logs_nothing, 0);
RUN_TEST(test_one_push_enqueues_job_array_descriptor, 0);
RUN_TEST(test_redelivery_enqueues_independent_second_descriptor, 0);
return UNITY_END();
}