myxce/.kiro/specs/resreq-maxtasks-wiring/design.md

Overview

This feature closes bug-1770: a job's real requested task count is already correctly threaded from the apiserver all the way to xce_sched_job_t.mintasks/maxtasks, but the parsed resource request the scheduler's allocator dispatch actually reads (xce_primary_resreq_t.minTasks/maxTasks) is unconditionally hardcoded to one task. This makes the scheduler-allocation feature's already-implemented, already-tested, already-soak-validated multi-host allocation logic (parjob_allocator) permanently unreachable from any real job submission today.

Users: cluster users submitting multi-task jobs, and cluster operators relying on existing single-task jobs continuing to behave identically.

Impact: changes the resource request the scheduler dispatches on from an always-1 constant to a real, per-job derived value for standalone primary resource requests, at every point a resreq is parsed from a job's submitted or persisted state. No existing component's ownership, wire format, or public interface changes shape — this is a targeted derivation fix plus the validation needed to trust it.

Goals

• A standalone primary resource request's minTasks/maxTasks reflect the job's real requested task count, at fresh ingest and at snapshot restore, and remain correct across a subsequent resource-request update (a guarantee this feature relies on an existing structural property for, rather than actively enforcing — see Existing Architecture Analysis).
• A job requiring more tasks than one host can provide is genuinely evaluated for, and where appropriate allocated across, multiple hosts, through the real scheduler dispatch path.
• A committed multi-host allocation is proven, end to end, to actually launch and run on real (or virtual) workers.
• Zero behavioral change for any job with a requested task count of one, and zero behavioral change for compound/alternative resource requests.

Non-Goals

• Implementing span[ptile=N] resreq-grammar semantics (remains parsed and discarded, unchanged).
• Wiring task count into primary resource requests nested inside a compound/alternative resource request (bug-1768 already makes their resource checking meaningless regardless of task count).
• Fixing the snapshot-restore resreq_fp computation gap (bug-1772, tracked separately) or its sibling in make_sched_job() (bug-1774) — the latter is dead code today (confirmed: make_sched_job()'s entire call chain, via sched_update_jobs()/sched_update(), has zero live callers anywhere in this codebase), so this feature does not wire it at all, not merely defer fixing it.
• Fixing a job's resource-request update never taking effect for scheduling purposes for any field (bug-1773) — this feature's own task-count guarantee across an update holds for that same structural reason, not because this feature actively re-wires anything on update.
• Modifying the controller's or worker's existing multi-host launch code (mk_launchinfo(), job_start_remote_task(), create_mpi_cmdline()) — this design depends on and validates that code, but does not change it.
• Fixing the pre-existing per-host live memory-usage reconciliation limitation for multi-host jobs.

Boundary Commitments

This Spec Owns

• The derivation that sets a standalone primary resource request's minTasks/maxTasks from the job's real requested task count, at every live call site whose parsed resreq actually reaches scheduler dispatch.
• The defensive invariant that an invalid (minTasks > maxTasks) pair cannot reach allocator dispatch, regardless of upstream validation.
• A new automated end-to-end test proving a real multi-task job, submitted through the normal submission path, is genuinely allocated across multiple hosts by the scheduler and actually runs to completion.

Out of Boundary

• span[...] resreq-grammar syntax (parsed, discarded, unchanged).
• Primary resource requests nested inside a compound/alternative resource request (bug-1768).
• The snapshot-restore resreq_fp gap (bug-1772).
• Per-host live memory-usage reconciliation for multi-host jobs (pre-existing, documented limitation).
• The scheduler's plan-generation/allocation algorithm itself (owned by scheduler-allocation; this spec only makes it reachable).
• The controller/worker multi-host launch mechanism's internal implementation (pre-existing; this spec validates it, does not modify it).

Allowed Dependencies

• scheduler-allocation's parjob_allocator/seqjob_allocator/alloc_cache — consumed as-is; this spec must not modify their dispatch logic.
• The apiserver's existing checkNumProcessors() validation and backend/convert.go's mintasks/maxtasks clamp — relied upon as the primary validation layer; this spec adds a redundant defensive check, not a replacement.
• The controller/worker's existing multi-host launch code (mk_launchinfo(), job_start_remote_task(), create_mpi_cmdline()) — read-only dependency for the new end-to-end test.

Revalidation Triggers

• Any change to parse_resreq_primary()'s or parse_resreq()'s signature, or to xce_primary_resreq_t's struct shape.
• Any change to how a resreq's top-level tag distinguishes a standalone primary from a COMPOUND/ALTER tree.
• Any change to job_submit_data_t.mintasks/maxtasks or its Go-side validation (checkNumProcessors(), convert.go's clamp).
• Any change to scheduler-allocation's dispatch gate (sched_make_primary_alloc_()'s maxTasks == 1 branch) or to bug-1768's compound-nested resource-checking gap, since a future fix there would change what "out of boundary" means for compound resource requests.

Architecture

Existing Architecture Analysis

• parse_resreq_primary(xce_primary_resreq_t*, char*, var, char**) and parse_resreq(xce_resreq_t*, char*, var, char**) (src/lib/resreq.c) take no job-context parameter — they cannot see a job's requested task count. Six existing test call sites invoke the bare production signature directly with no job object in scope, ruling out a signature change as the fix mechanism.
• Five call sites in total reach this parser in the source tree, but only two are reachable in production today. The scheduler's actual dispatch reads job->schedobj->u.primary->resreq (scheduler.c:315), a pointer set exactly once, by whichever function constructs job->schedobj. Exactly three functions ever do this (job->schedobj = grepped exhaustively): core_build_job() (core_events.c), make_sched_job() (schedobj.c), and the snapshot-restore path (snapshot_load.c). Of these, make_sched_job() is dead code: its only caller, sched_update_jobs(), is only called from inside sched_update()'s switch, and sched_update() (scheduler.c) has zero live callers anywhere in this codebase — confirmed by exhaustive grep and corroborated by the codebase's own pre-existing comments describing it as dead/superseded, and by the identical, already-fixed sibling case in bug-1740. core_build_job() (fresh submission, via the driver-translated XCE_CORE_EVENT_JOBS_PUSH path) and the snapshot_load.c restore path are the two genuinely live schedobj-construction sites, and are this spec's only implementation targets.
• The remaining two of the five parser call sites are job-update handlers (sched_update_existing_job() in schedobj.c, the JOB_UPDATE handler in core_events.c) that replace job->resreq but never rebuild job->schedobj. sched_update_existing_job() is itself dead code (its sole caller is also inside sched_update()'s dead switch). core_events.c's JOB_UPDATE handler IS live (reached via the driver-translated path, confirmed by tracing main.c → driver.c's event translation → xce_core_push_events()) — but it never affects scheduling regardless, since job->schedobj isn't rebuilt there either, and no live feature in this codebase today ever actually changes an existing job's submit->resreq (grepped the whole controller: nothing mutates it post-submission). Applying this feature's helper at either update call site would be dead code with zero observable effect. Requirement 1.4's guarantee (task count unaffected by an update) already holds today for this structural reason, independent of this feature. The update path never taking effect for scheduling at all, for any field, is a real, separately-tracked, pre-existing defect (bug-1773).
• "Snapshot restore" (Requirement 1.3/1.4's snapshot_load.c site) is live only via the simulation/replay driver, not live-cluster crash recovery. xce_sched_core_load_snapshot()'s only real caller anywhere in src/ is sim.c's sim_locate_and_load_snapshot(), reachable only through main.c's --sim FILE offline-replay flag; normal-mode main.c has no snapshot-load call or flag at all. A live scheduler restart (including post-failover, e.g. run-ha-soak.sh's own scheduler-restart step) starts with empty in-memory state and rejoins via on_scheduler_join() → the same live core_build_job() path fresh ingest uses — it never touches snapshot_load.c. This means the HA soak (Requirement 5.2) provides zero exercise of this call site; Requirement 5.6 validates it separately. The fix is still worth wiring here — --sim mode drives the identical parjob_allocator/seqjob_allocator dispatch per this project's own driver-equivalence invariant, so a saved snapshot's task count needs to be correct for simulation/tuning fidelity — but this is a genuinely different (and narrower) liveness scope than "post-restart/post-failover" phrasing might suggest, and bug-1772's own description has been corrected accordingly.
• A single job's resreq parse tree can contain more than one primary when COMPOUND/ALTER syntax is used; parse_resreq_primary() already runs once per primary node in the tree, uniformly. Only the job's own top-level standalone primary (tag XCE_RESREQ_PRIMARY) is in this spec's boundary — a nested primary inside COMPOUND/ALTER must be left exactly as today.
• job_submit_data_t.mintasks/maxtasks (the job-level source of truth) is already correctly threaded end-to-end from the apiserver's num_processors/maxnum_processors fields, validated twice (checkNumProcessors(), convert.go's clamp) before reaching the C layer, and already correctly round-tripped through the WAL/snapshot wire format. This spec's boundary starts exactly where that correct value stops propagating.

Project Invariant Cross-Check

core_events.c (home of core_build_job(), one of this spec's real call sites) is part of sched_core_isolation_lib — the "one core, two consumers" layer shared between the real driver (driver.c) and the simulation driver (sim_loop.c), per this project's own scheduler-allocation task 5.2 lesson (a fix applied only to the real-driver path there caused a real driver/simulation divergence that corrupted xce_sched_sim_driver_t state). Confirmed, not merely assumed: driver.c and sim_loop.c both build XCE_CORE_EVENT_JOBS_PUSH events and call the same shared xce_core_push_events() (core_events.c), whose JOBS_PUSH case calls core_build_job() directly — no parallel, sim-specific job constructor exists anywhere. Wiring the fix in core_build_job() therefore applies identically to real and simulated job ingest with no divergence risk. test_maxtasks_wiring_ingest.c still includes a regression test exercising this path through sim_loop.c's own entry point, to keep this confirmed property under test rather than resting on a one-time design-time check.

Architecture Pattern & Boundary Map

graph TB
    ApiServer[Apiserver job submission] --> SubmitData[job_submit_data_t mintasks maxtasks]
    SubmitData --> Ingest[Scheduler ingest call sites]
    Ingest --> ParseResreq[parse_resreq existing parser]
    ParseResreq --> Helper[apply_job_task_count new helper]
    Helper --> Primary[xce_primary_resreq_t minTasks maxTasks]
    Primary --> Dispatch[Scheduler dispatch existing]
    Dispatch --> SeqJob[seqjob_allocator existing]
    Dispatch --> ParJob[parjob_allocator existing]
    ParJob --> Commit[Multi host allocation committed]
    Commit --> Controller[Controller mk_launchinfo existing]
    Controller --> Worker[Worker job_start_remote_task existing]
    Worker --> Running[Job reaches RUN then DONE]

Architecture Integration:

• Selected pattern: a single new helper function, invoked by each of the two ingest call sites whose result actually reaches dispatch (core_build_job() and the snapshot-restore path), immediately after its existing parse_resreq()/parse_resreq_primary() call returns.
• Domain boundaries: src/lib/resreq.c owns the new helper and its tag-gating/clamp logic; each scheduler-side call site owns invoking it with whichever job-context task count is already in scope there.
• Existing patterns preserved: this mirrors the codebase's own existing idiom of copying a field from submit-> onto a target struct immediately after that struct is built — already done identically for job->mintasks/maxtasks itself at both of these sites.
• New component rationale: one shared helper avoids duplicating the COMPOUND/ALTER tag-check twice and gives one independently unit-testable unit for the only real logic this fix introduces.
• Steering compliance: no new WAL/snapshot field is introduced (see Data Models); resreq.c is a shared C library untouched by the scheduler-core isolation boundary; the fix does not alter scheduler-allocation's dispatch or allocation logic, only what reaches it.

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
Shared C library	Existing C (gnu17), src/lib/resreq.c	New apply_job_task_count() helper	No new dependency
Scheduler (C)	Existing C, src/scheduler/src/	Two live ingest call-site wiring	core_events.c, snapshot_load.c
Test infrastructure	Existing xce_test debug CLI + xcecd_virtual	New end-to-end test	Extends the pattern already used by xce-test-resources/xce-test-ha

File Structure Plan

Modified Files

• src/lib/resreq.c — add apply_job_task_count(); parse_resreq_primary()'s existing minTasks = 1; maxTasks = 1; hardcode is left in place as the correct default for a freshly-parsed resreq before the helper is applied, and as the permanent behavior for nested COMPOUND/ALTER primaries the helper deliberately skips.
• src/include/resreq.h — declare apply_job_task_count().
• src/scheduler/src/core_events.c — call the helper after core_build_job()'s parse_resreq() call, using j->submit->mintasks/maxtasks. core_events.c's JOB_UPDATE handler is NOT modified — it's live, but job->schedobj (what dispatch actually reads) is never rebuilt by an update, so applying the helper to the freshly-parsed job->resreq there would be dead code (see Existing Architecture Analysis and bug-1773).
• src/scheduler/src/snapshot_load.c — call the helper after the restore path's parse_resreq() call, using job->mintasks/maxtasks (already restored earlier in the same function, before this call).
• src/scheduler/src/schedobj.c is NOT modified — make_sched_job() and sched_update_existing_job() are both dead code (confirmed: unreachable via sched_update(), which has zero live callers), so wiring either would have no production effect.
• src/scheduler/testsuite/test_parjob_e2e_multislot.c — remove the PATCH_MAXTASKS_FOR_BUG_1770 workaround now that real ingest correctly wires maxTasks, converting it to drive the same scenario through unmodified real ingest.

New Files

• src/lib/testsuite/test_resreq_task_count.c — unit tests for apply_job_task_count() in isolation.
• src/scheduler/testsuite/test_maxtasks_wiring_ingest.c — white-box tests confirming each of the two wired ingest call sites applies the wiring correctly (including the fresh-ingest-vs-snapshot-restore comparative assertion for Requirement 5.6), plus the structural regression test for Requirement 1.4.
• src/worker/testsuite/test_worker_launch_type_dispatch.c — white-box test targeting job_start_remote_task()'s launch-type gate (controller_connector.c) first: confirms it correctly no-ops (defers to MPI's own remote-launch mechanism) for an MPI launch type, and correctly fans the job out via XCE's own per-host dispatch for the default launch type — this is the actual decision point Requirement 4.4 depends on, and today has zero test coverage of any kind. Secondarily confirms job_start_local_task()'s branch into create_mpi_cmdline() builds the right host list and command. Neither requires a real MPI runtime installed in the test environment (Requirement 4.4).
• .claude/skills/xce-test-multihost/ — new skill (SKILL.md + driver script), following xce-test-resources's single-controller-node pattern: deploys one controller/scheduler plus two named xcecd_virtual workers, submits a real multi-task job via jobsub, and verifies genuine multi-host RUN-then-DONE execution on both hosts, plus one array-job variant.

System Flows

sequenceDiagram
    participant User
    participant Api as Apiserver
    participant Sched as Scheduler ingest
    participant Alloc as Scheduler dispatch
    participant Ctrl as Controller
    participant W1 as Worker host A
    participant W2 as Worker host B

    User->>Api: submit job with -n 4,6
    Api->>Api: checkNumProcessors validate
    Api->>Sched: job_submit_data_t mintasks maxtasks
    Sched->>Sched: parse_resreq existing
    Sched->>Sched: apply_job_task_count new
    Sched->>Alloc: standalone primary minTasks maxTasks real value
    Alloc->>Alloc: maxTasks greater than one route to parjob_allocator
    Alloc->>Ctrl: commit multi host allocation
    Ctrl->>W1: launch info host A share
    Ctrl->>W2: launch info host B share
    W1-->>Ctrl: task started
    W2-->>Ctrl: task started
    Ctrl-->>Sched: job running
    Sched-->>User: job reaches DONE

Key decision visible in the diagram: apply_job_task_count() sits strictly between the existing parser and the existing dispatch gate — nothing downstream of it (allocator, controller, worker) changes; nothing upstream of it (apiserver validation, job_submit_data_t wiring) changes.

Requirements Traceability

Requirement	Summary	Components	Interfaces	Flows
1.1	Real task count reaches the standalone primary	TaskCountWiring, IngestTaskCountWiring	apply_job_task_count	Sequence diagram, steps 1-6
1.2	Default (one task) behavior unchanged	TaskCountWiring	apply_job_task_count	—
1.3	Jobs restored via the simulation/replay driver's snapshot loading derive the same value as fresh ingest	IngestTaskCountWiring (snapshot_load.c site)	apply_job_task_count	—
1.4	A resource-request update does not change the job's dispatched task count	test_maxtasks_wiring_ingest.c (structural regression test; no production change needed — see Existing Architecture Analysis and bug-1773)	—	—
1.5	Invalid pair cannot reach allocator dispatch	TaskCountWiring	apply_job_task_count (clamp)	—
2.1	Candidate plans evaluated for maxTasks > 1	Existing scheduler-allocation dispatch (unmodified)	—	Sequence diagram, step 7
2.2	Multi-host allocation committed when satisfiable	Existing parjob_allocator (unmodified)	—	Sequence diagram, step 8
2.3	maxTasks == 1 unaffected	Existing seqjob_allocator (unmodified)	—	—
3.1	Identical single-task results before/after	test_maxtasks_wiring_ingest.c	—	—
3.2	No cluster-wide behavior change while maxTasks always 1	test_maxtasks_wiring_ingest.c	—	—
3.3	Compound-nested primaries unaffected	TaskCountWiring's tag gate	apply_job_task_count	—
4.1	Controller ensures every host has launch info	Existing controller (unmodified), validated by xce-test-multihost	—	Sequence diagram, steps 8-10
4.2	Worker starts tasks on every allocated host	Existing worker (unmodified), validated by xce-test-multihost	—	Sequence diagram, steps 9-12
4.3	Scheduler reports running via existing mechanism	Existing scheduler (unmodified), validated by xce-test-multihost	—	Sequence diagram, step 13
4.4	Non-default launch type honored	WorkerLaunchTypeCoverage	—	—
5.1	Existing suites pass unmodified	Full regression run	—	—
5.2	HA soak shows no regression (does not exercise the simulation/replay driver's snapshot-loading path — see 5.6)	Existing xce-test-ha (unmodified)	—	—
5.3	Real multi-task job proven end to end	xce-test-multihost	—	Full sequence diagram
5.4	No cache collision across differing task counts, scoped to the ingest path with a correct fingerprint	test_resreq_task_count.c / scheduler-allocation's alloc_cache tests (fresh-ingest, core_build_job(), only; snapshot-restore is the documented bug-1772 exception — it never computes resreq_fp, confirmed live and unfixed by this feature; make_sched_job()'s path is dead code and out of consideration entirely)	—	—
5.5	Multi-task array job proven end to end	xce-test-multihost (array-job scenario)	—	Full sequence diagram, array variant
5.6	Simulation/replay-loaded snapshot derives the same task count as fresh submission	test_maxtasks_wiring_ingest.c (snapshot-restore site, comparative assertion against fresh-ingest for equivalent input)	apply_job_task_count	—

Components and Interfaces

Component	Domain/Layer	Intent	Req Coverage	Key Dependencies (P0/P1)	Contracts
TaskCountWiring	Shared C library (src/lib)	Apply a job's real task count onto its resreq's standalone primary	1.1, 1.2, 1.5, 3.3	None (pure function)	Service
IngestTaskCountWiring	Scheduler (src/scheduler/src)	Invoke the helper at every ingest call site whose result actually reaches dispatch	1.1, 1.3	TaskCountWiring (P0)	Service
xce-test-multihost	Test infrastructure	Prove real multi-host allocation and execution end to end	4.1, 4.2, 4.3, 5.3, 5.5	Existing controller/worker/scheduler binaries (P0)	Batch
WorkerLaunchTypeCoverage	Worker (src/worker/src)	Close a pre-existing, zero-coverage gap in the worker's launch-type dispatch decision	4.4	controller_connector.c (P0)	Service

Shared C Library (src/lib)

TaskCountWiring

Field	Detail
Intent	Overwrite minTasks/maxTasks on a resreq's own top-level primary with the job's real requested task count, leaving nested/compound primaries untouched.
Requirements	1.1, 1.2, 1.5, 3.3

Responsibilities & Constraints

• Sole owner of the tag-gate distinguishing a standalone primary (XCE_RESREQ_PRIMARY) from a COMPOUND/ALTER tree.
• Sole owner of the defensive minTasks <= maxTasks clamp.
• Never allocates memory, never mutates select/rusages/rusage_hash/span/same/affinity.

Dependencies

• Inbound: IngestTaskCountWiring — the two live scheduler-side call sites (P0).

Contracts: Service [x]

Service Interface

/*
 * Overwrite minTasks/maxTasks on resreq's own top-level primary with the
 * job's real requested task count. No-op if resreq is NULL or if its
 * top-level tag is not XCE_RESREQ_PRIMARY -- a COMPOUND or ALTER resreq's
 * nested primaries are left exactly as parse_resreq_primary() set them.
 */
void apply_job_task_count(xce_resreq_t* resreq,
                           uint32_t mintasks,
                           uint32_t maxtasks);

• Preconditions: resreq has already been successfully returned by parse_resreq()/parse_resreq_primary().
• Postconditions: if resreq->tag == XCE_RESREQ_PRIMARY, its primary->minTasks/primary->maxTasks equal mintasks/maxtasks, with maxtasks raised to mintasks if mintasks > maxtasks (mirroring convert.go's clamp direction exactly). Otherwise resreq is unchanged.
• Invariants: idempotent — calling it twice with the same inputs produces the same result.
• Type note: parameters are uint32_t, matching job_submit_data_t.mintasks/maxtasks and xce_sched_job_t.mintasks/maxtasks (both uint32_t, confirmed in types.h) — the values every call site actually holds. xce_primary_resreq_t.minTasks/maxTasks (the assignment target) are plain int (resreq.h:239-240); this function performs a deliberate, documented narrowing conversion at the write, not a silent one. Safe for realistic task counts (well under INT_MAX), and no existing code path today assigns across this int/uint32_t boundary, so this is a new, one-time, explicitly acknowledged conversion rather than a pattern already relied upon elsewhere.

Implementation Notes

• Integration: called once, immediately after each of the two live ingest parse_resreq()/parse_resreq_primary() call sites returns successfully.
• Validation: test_resreq_task_count.c covers the standalone-primary case, the COMPOUND/ALTER no-op case, and the defensive clamp case.
• Risks: none identified — the "undiscovered sixth call site" risk originally recorded in research.md has since been independently re-verified twice (exhaustive grep, no additional caller found either time).

Scheduler (src/scheduler/src)

IngestTaskCountWiring

Field	Detail
Intent	Invoke apply_job_task_count() at every ingest call site whose result actually reaches dispatch, using whichever job-context task count is already in scope there.
Requirements	1.1, 1.3

Responsibilities & Constraints

• Two call sites, each a single new line: core_build_job() (core_events.c); the restore path (snapshot_load.c).
• schedobj.c is untouched entirely: make_sched_job() and sched_update_existing_job() are both dead code (unreachable via sched_update(), which has zero live callers).
• core_events.c's JOB_UPDATE handler is deliberately NOT touched either — it's live, but rebuilds job->resreq and never job->schedobj, so the actually-dispatched object never sees the update regardless of whether this helper runs there (bug-1773).

Dependencies

• Outbound: TaskCountWiring (P0).

Contracts: Service [x]

Service Interface

No new interface — this is two one-line call-site additions to existing functions, not a new function.

Implementation Notes

• Integration: ordering matters only in that the call must happen after parse_resreq()/parse_resreq_primary() returns and before any downstream code (e.g. core_build_primary_schedobj()'s resreq_fp computation) reads minTasks/maxTasks — confirmed both wired sites already satisfy this ordering naturally.
• Validation: test_maxtasks_wiring_ingest.c white-box tests each site directly.
• Risks: none identified; each site's change is independently reviewable and revertable.

Worker (src/worker/src)

WorkerLaunchTypeCoverage

Field	Detail
Intent	Close a pre-existing, zero-coverage gap in the worker's launch-type dispatch decision, adjacent to (but not caused by) this feature's own fix.
Requirements	4.4

Responsibilities & Constraints

• Does not modify controller_connector.c — pure test addition against existing, unmodified production code.
• Primary target: job_start_remote_task()'s launch-type gate (correct no-op for an MPI launch type, deferring to MPI's own remote-launch mechanism; correct fan-out via XCE's per-host dispatch for the default type) — the real decision point Requirement 4.4 depends on, with no existing coverage of any kind.
• Secondary target: job_start_local_task()'s branch into create_mpi_cmdline(), confirming the built host list and command are correct.

Dependencies

• External: controller_connector.c's existing, unmodified launch-dispatch functions (P0).

Contracts: Service [x]

Service Interface

No new interface — a white-box test file (#include-ing controller_connector.c directly, the same technique already established by test_controller_connector.c) exercising existing static functions.

Implementation Notes

• Integration: no production code touched.
• Validation: test_worker_launch_type_dispatch.c.
• Risks: none — this is a pure test addition against unmodified code, adjacent-gap discovery rather than a fix this feature's own change requires.

Test Infrastructure

xce-test-multihost

Field	Detail
Intent	Prove a real multi-task job is genuinely allocated across multiple hosts by the scheduler's own decision, and actually runs to completion.
Requirements	4.1, 4.2, 4.3, 5.3, 5.5

Responsibilities & Constraints

• Deploys a single controller/scheduler node (no HA needed for this proof) plus two named xcecd_virtual instances, each sized so neither alone can satisfy the test job's requested task count.
• Drives submission via plain jobsub, never xce_test's jobrun manual-allocation bypass — the scheduler's own decision to span hosts is what this test exists to prove.
• Follows the pre-flight zombie-process/port check convention already established for xce-test-ha.

Dependencies

• External: real xce_controller, xce_scheduler, xcecd_virtual binaries (P0, unmodified by this spec).

Contracts: Batch [x]

Batch / Job Contract

• Trigger: manual invocation (bash .claude/skills/xce-test-multihost/run.sh), same convention as xce-test-ha/xce-test-resources.
• Input / validation: none beyond the built binaries; the script deploys its own fixture cluster.
• Output / destination: pass/fail verdict line matching the existing skills' [xce-test-*] PASS/FAIL convention, plus per-host worker log confirmation.
• Idempotency & recovery: each run deploys a fresh, uniquely-named cluster instance; no shared state across runs.

Implementation Notes

• Integration: reuses run-ha-soak.sh's submit_worker_job()/verify_worker_rejoin_resync() polling idiom (real job, poll jobinfo for RUN then DONE, grep each worker's own log) rather than inventing a new verification mechanism.
• Validation: two scenarios — a plain multi-task job, and a multi-task job submitted as part of an array job (Requirement 5.5).
• Risks: soak-style process/port flakiness, mitigated by the same pre-flight check xce-test-ha already established.

Data Models

Domain Model

No new entities or fields. xce_primary_resreq_t.minTasks/.maxTasks (existing fields) transition from a compile-time constant to a per-job derived value.

Persistence / Save-Restore Completeness

Per persistence.md's "Snapshot Save/Load Field-Completeness" section: minTasks/maxTasks are not independently persisted in the WAL or snapshot wire format. snapshot_wire.c persists job_submit_data_t.mintasks/maxtasks (the job-level source of truth, already correctly round-tripped, confirmed unchanged by this design); snapshot_load.c restores that value and then calls the same parse_resreq() chain fresh — the exact derivation this spec fixes. There is no field being newly persisted, so there is no new round-trip gap to enumerate: fixing the shared derivation fixes fresh-ingest and snapshot-restored jobs identically, by construction.

Error Handling

Error Categories and Responses

Invalid task-count pair (minTasks > maxTasks reaching apply_job_task_count()): defensive-only, since the apiserver already guarantees this invariant twice before any job reaches the C layer. maxtasks is raised to mintasks, mirroring convert.go's clamp direction exactly — no new job-rejecting error path is introduced.

mintasks == 0: not separately validated by this feature, since it's unreachable from either live call site (both convert.go and xce_test's jobsub independently clamp mintasks to at least 1 before a job ever reaches the C layer). A genuinely-zero pair (only reachable via a hand-crafted --sim snapshot bypassing both guards) degrades gracefully through scheduler-allocation's own pre-existing maxTasks <= 1 bail in parjob_allocator.c — the job pends as unsatisfiable rather than reaching an inconsistent dispatch state.

Testing Strategy

• Unit Tests (test_resreq_task_count.c): standalone primary receives the applied task count; COMPOUND/ALTER-tagged resreq is left untouched; minTasks > maxTasks is defensively clamped.
• Integration Tests (test_maxtasks_wiring_ingest.c): each of the two live ingest call sites applies the wiring correctly (fresh ingest via core_build_job(), and the simulation/replay driver's snapshot-loading path via snapshot_load.c); the snapshot-loading test asserts the same task count is derived as an equivalent fresh-ingest input would produce, satisfying Requirement 5.6's comparative claim directly; confirms sim_loop.c's job-ingest path reaches the same core_build_job()/core_events.c call site as the real driver, so simulated and real job ingest apply the fix identically with no driver/simulation divergence; a structural regression test drives a real resource-request update (Requirement 1.4) and confirms the job's dispatched task count is unchanged, proving the guarantee holds via job->schedobj never being rebuilt (bug-1773) rather than via any new code this feature adds. test_parjob_e2e_multislot.c is updated to drive its multi-host scenario through unmodified real ingest, closing the loop on the bug it originally characterized.
• E2E Tests (xce-test-multihost): a real multi-task job submitted via plain jobsub against two real workers is genuinely split across both hosts by the scheduler's own decision and reaches DONE on both (4.1-4.3); a second scenario repeats this for a multi-task array job (5.5).
• Non-default launch type (test_worker_launch_type_dispatch.c, white-box): confirms job_start_remote_task()'s launch-type gate correctly no-ops for an MPI launch type and correctly fans out via XCE's own per-host dispatch for the default type — the actual decision point Requirement 4.4 depends on, currently with zero coverage of any kind — and confirms job_start_local_task()'s branch into create_mpi_cmdline() builds the correct host list and command (4.4), without requiring a real MPI runtime in the test environment. The E2E harness above only exercises the default launch path, whose availability is guaranteed.
• Regression: full existing scheduler/controller/worker test suites re-run unmodified (5.1); existing xce-test-ha sequential and chaotic soak re-run unmodified as non-regression evidence (5.2); a cache-key collision test (reusing scheduler-allocation's alloc_cache test conventions) confirms two otherwise-identical jobs with different task counts don't collide in the unsatisfiable-resreq cache, for the fresh-ingest path (core_build_job()) where resreq_fp is correctly computed (5.4) — the snapshot-restore path is explicitly excluded from this test per the documented, pre-existing bug-1772 exception (it never computes resreq_fp, confirmed live and unfixed by this feature), not silently skipped; make_sched_job()'s path is dead code and not a live production concern at all.