myxce/.kiro/specs/resreq-maxtasks-wiring/design.md

## Overview
This feature closes bug-1770: a job's real requested task count is already correctly threaded from the apiserver all the way to `xce_sched_job_t.mintasks/maxtasks`, but the parsed resource request the scheduler's allocator dispatch actually reads (`xce_primary_resreq_t.minTasks/maxTasks`) is unconditionally hardcoded to one task. This makes the scheduler-allocation feature's already-implemented, already-tested, already-soak-validated multi-host allocation logic (`parjob_allocator`) permanently unreachable from any real job submission today.

**Users**: cluster users submitting multi-task jobs, and cluster operators relying on existing single-task jobs continuing to behave identically.

**Impact**: changes the resource request the scheduler dispatches on from an always-`1` constant to a real, per-job derived value for standalone primary resource requests, at every point a resreq is parsed from a job's submitted or persisted state. No existing component's ownership, wire format, or public interface changes shape — this is a targeted derivation fix plus the validation needed to trust it.

### Goals
- A standalone primary resource request's `minTasks`/`maxTasks` reflect the job's real requested task count, at fresh ingest and at snapshot restore, and remain correct across a subsequent resource-request update (a guarantee this feature relies on an existing structural property for, rather than actively enforcing — see Existing Architecture Analysis).
- A job requiring more tasks than one host can provide is genuinely evaluated for, and where appropriate allocated across, multiple hosts, through the real scheduler dispatch path.
- A committed multi-host allocation is proven, end to end, to actually launch and run on real (or virtual) workers.
- Zero behavioral change for any job with a requested task count of one, and zero behavioral change for compound/alternative resource requests.

### Non-Goals
- Implementing `span[ptile=N]` resreq-grammar semantics (remains parsed and discarded, unchanged).
- Wiring task count into primary resource requests nested inside a compound/alternative resource request (bug-1768 already makes their resource checking meaningless regardless of task count).
- Fixing the snapshot-restore `resreq_fp` computation gap (bug-1772, tracked separately) or its sibling in `make_sched_job()` (bug-1774) — the latter is dead code today (confirmed: `make_sched_job()`'s entire call chain, via `sched_update_jobs()`/`sched_update()`, has zero live callers anywhere in this codebase), so this feature does not wire it at all, not merely defer fixing it.
- Fixing a job's resource-request update never taking effect for scheduling purposes for any field (bug-1773) — this feature's own task-count guarantee across an update holds for that same structural reason, not because this feature actively re-wires anything on update.
- Modifying the controller's or worker's existing multi-host launch code (`mk_launchinfo()`, `job_start_remote_task()`, `create_mpi_cmdline()`) — this design depends on and validates that code, but does not change it.
- Fixing the pre-existing per-host live memory-usage reconciliation limitation for multi-host jobs.

## Boundary Commitments

### This Spec Owns
- The derivation that sets a standalone primary resource request's `minTasks`/`maxTasks` from the job's real requested task count, at every live call site whose parsed resreq actually reaches scheduler dispatch.
- The defensive invariant that an invalid (`minTasks > maxTasks`) pair cannot reach allocator dispatch, regardless of upstream validation.
- A new automated end-to-end test proving a real multi-task job, submitted through the normal submission path, is genuinely allocated across multiple hosts by the scheduler and actually runs to completion.

### Out of Boundary
- `span[...]` resreq-grammar syntax (parsed, discarded, unchanged).
- Primary resource requests nested inside a compound/alternative resource request (bug-1768).
- The snapshot-restore `resreq_fp` gap (bug-1772).
- Per-host live memory-usage reconciliation for multi-host jobs (pre-existing, documented limitation).
- The scheduler's plan-generation/allocation algorithm itself (owned by `scheduler-allocation`; this spec only makes it reachable).
- The controller/worker multi-host launch mechanism's internal implementation (pre-existing; this spec validates it, does not modify it).

### Allowed Dependencies
- `scheduler-allocation`'s `parjob_allocator`/`seqjob_allocator`/`alloc_cache` — consumed as-is; this spec must not modify their dispatch logic.
- The apiserver's existing `checkNumProcessors()` validation and `backend/convert.go`'s `mintasks`/`maxtasks` clamp — relied upon as the primary validation layer; this spec adds a redundant defensive check, not a replacement.
- The controller/worker's existing multi-host launch code (`mk_launchinfo()`, `job_start_remote_task()`, `create_mpi_cmdline()`) — read-only dependency for the new end-to-end test.

### Revalidation Triggers
- Any change to `parse_resreq_primary()`'s or `parse_resreq()`'s signature, or to `xce_primary_resreq_t`'s struct shape.
- Any change to how a resreq's top-level tag distinguishes a standalone primary from a `COMPOUND`/`ALTER` tree.
- Any change to `job_submit_data_t.mintasks/maxtasks` or its Go-side validation (`checkNumProcessors()`, `convert.go`'s clamp).
- Any change to `scheduler-allocation`'s dispatch gate (`sched_make_primary_alloc_()`'s `maxTasks == 1` branch) or to bug-1768's compound-nested resource-checking gap, since a future fix there would change what "out of boundary" means for compound resource requests.

## Architecture

### Existing Architecture Analysis
- `parse_resreq_primary(xce_primary_resreq_t*, char*, var, char**)` and `parse_resreq(xce_resreq_t*, char*, var, char**)` (`src/lib/resreq.c`) take no job-context parameter — they cannot see a job's requested task count. Six existing test call sites invoke the bare production signature directly with no job object in scope, ruling out a signature change as the fix mechanism.
- Five call sites in total reach this parser in the source tree, but only **two are reachable in production today**. The scheduler's actual dispatch reads `job->schedobj->u.primary->resreq` (`scheduler.c:315`), a pointer set exactly once, by whichever function constructs `job->schedobj`. Exactly three functions ever do this (`job->schedobj =` grepped exhaustively): `core_build_job()` (`core_events.c`), `make_sched_job()` (`schedobj.c`), and the snapshot-restore path (`snapshot_load.c`). Of these, `make_sched_job()` is **dead code**: its only caller, `sched_update_jobs()`, is only called from inside `sched_update()`'s switch, and `sched_update()` (`scheduler.c`) has zero live callers anywhere in this codebase — confirmed by exhaustive grep and corroborated by the codebase's own pre-existing comments describing it as dead/superseded, and by the identical, already-fixed sibling case in bug-1740. `core_build_job()` (fresh submission, via the driver-translated `XCE_CORE_EVENT_JOBS_PUSH` path) and the `snapshot_load.c` restore path are the two genuinely live schedobj-construction sites, and are this spec's only implementation targets.
- The remaining two of the five parser call sites are job-update handlers (`sched_update_existing_job()` in `schedobj.c`, the `JOB_UPDATE` handler in `core_events.c`) that replace `job->resreq` but never rebuild `job->schedobj`. `sched_update_existing_job()` is itself dead code (its sole caller is also inside `sched_update()`'s dead switch). `core_events.c`'s `JOB_UPDATE` handler IS live (reached via the driver-translated path, confirmed by tracing `main.c` → `driver.c`'s event translation → `xce_core_push_events()`) — but it never affects scheduling regardless, since `job->schedobj` isn't rebuilt there either, and no live feature in this codebase today ever actually changes an existing job's `submit->resreq` (grepped the whole controller: nothing mutates it post-submission). Applying this feature's helper at either update call site would be dead code with zero observable effect. Requirement 1.4's guarantee (task count unaffected by an update) already holds today for this structural reason, independent of this feature. The update path never taking effect for scheduling at all, for any field, is a real, separately-tracked, pre-existing defect (bug-1773).
- **"Snapshot restore" (Requirement 1.3/1.4's `snapshot_load.c` site) is live only via the simulation/replay driver, not live-cluster crash recovery.** `xce_sched_core_load_snapshot()`'s only real caller anywhere in `src/` is `sim.c`'s `sim_locate_and_load_snapshot()`, reachable only through `main.c`'s `--sim FILE` offline-replay flag; normal-mode `main.c` has no snapshot-load call or flag at all. A live scheduler restart (including post-failover, e.g. `run-ha-soak.sh`'s own scheduler-restart step) starts with empty in-memory state and rejoins via `on_scheduler_join()` → the same live `core_build_job()` path fresh ingest uses — it never touches `snapshot_load.c`. This means the HA soak (Requirement 5.2) provides zero exercise of this call site; Requirement 5.6 validates it separately. The fix is still worth wiring here — `--sim` mode drives the identical `parjob_allocator`/`seqjob_allocator` dispatch per this project's own driver-equivalence invariant, so a saved snapshot's task count needs to be correct for simulation/tuning fidelity — but this is a genuinely different (and narrower) liveness scope than "post-restart/post-failover" phrasing might suggest, and bug-1772's own description has been corrected accordingly.
- A single job's resreq parse tree can contain more than one primary when `COMPOUND`/`ALTER` syntax is used; `parse_resreq_primary()` already runs once per primary node in the tree, uniformly. Only the job's own top-level standalone primary (tag `XCE_RESREQ_PRIMARY`) is in this spec's boundary — a nested primary inside `COMPOUND`/`ALTER` must be left exactly as today.
- `job_submit_data_t.mintasks/maxtasks` (the job-level source of truth) is already correctly threaded end-to-end from the apiserver's `num_processors`/`maxnum_processors` fields, validated twice (`checkNumProcessors()`, `convert.go`'s clamp) before reaching the C layer, and already correctly round-tripped through the WAL/snapshot wire format. This spec's boundary starts exactly where that correct value stops propagating.

### Project Invariant Cross-Check
`core_events.c` (home of `core_build_job()`, one of this spec's real call sites) is part of `sched_core_isolation_lib` — the "one core, two consumers" layer shared between the real driver (`driver.c`) and the simulation driver (`sim_loop.c`), per this project's own scheduler-allocation task 5.2 lesson (a fix applied only to the real-driver path there caused a real driver/simulation divergence that corrupted `xce_sched_sim_driver_t` state). **Confirmed, not merely assumed**: `driver.c` and `sim_loop.c` both build `XCE_CORE_EVENT_JOBS_PUSH` events and call the same shared `xce_core_push_events()` (`core_events.c`), whose `JOBS_PUSH` case calls `core_build_job()` directly — no parallel, sim-specific job constructor exists anywhere. Wiring the fix in `core_build_job()` therefore applies identically to real and simulated job ingest with no divergence risk. `test_maxtasks_wiring_ingest.c` still includes a regression test exercising this path through `sim_loop.c`'s own entry point, to keep this confirmed property under test rather than resting on a one-time design-time check.

### Architecture Pattern & Boundary Map

```mermaid
graph TB
    ApiServer[Apiserver job submission] --> SubmitData[job_submit_data_t mintasks maxtasks]
    SubmitData --> Ingest[Scheduler ingest call sites]
    Ingest --> ParseResreq[parse_resreq existing parser]
    ParseResreq --> Helper[apply_job_task_count new helper]
    Helper --> Primary[xce_primary_resreq_t minTasks maxTasks]
    Primary --> Dispatch[Scheduler dispatch existing]
    Dispatch --> SeqJob[seqjob_allocator existing]
    Dispatch --> ParJob[parjob_allocator existing]
    ParJob --> Commit[Multi host allocation committed]
    Commit --> Controller[Controller mk_launchinfo existing]
    Controller --> Worker[Worker job_start_remote_task existing]
    Worker --> Running[Job reaches RUN then DONE]
```

**Architecture Integration**:
- Selected pattern: a single new helper function, invoked by each of the two ingest call sites whose result actually reaches dispatch (`core_build_job()` and the snapshot-restore path), immediately after its existing `parse_resreq()`/`parse_resreq_primary()` call returns.
- Domain boundaries: `src/lib/resreq.c` owns the new helper and its tag-gating/clamp logic; each scheduler-side call site owns invoking it with whichever job-context task count is already in scope there.
- Existing patterns preserved: this mirrors the codebase's own existing idiom of copying a field from `submit->` onto a target struct immediately after that struct is built — already done identically for `job->mintasks/maxtasks` itself at both of these sites.
- New component rationale: one shared helper avoids duplicating the `COMPOUND`/`ALTER` tag-check twice and gives one independently unit-testable unit for the only real logic this fix introduces.
- Steering compliance: no new WAL/snapshot field is introduced (see Data Models); `resreq.c` is a shared C library untouched by the scheduler-core isolation boundary; the fix does not alter `scheduler-allocation`'s dispatch or allocation logic, only what reaches it.

### Technology Stack

| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|------------------|-------|
| Shared C library | Existing C (gnu17), `src/lib/resreq.c` | New `apply_job_task_count()` helper | No new dependency |
| Scheduler (C) | Existing C, `src/scheduler/src/` | Two live ingest call-site wiring | `core_events.c`, `snapshot_load.c` |
| Test infrastructure | Existing `xce_test` debug CLI + `xcecd_virtual` | New end-to-end test | Extends the pattern already used by `xce-test-resources`/`xce-test-ha` |

## File Structure Plan

### Modified Files
- `src/lib/resreq.c` — add `apply_job_task_count()`; `parse_resreq_primary()`'s existing `minTasks = 1; maxTasks = 1;` hardcode is left in place as the correct default for a freshly-parsed resreq before the helper is applied, and as the permanent behavior for nested `COMPOUND`/`ALTER` primaries the helper deliberately skips.
- `src/include/resreq.h` — declare `apply_job_task_count()`.
- `src/scheduler/src/core_events.c` — call the helper after `core_build_job()`'s `parse_resreq()` call, using `j->submit->mintasks/maxtasks`. `core_events.c`'s `JOB_UPDATE` handler is NOT modified — it's live, but `job->schedobj` (what dispatch actually reads) is never rebuilt by an update, so applying the helper to the freshly-parsed `job->resreq` there would be dead code (see Existing Architecture Analysis and bug-1773).
- `src/scheduler/src/snapshot_load.c` — call the helper after the restore path's `parse_resreq()` call, using `job->mintasks/maxtasks` (already restored earlier in the same function, before this call).
- `src/scheduler/src/schedobj.c` is NOT modified — `make_sched_job()` and `sched_update_existing_job()` are both dead code (confirmed: unreachable via `sched_update()`, which has zero live callers), so wiring either would have no production effect.
- `src/scheduler/testsuite/test_parjob_e2e_multislot.c` — remove the `PATCH_MAXTASKS_FOR_BUG_1770` workaround now that real ingest correctly wires `maxTasks`, converting it to drive the same scenario through unmodified real ingest.

### New Files
- `src/lib/testsuite/test_resreq_task_count.c` — unit tests for `apply_job_task_count()` in isolation.
- `src/scheduler/testsuite/test_maxtasks_wiring_ingest.c` — white-box tests confirming each of the two wired ingest call sites applies the wiring correctly (including the fresh-ingest-vs-snapshot-restore comparative assertion for Requirement 5.6), plus the structural regression test for Requirement 1.4.
- `src/worker/testsuite/test_worker_launch_type_dispatch.c` — white-box test targeting `job_start_remote_task()`'s launch-type gate (`controller_connector.c`) first: confirms it correctly no-ops (defers to MPI's own remote-launch mechanism) for an MPI launch type, and correctly fans the job out via XCE's own per-host dispatch for the default launch type — this is the actual decision point Requirement 4.4 depends on, and today has zero test coverage of any kind. Secondarily confirms `job_start_local_task()`'s branch into `create_mpi_cmdline()` builds the right host list and command. Neither requires a real MPI runtime installed in the test environment (Requirement 4.4).
- `.claude/skills/xce-test-multihost/` — new skill (`SKILL.md` + driver script), following `xce-test-resources`'s single-controller-node pattern: deploys one controller/scheduler plus two named `xcecd_virtual` workers, submits a real multi-task job via `jobsub`, and verifies genuine multi-host `RUN`-then-`DONE` execution on both hosts, plus one array-job variant.

## System Flows

```mermaid
sequenceDiagram
    participant User
    participant Api as Apiserver
    participant Sched as Scheduler ingest
    participant Alloc as Scheduler dispatch
    participant Ctrl as Controller
    participant W1 as Worker host A
    participant W2 as Worker host B

    User->>Api: submit job with -n 4,6
    Api->>Api: checkNumProcessors validate
    Api->>Sched: job_submit_data_t mintasks maxtasks
    Sched->>Sched: parse_resreq existing
    Sched->>Sched: apply_job_task_count new
    Sched->>Alloc: standalone primary minTasks maxTasks real value
    Alloc->>Alloc: maxTasks greater than one route to parjob_allocator
    Alloc->>Ctrl: commit multi host allocation
    Ctrl->>W1: launch info host A share
    Ctrl->>W2: launch info host B share
    W1-->>Ctrl: task started
    W2-->>Ctrl: task started
    Ctrl-->>Sched: job running
    Sched-->>User: job reaches DONE
```

Key decision visible in the diagram: `apply_job_task_count()` sits strictly between the existing parser and the existing dispatch gate — nothing downstream of it (allocator, controller, worker) changes; nothing upstream of it (apiserver validation, `job_submit_data_t` wiring) changes.

## Requirements Traceability

| Requirement | Summary | Components | Interfaces | Flows |
|-------------|---------|-------------|------------|-------|
| 1.1 | Real task count reaches the standalone primary | `TaskCountWiring`, `IngestTaskCountWiring` | `apply_job_task_count` | Sequence diagram, steps 1-6 |
| 1.2 | Default (one task) behavior unchanged | `TaskCountWiring` | `apply_job_task_count` | — |
| 1.3 | Jobs restored via the simulation/replay driver's snapshot loading derive the same value as fresh ingest | `IngestTaskCountWiring` (snapshot_load.c site) | `apply_job_task_count` | — |
| 1.4 | A resource-request update does not change the job's dispatched task count | `test_maxtasks_wiring_ingest.c` (structural regression test; no production change needed — see Existing Architecture Analysis and bug-1773) | — | — |
| 1.5 | Invalid pair cannot reach allocator dispatch | `TaskCountWiring` | `apply_job_task_count` (clamp) | — |
| 2.1 | Candidate plans evaluated for maxTasks > 1 | Existing `scheduler-allocation` dispatch (unmodified) | — | Sequence diagram, step 7 |
| 2.2 | Multi-host allocation committed when satisfiable | Existing `parjob_allocator` (unmodified) | — | Sequence diagram, step 8 |
| 2.3 | maxTasks == 1 unaffected | Existing `seqjob_allocator` (unmodified) | — | — |
| 3.1 | Identical single-task results before/after | `test_maxtasks_wiring_ingest.c` | — | — |
| 3.2 | No cluster-wide behavior change while maxTasks always 1 | `test_maxtasks_wiring_ingest.c` | — | — |
| 3.3 | Compound-nested primaries unaffected | `TaskCountWiring`'s tag gate | `apply_job_task_count` | — |
| 4.1 | Controller ensures every host has launch info | Existing controller (unmodified), validated by `xce-test-multihost` | — | Sequence diagram, steps 8-10 |
| 4.2 | Worker starts tasks on every allocated host | Existing worker (unmodified), validated by `xce-test-multihost` | — | Sequence diagram, steps 9-12 |
| 4.3 | Scheduler reports running via existing mechanism | Existing scheduler (unmodified), validated by `xce-test-multihost` | — | Sequence diagram, step 13 |
| 4.4 | Non-default launch type honored | `WorkerLaunchTypeCoverage` | — | — |
| 5.1 | Existing suites pass unmodified | Full regression run | — | — |
| 5.2 | HA soak shows no regression (does not exercise the simulation/replay driver's snapshot-loading path — see 5.6) | Existing `xce-test-ha` (unmodified) | — | — |
| 5.3 | Real multi-task job proven end to end | `xce-test-multihost` | — | Full sequence diagram |
| 5.4 | No cache collision across differing task counts, scoped to the ingest path with a correct fingerprint | `test_resreq_task_count.c` / `scheduler-allocation`'s `alloc_cache` tests (fresh-ingest, `core_build_job()`, only; snapshot-restore is the documented bug-1772 exception — it never computes `resreq_fp`, confirmed live and unfixed by this feature; `make_sched_job()`'s path is dead code and out of consideration entirely) | — | — |
| 5.5 | Multi-task array job proven end to end | `xce-test-multihost` (array-job scenario) | — | Full sequence diagram, array variant |
| 5.6 | Simulation/replay-loaded snapshot derives the same task count as fresh submission | `test_maxtasks_wiring_ingest.c` (snapshot-restore site, comparative assertion against fresh-ingest for equivalent input) | `apply_job_task_count` | — |

## Components and Interfaces

| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|-----------|--------------|--------|---------------|---------------------------|-----------|
| `TaskCountWiring` | Shared C library (`src/lib`) | Apply a job's real task count onto its resreq's standalone primary | 1.1, 1.2, 1.5, 3.3 | None (pure function) | Service |
| `IngestTaskCountWiring` | Scheduler (`src/scheduler/src`) | Invoke the helper at every ingest call site whose result actually reaches dispatch | 1.1, 1.3 | `TaskCountWiring` (P0) | Service |
| `xce-test-multihost` | Test infrastructure | Prove real multi-host allocation and execution end to end | 4.1, 4.2, 4.3, 5.3, 5.5 | Existing controller/worker/scheduler binaries (P0) | Batch |
| `WorkerLaunchTypeCoverage` | Worker (`src/worker/src`) | Close a pre-existing, zero-coverage gap in the worker's launch-type dispatch decision | 4.4 | `controller_connector.c` (P0) | Service |

### Shared C Library (`src/lib`)

#### TaskCountWiring

| Field | Detail |
|-------|--------|
| Intent | Overwrite `minTasks`/`maxTasks` on a resreq's own top-level primary with the job's real requested task count, leaving nested/compound primaries untouched. |
| Requirements | 1.1, 1.2, 1.5, 3.3 |

**Responsibilities & Constraints**
- Sole owner of the tag-gate distinguishing a standalone primary (`XCE_RESREQ_PRIMARY`) from a `COMPOUND`/`ALTER` tree.
- Sole owner of the defensive `minTasks <= maxTasks` clamp.
- Never allocates memory, never mutates `select`/`rusages`/`rusage_hash`/`span`/`same`/`affinity`.

**Dependencies**
- Inbound: `IngestTaskCountWiring` — the two live scheduler-side call sites (P0).

**Contracts**: Service [x]

##### Service Interface
```c
/*
 * Overwrite minTasks/maxTasks on resreq's own top-level primary with the
 * job's real requested task count. No-op if resreq is NULL or if its
 * top-level tag is not XCE_RESREQ_PRIMARY -- a COMPOUND or ALTER resreq's
 * nested primaries are left exactly as parse_resreq_primary() set them.
 */
void apply_job_task_count(xce_resreq_t* resreq,
                           uint32_t mintasks,
                           uint32_t maxtasks);
```
- Preconditions: `resreq` has already been successfully returned by `parse_resreq()`/`parse_resreq_primary()`.
- Postconditions: if `resreq->tag == XCE_RESREQ_PRIMARY`, its `primary->minTasks`/`primary->maxTasks` equal `mintasks`/`maxtasks`, with `maxtasks` raised to `mintasks` if `mintasks > maxtasks` (mirroring `convert.go`'s clamp direction exactly). Otherwise `resreq` is unchanged.
- Invariants: idempotent — calling it twice with the same inputs produces the same result.
- Type note: parameters are `uint32_t`, matching `job_submit_data_t.mintasks/maxtasks` and `xce_sched_job_t.mintasks/maxtasks` (both `uint32_t`, confirmed in `types.h`) — the values every call site actually holds. `xce_primary_resreq_t.minTasks/maxTasks` (the assignment target) are plain `int` (`resreq.h:239-240`); this function performs a deliberate, documented narrowing conversion at the write, not a silent one. Safe for realistic task counts (well under `INT_MAX`), and no existing code path today assigns across this `int`/`uint32_t` boundary, so this is a new, one-time, explicitly acknowledged conversion rather than a pattern already relied upon elsewhere.

**Implementation Notes**
- Integration: called once, immediately after each of the two live ingest `parse_resreq()`/`parse_resreq_primary()` call sites returns successfully.
- Validation: `test_resreq_task_count.c` covers the standalone-primary case, the `COMPOUND`/`ALTER` no-op case, and the defensive clamp case.
- Risks: none identified — the "undiscovered sixth call site" risk originally recorded in `research.md` has since been independently re-verified twice (exhaustive grep, no additional caller found either time).

### Scheduler (`src/scheduler/src`)

#### IngestTaskCountWiring

| Field | Detail |
|-------|--------|
| Intent | Invoke `apply_job_task_count()` at every ingest call site whose result actually reaches dispatch, using whichever job-context task count is already in scope there. |
| Requirements | 1.1, 1.3 |

**Responsibilities & Constraints**
- Two call sites, each a single new line: `core_build_job()` (`core_events.c`); the restore path (`snapshot_load.c`).
- `schedobj.c` is untouched entirely: `make_sched_job()` and `sched_update_existing_job()` are both dead code (unreachable via `sched_update()`, which has zero live callers).
- `core_events.c`'s `JOB_UPDATE` handler is deliberately NOT touched either — it's live, but rebuilds `job->resreq` and never `job->schedobj`, so the actually-dispatched object never sees the update regardless of whether this helper runs there (bug-1773).

**Dependencies**
- Outbound: `TaskCountWiring` (P0).

**Contracts**: Service [x]

##### Service Interface
No new interface — this is two one-line call-site additions to existing functions, not a new function.

**Implementation Notes**
- Integration: ordering matters only in that the call must happen after `parse_resreq()`/`parse_resreq_primary()` returns and before any downstream code (e.g. `core_build_primary_schedobj()`'s `resreq_fp` computation) reads `minTasks`/`maxTasks` — confirmed both wired sites already satisfy this ordering naturally.
- Validation: `test_maxtasks_wiring_ingest.c` white-box tests each site directly.
- Risks: none identified; each site's change is independently reviewable and revertable.

### Worker (`src/worker/src`)

#### WorkerLaunchTypeCoverage

| Field | Detail |
|-------|--------|
| Intent | Close a pre-existing, zero-coverage gap in the worker's launch-type dispatch decision, adjacent to (but not caused by) this feature's own fix. |
| Requirements | 4.4 |

**Responsibilities & Constraints**
- Does not modify `controller_connector.c` — pure test addition against existing, unmodified production code.
- Primary target: `job_start_remote_task()`'s launch-type gate (correct no-op for an MPI launch type, deferring to MPI's own remote-launch mechanism; correct fan-out via XCE's per-host dispatch for the default type) — the real decision point Requirement 4.4 depends on, with no existing coverage of any kind.
- Secondary target: `job_start_local_task()`'s branch into `create_mpi_cmdline()`, confirming the built host list and command are correct.

**Dependencies**
- External: `controller_connector.c`'s existing, unmodified launch-dispatch functions (P0).

**Contracts**: Service [x]

##### Service Interface
No new interface — a white-box test file (`#include`-ing `controller_connector.c` directly, the same technique already established by `test_controller_connector.c`) exercising existing static functions.

**Implementation Notes**
- Integration: no production code touched.
- Validation: `test_worker_launch_type_dispatch.c`.
- Risks: none — this is a pure test addition against unmodified code, adjacent-gap discovery rather than a fix this feature's own change requires.

### Test Infrastructure

#### xce-test-multihost

| Field | Detail |
|-------|--------|
| Intent | Prove a real multi-task job is genuinely allocated across multiple hosts by the scheduler's own decision, and actually runs to completion. |
| Requirements | 4.1, 4.2, 4.3, 5.3, 5.5 |

**Responsibilities & Constraints**
- Deploys a single controller/scheduler node (no HA needed for this proof) plus two named `xcecd_virtual` instances, each sized so neither alone can satisfy the test job's requested task count.
- Drives submission via plain `jobsub`, never `xce_test`'s `jobrun` manual-allocation bypass — the scheduler's own decision to span hosts is what this test exists to prove.
- Follows the pre-flight zombie-process/port check convention already established for `xce-test-ha`.

**Dependencies**
- External: real `xce_controller`, `xce_scheduler`, `xcecd_virtual` binaries (P0, unmodified by this spec).

**Contracts**: Batch [x]

##### Batch / Job Contract
- Trigger: manual invocation (`bash .claude/skills/xce-test-multihost/run.sh`), same convention as `xce-test-ha`/`xce-test-resources`.
- Input / validation: none beyond the built binaries; the script deploys its own fixture cluster.
- Output / destination: pass/fail verdict line matching the existing skills' `[xce-test-*] PASS/FAIL` convention, plus per-host worker log confirmation.
- Idempotency & recovery: each run deploys a fresh, uniquely-named cluster instance; no shared state across runs.

**Implementation Notes**
- Integration: reuses `run-ha-soak.sh`'s `submit_worker_job()`/`verify_worker_rejoin_resync()` polling idiom (real job, poll `jobinfo` for `RUN` then `DONE`, grep each worker's own log) rather than inventing a new verification mechanism.
- Validation: two scenarios — a plain multi-task job, and a multi-task job submitted as part of an array job (Requirement 5.5).
- Risks: soak-style process/port flakiness, mitigated by the same pre-flight check `xce-test-ha` already established.

## Data Models

### Domain Model
No new entities or fields. `xce_primary_resreq_t.minTasks`/`.maxTasks` (existing fields) transition from a compile-time constant to a per-job derived value.

### Persistence / Save-Restore Completeness
Per `persistence.md`'s "Snapshot Save/Load Field-Completeness" section: `minTasks`/`maxTasks` are **not** independently persisted in the WAL or snapshot wire format. `snapshot_wire.c` persists `job_submit_data_t.mintasks/maxtasks` (the job-level source of truth, already correctly round-tripped, confirmed unchanged by this design); `snapshot_load.c` restores that value and then calls the same `parse_resreq()` chain fresh — the exact derivation this spec fixes. There is no field being newly persisted, so there is no new round-trip gap to enumerate: fixing the shared derivation fixes fresh-ingest and snapshot-restored jobs identically, by construction.

## Error Handling

### Error Categories and Responses
**Invalid task-count pair** (`minTasks > maxTasks` reaching `apply_job_task_count()`): defensive-only, since the apiserver already guarantees this invariant twice before any job reaches the C layer. `maxtasks` is raised to `mintasks`, mirroring `convert.go`'s clamp direction exactly — no new job-rejecting error path is introduced.

**`mintasks == 0`**: not separately validated by this feature, since it's unreachable from either live call site (both `convert.go` and `xce_test`'s `jobsub` independently clamp `mintasks` to at least 1 before a job ever reaches the C layer). A genuinely-zero pair (only reachable via a hand-crafted `--sim` snapshot bypassing both guards) degrades gracefully through `scheduler-allocation`'s own pre-existing `maxTasks <= 1` bail in `parjob_allocator.c` — the job pends as unsatisfiable rather than reaching an inconsistent dispatch state.

## Testing Strategy

- **Unit Tests** (`test_resreq_task_count.c`): standalone primary receives the applied task count; `COMPOUND`/`ALTER`-tagged resreq is left untouched; `minTasks > maxTasks` is defensively clamped.
- **Integration Tests** (`test_maxtasks_wiring_ingest.c`): each of the two live ingest call sites applies the wiring correctly (fresh ingest via `core_build_job()`, and the simulation/replay driver's snapshot-loading path via `snapshot_load.c`); the snapshot-loading test asserts the same task count is derived as an equivalent fresh-ingest input would produce, satisfying Requirement 5.6's comparative claim directly; confirms `sim_loop.c`'s job-ingest path reaches the same `core_build_job()`/`core_events.c` call site as the real driver, so simulated and real job ingest apply the fix identically with no driver/simulation divergence; a structural regression test drives a real resource-request update (Requirement 1.4) and confirms the job's dispatched task count is unchanged, proving the guarantee holds via `job->schedobj` never being rebuilt (bug-1773) rather than via any new code this feature adds. `test_parjob_e2e_multislot.c` is updated to drive its multi-host scenario through unmodified real ingest, closing the loop on the bug it originally characterized.
- **E2E Tests** (`xce-test-multihost`): a real multi-task job submitted via plain `jobsub` against two real workers is genuinely split across both hosts by the scheduler's own decision and reaches `DONE` on both (4.1-4.3); a second scenario repeats this for a multi-task array job (5.5).
- **Non-default launch type** (`test_worker_launch_type_dispatch.c`, white-box): confirms `job_start_remote_task()`'s launch-type gate correctly no-ops for an MPI launch type and correctly fans out via XCE's own per-host dispatch for the default type — the actual decision point Requirement 4.4 depends on, currently with zero coverage of any kind — and confirms `job_start_local_task()`'s branch into `create_mpi_cmdline()` builds the correct host list and command (4.4), without requiring a real MPI runtime in the test environment. The E2E harness above only exercises the default launch path, whose availability is guaranteed.
- **Regression**: full existing scheduler/controller/worker test suites re-run unmodified (5.1); existing `xce-test-ha` sequential and chaotic soak re-run unmodified as non-regression evidence (5.2); a cache-key collision test (reusing `scheduler-allocation`'s `alloc_cache` test conventions) confirms two otherwise-identical jobs with different task counts don't collide in the unsatisfiable-resreq cache, for the fresh-ingest path (`core_build_job()`) where `resreq_fp` is correctly computed (5.4) — the snapshot-restore path is explicitly excluded from this test per the documented, pre-existing bug-1772 exception (it never computes `resreq_fp`, confirmed live and unfixed by this feature), not silently skipped; `make_sched_job()`'s path is dead code and not a live production concern at all.