# Implementation Plan - [x] 1. Implement and test the task-count derivation helper - Overwrite a standalone (top-level) primary resource request's minimum/maximum task count with the job's real requested minimum/maximum task count - Leave a compound/alternative resource request's nested primary resource requests untouched (no-op) — only the job's own top-level standalone primary is in scope - Defensively clamp an invalid pair by raising the maximum to the minimum when the requested minimum exceeds the requested maximum, mirroring the apiserver's own clamp direction - Write unit tests covering: a standalone primary receiving the applied task count, a compound/alternative resreq being left untouched, and the minimum-exceeds-maximum clamp - Observable: a passing unit test suite demonstrates all three cases (override, no-op, clamp) - _Requirements: 1.1, 1.2, 1.5, 3.3_ - _Boundary: TaskCountWiring_ - [ ] 2. Core: wire the derived task count into live scheduler ingest paths and close the adjacent worker launch-type gap - [x] 2.1 (P) Wire fresh job-ingest to apply the job's real requested task count - Call the new helper immediately after fresh job ingest's existing resource-request parse, using the job's real submitted minimum/maximum task count already in scope there - Observable: a job's dispatched resource request (the one the allocator actually reads) reflects the real requested task count immediately after fresh ingest, not the previous always-one-task default - _Requirements: 1.1_ - _Boundary: core_events.c (core_build_job)_ - [x] 2.2 (P) Wire simulation/replay snapshot-restore ingest to apply the same real task count - Call the new helper immediately after the snapshot-restore path's existing resource-request parse, using the job's already-restored minimum/maximum task count - Observable: a job reconstructed from a saved snapshot via the simulation/replay driver has its dispatched resource request reflect the same real task count fresh ingest would produce - _Requirements: 1.3_ - _Boundary: snapshot_load.c_ - [x] 2.3 (P) Close the worker's non-default launch-type coverage gap - Write a white-box test proving the worker's per-host launch dispatch correctly defers to an MPI launch type's own remote-launch mechanism instead of fanning the job out itself - Write a white-box test proving the same dispatch correctly fans a default-launch-type job out across every allocated host itself - Write a white-box test proving the local-launch path's MPI command-line construction builds the correct host list and command, without requiring a real MPI runtime in the test environment - Observable: all three launch-type dispatch tests pass against the existing, unmodified worker launch code, closing a gap that previously had zero coverage - _Requirements: 4.4_ - _Boundary: test_worker_launch_type_dispatch.c_ - [ ] 3. Integration: verify the ingest wiring end-to-end - [x] 3.1 (P) Verify both live ingest call sites, snapshot-vs-fresh parity, and single-task-job no-regression - Write a white-box test proving fresh job-ingest's dispatched resource request reflects the real requested task count for a multi-task job - Write a white-box test proving snapshot-restore ingest derives the identical task count an equivalent fresh submission would, directly satisfying the comparative claim between the two paths - Write a test proving a job requesting exactly one task (or no explicit count) produces identical host-selection and dispatch behavior through this wiring as it did before this feature - Observable: all three tests pass, together proving both live call sites are wired correctly and that no job requesting one task observes any behavior change - _Requirements: 1.1, 1.3, 3.1, 3.2, 5.6_ - _Boundary: test_maxtasks_wiring_ingest.c_ - _Depends: 2.1, 2.2_ - [x] 3.2 Verify driver/simulation ingest parity, update-immunity, and no cache collision across task counts - Write a regression test proving job ingest driven through the simulation driver's own entry point reaches the identical shared ingest code fresh (real-driver) ingest uses, applying the fix identically with no driver/simulation divergence - Write a structural regression test proving that after a job is ingested and its resource request is then updated, the job's already-dispatched task count remains unchanged by that update - Write a test submitting two otherwise-identical jobs through fresh ingest that differ only in requested task count, confirming an unsatisfiable outcome cached for one does not produce a cache hit for the other - Observable: all three tests pass, proving real/simulated ingest parity, that an update never silently changes an already-dispatched task count, and that differing task counts never collide in the unsatisfiable-resource-request cache on the fresh-ingest path - _Requirements: 1.4, 5.4_ - _Boundary: test_maxtasks_wiring_ingest.c, test_maxtasks_wiring_sim_driver_parity.c (see Implementation Notes)_ - _Depends: 2.1, 2.2_ - [x] 3.3 (P) Retire the multi-host allocator characterization test's manual workaround - Replace the existing multi-host allocator test's manual task-count patch with real ingest of an equivalent multi-task job, exercising the same scenario end to end through the now-fixed wiring - Observable: the updated test passes while driving its scenario entirely through unmodified real ingest, with the workaround removed - _Requirements: 2.1, 2.2_ - _Boundary: test_parjob_e2e_multislot.c_ - _Depends: 2.1_ - [ ] 4. Validation: real multi-host end-to-end proof - [x] 4.1 Build the multi-host end-to-end test harness - Add a new test skill that deploys one controller/scheduler node plus two named virtual workers, each sized so neither alone can satisfy the scenario's requested task count - Follow the existing soak-test convention for pre-flight zombie-process/port checks and a pass/fail verdict line - Observable: the new harness deploys its fixture cluster and tears it down cleanly, with no scenario wired in yet - _Requirements: 4.1, 4.2, 4.3, 5.3_ - _Boundary: xce-test-multihost_ - _Depends: 2.1, 2.2_ - [x] 4.2 Prove a plain multi-task job is genuinely scheduled and completes across multiple hosts - Submit a real multi-task job through normal job submission (not a manual-allocation bypass) and poll until it reaches a running state, then confirm it reaches completion - Confirm each allocated host's own log shows its share of the job's tasks actually starting - Observable: a PASS verdict confirms the scheduler's own decision spans both hosts and the job completes successfully on both - _Requirements: 4.1, 4.2, 4.3, 5.3_ - [x] 4.3 Prove a multi-task array-job element receives identical wiring and completes across multiple hosts - Submit a multi-task job as one element of an array-job submission and repeat the same running-then-complete verification as 4.2 - Confirm the array element's per-element task count wiring matches what an equivalent non-array submission would receive - Observable: a PASS verdict for the array-job scenario, with the same task-count wiring and multi-host completion as the non-array case - _Requirements: 5.5_ - [ ] 5. Validation: full non-regression confirmation - [x] 5.1 Run and confirm the full existing scheduler, controller, and worker test suites - Re-run the complete existing suites and confirm every assertion still passes unmodified except for the intentional test changes made in this feature's own tasks - Observable: a clean full-suite pass recorded as this feature's non-regression evidence - _Requirements: 2.3, 5.1_ - [x] 5.2 Run and confirm the existing HA soak shows no regression - Re-run the existing sequential and chaotic HA soak unmodified and confirm no new job loss, divergence, or liveness stall attributable to this feature - Observable: a clean soak pass recorded as this feature's HA non-regression evidence - _Requirements: 5.2_ ## Implementation Notes - Task 4.1's `xce-test-multihost` harness reuses a fixed, wipe-and-recreate deploy directory (`xce-test-ha`'s convention) instead of design.md's stated "fresh, uniquely-named cluster instance; no shared state across runs" bullet (`xce-test-resources`'s convention). Reason: task 4.1's own text requires "the existing soak-test convention for pre-flight zombie-process/port checks," and a meaningful pre-flight check needs a stable, known directory/port to check against — `xce-test-resources`'s fresh-mktemp-per-run model has no such check at all (confirmed: it has none) because one would be moot there. The "no shared state" intent is preserved functionally: `deploy()` unconditionally wipes and recreates the directory on every invocation, verified clean (zero cross-run residue) across two consecutive live runs. No production code or requirement scope changed. - Task 3.2's driver/simulation-parity test was split into a second new file, `src/scheduler/testsuite/test_maxtasks_wiring_sim_driver_parity.c` (registered separately in `meson.build`), rather than living inside `test_maxtasks_wiring_ingest.c` as originally scoped. Reason: proving this specific claim requires white-box `#include`-ing `sim_loop.c`/`core_events.c` to reach their real `static` translators (the only way to avoid a mirrored test that can't catch a real translator regression — confirmed via an independent mutation-test reproduction during review), but `test_maxtasks_wiring_ingest.c` already links the full real `scheduler_srcs` object list for its own task-3.1 test, and the two techniques collide at link time (duplicate symbols) in one translation unit. No production code or requirement scope changed; this is a file-boundary split only, driven by a real toolchain constraint discovered during implementation. - Task 2.1 surfaced an out-of-boundary side effect: `src/scheduler/testsuite/test_repl_breakpoints_inspection.c:735`'s `test_info_job_reports_state_queue_priority` hardcodes an incidental `"Tasks: min = 0, max = 0"` byproduct of its fixture's `parse_select()` failing on an unregistered `"type"` resource (which previously left the primary's task-count fields at memset-zero, before `apply_job_task_count()` existed). Now correctly overwritten to `(1,1)` per Requirement 1.2. Before task 5.1 is considered complete, change that assertion's expected substring to `"Tasks: min = 1, max = 1"`. (Applied in task 5.1, commit c454f065.) - Tasks 2.1 and 2.2 each added their own focused white-box test directly into the pre-existing scheduler test files for the call sites they wired (`test_core_events.c`: `test_core_build_job_wires_real_task_count_into_resreq_primary`, `test_core_build_job_default_task_count_still_one`; `test_snapshot_save_load.c`: `test_snapshot_load_wires_real_task_count_into_resreq_primary`) rather than deferring all ingest-level test coverage to task 3.1's new `test_maxtasks_wiring_ingest.c` file. design.md's File Structure Plan doesn't list these two existing files under "Modified Files." Reason: each one-line production wiring change needed its own colocated, immediate proof (matching this codebase's own established convention of a test living alongside the function it directly modifies); task 3.1's later file added complementary cross-call-site comparative and behavioral-parity coverage that neither of these narrower tests attempted (see task 3.1's own text). No production code or requirement scope changed; feature-level validation (`/kiro-validate-impl`) confirmed both additions are correct, non-redundant, and appropriately scoped.