myxce/.kiro/specs/resreq-maxtasks-wiring/tasks.md

Implementation Plan

• 1. Implement and test the task-count derivation helper

  • Overwrite a standalone (top-level) primary resource request's minimum/maximum task count with the job's real requested minimum/maximum task count
  • Leave a compound/alternative resource request's nested primary resource requests untouched (no-op) — only the job's own top-level standalone primary is in scope
  • Defensively clamp an invalid pair by raising the maximum to the minimum when the requested minimum exceeds the requested maximum, mirroring the apiserver's own clamp direction
  • Write unit tests covering: a standalone primary receiving the applied task count, a compound/alternative resreq being left untouched, and the minimum-exceeds-maximum clamp
  • Observable: a passing unit test suite demonstrates all three cases (override, no-op, clamp)
  • Requirements: 1.1, 1.2, 1.5, 3.3
  • Boundary: TaskCountWiring

• 2. Core: wire the derived task count into live scheduler ingest paths and close the adjacent worker launch-type gap

• 2.1 (P) Wire fresh job-ingest to apply the job's real requested task count

  • Call the new helper immediately after fresh job ingest's existing resource-request parse, using the job's real submitted minimum/maximum task count already in scope there
  • Observable: a job's dispatched resource request (the one the allocator actually reads) reflects the real requested task count immediately after fresh ingest, not the previous always-one-task default
  • Requirements: 1.1
  • Boundary: core_events.c (core_build_job)

• 2.2 (P) Wire simulation/replay snapshot-restore ingest to apply the same real task count

  • Call the new helper immediately after the snapshot-restore path's existing resource-request parse, using the job's already-restored minimum/maximum task count
  • Observable: a job reconstructed from a saved snapshot via the simulation/replay driver has its dispatched resource request reflect the same real task count fresh ingest would produce
  • Requirements: 1.3
  • Boundary: snapshot_load.c

• 2.3 (P) Close the worker's non-default launch-type coverage gap

  • Write a white-box test proving the worker's per-host launch dispatch correctly defers to an MPI launch type's own remote-launch mechanism instead of fanning the job out itself
  • Write a white-box test proving the same dispatch correctly fans a default-launch-type job out across every allocated host itself
  • Write a white-box test proving the local-launch path's MPI command-line construction builds the correct host list and command, without requiring a real MPI runtime in the test environment
  • Observable: all three launch-type dispatch tests pass against the existing, unmodified worker launch code, closing a gap that previously had zero coverage
  • Requirements: 4.4
  • Boundary: test_worker_launch_type_dispatch.c

• 3. Integration: verify the ingest wiring end-to-end

• 3.1 (P) Verify both live ingest call sites, snapshot-vs-fresh parity, and single-task-job no-regression

  • Write a white-box test proving fresh job-ingest's dispatched resource request reflects the real requested task count for a multi-task job
  • Write a white-box test proving snapshot-restore ingest derives the identical task count an equivalent fresh submission would, directly satisfying the comparative claim between the two paths
  • Write a test proving a job requesting exactly one task (or no explicit count) produces identical host-selection and dispatch behavior through this wiring as it did before this feature
  • Observable: all three tests pass, together proving both live call sites are wired correctly and that no job requesting one task observes any behavior change
  • Requirements: 1.1, 1.3, 3.1, 3.2, 5.6
  • Boundary: test_maxtasks_wiring_ingest.c
  • Depends: 2.1, 2.2

• 3.2 Verify driver/simulation ingest parity, update-immunity, and no cache collision across task counts

  • Write a regression test proving job ingest driven through the simulation driver's own entry point reaches the identical shared ingest code fresh (real-driver) ingest uses, applying the fix identically with no driver/simulation divergence
  • Write a structural regression test proving that after a job is ingested and its resource request is then updated, the job's already-dispatched task count remains unchanged by that update
  • Write a test submitting two otherwise-identical jobs through fresh ingest that differ only in requested task count, confirming an unsatisfiable outcome cached for one does not produce a cache hit for the other
  • Observable: all three tests pass, proving real/simulated ingest parity, that an update never silently changes an already-dispatched task count, and that differing task counts never collide in the unsatisfiable-resource-request cache on the fresh-ingest path
  • Requirements: 1.4, 5.4
  • Boundary: test_maxtasks_wiring_ingest.c, test_maxtasks_wiring_sim_driver_parity.c (see Implementation Notes)
  • Depends: 2.1, 2.2

• 3.3 (P) Retire the multi-host allocator characterization test's manual workaround

  • Replace the existing multi-host allocator test's manual task-count patch with real ingest of an equivalent multi-task job, exercising the same scenario end to end through the now-fixed wiring
  • Observable: the updated test passes while driving its scenario entirely through unmodified real ingest, with the workaround removed
  • Requirements: 2.1, 2.2
  • Boundary: test_parjob_e2e_multislot.c
  • Depends: 2.1

• 4. Validation: real multi-host end-to-end proof

• 4.1 Build the multi-host end-to-end test harness

  • Add a new test skill that deploys one controller/scheduler node plus two named virtual workers, each sized so neither alone can satisfy the scenario's requested task count
  • Follow the existing soak-test convention for pre-flight zombie-process/port checks and a pass/fail verdict line
  • Observable: the new harness deploys its fixture cluster and tears it down cleanly, with no scenario wired in yet
  • Requirements: 4.1, 4.2, 4.3, 5.3
  • Boundary: xce-test-multihost
  • Depends: 2.1, 2.2

• 4.2 Prove a plain multi-task job is genuinely scheduled and completes across multiple hosts

  • Submit a real multi-task job through normal job submission (not a manual-allocation bypass) and poll until it reaches a running state, then confirm it reaches completion
  • Confirm each allocated host's own log shows its share of the job's tasks actually starting
  • Observable: a PASS verdict confirms the scheduler's own decision spans both hosts and the job completes successfully on both
  • Requirements: 4.1, 4.2, 4.3, 5.3

• 4.3 Prove a multi-task array-job element receives identical wiring and completes across multiple hosts

  • Submit a multi-task job as one element of an array-job submission and repeat the same running-then-complete verification as 4.2
  • Confirm the array element's per-element task count wiring matches what an equivalent non-array submission would receive
  • Observable: a PASS verdict for the array-job scenario, with the same task-count wiring and multi-host completion as the non-array case
  • Requirements: 5.5

• 5. Validation: full non-regression confirmation

• 5.1 Run and confirm the full existing scheduler, controller, and worker test suites

  • Re-run the complete existing suites and confirm every assertion still passes unmodified except for the intentional test changes made in this feature's own tasks
  • Observable: a clean full-suite pass recorded as this feature's non-regression evidence
  • Requirements: 2.3, 5.1

• 5.2 Run and confirm the existing HA soak shows no regression

  • Re-run the existing sequential and chaotic HA soak unmodified and confirm no new job loss, divergence, or liveness stall attributable to this feature
  • Observable: a clean soak pass recorded as this feature's HA non-regression evidence
  • Requirements: 5.2

Implementation Notes

• Task 4.1's xce-test-multihost harness reuses a fixed, wipe-and-recreate deploy directory (xce-test-ha's convention) instead of design.md's stated "fresh, uniquely-named cluster instance; no shared state across runs" bullet (xce-test-resources's convention). Reason: task 4.1's own text requires "the existing soak-test convention for pre-flight zombie-process/port checks," and a meaningful pre-flight check needs a stable, known directory/port to check against — xce-test-resources's fresh-mktemp-per-run model has no such check at all (confirmed: it has none) because one would be moot there. The "no shared state" intent is preserved functionally: deploy() unconditionally wipes and recreates the directory on every invocation, verified clean (zero cross-run residue) across two consecutive live runs. No production code or requirement scope changed.
• Task 3.2's driver/simulation-parity test was split into a second new file, src/scheduler/testsuite/test_maxtasks_wiring_sim_driver_parity.c (registered separately in meson.build), rather than living inside test_maxtasks_wiring_ingest.c as originally scoped. Reason: proving this specific claim requires white-box #include-ing sim_loop.c/core_events.c to reach their real static translators (the only way to avoid a mirrored test that can't catch a real translator regression — confirmed via an independent mutation-test reproduction during review), but test_maxtasks_wiring_ingest.c already links the full real scheduler_srcs object list for its own task-3.1 test, and the two techniques collide at link time (duplicate symbols) in one translation unit. No production code or requirement scope changed; this is a file-boundary split only, driven by a real toolchain constraint discovered during implementation.
• Task 2.1 surfaced an out-of-boundary side effect: src/scheduler/testsuite/test_repl_breakpoints_inspection.c:735's test_info_job_reports_state_queue_priority hardcodes an incidental "Tasks: min = 0, max = 0" byproduct of its fixture's parse_select() failing on an unregistered "type" resource (which previously left the primary's task-count fields at memset-zero, before apply_job_task_count() existed). Now correctly overwritten to (1,1) per Requirement 1.2. Before task 5.1 is considered complete, change that assertion's expected substring to "Tasks: min = 1, max = 1".