lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/.kiro/specs/resreq-maxtasks-wiring/requirements.md
Rongsong Shen 799a71ecb1
feat(resreq-maxtasks-wiring): add task-count derivation helper 
Adds apply_job_task_count() to overwrite a standalone primary resreq's
minTasks/maxTasks with a job's real requested task count, no-op for
COMPOUND/ALTER nested primaries, with a defensive min>max clamp.
Closes the gap that lets scheduler-allocation's multi-host allocator
ever be reached (bug-1770). Wiring at live ingest call sites follows
in subsequent tasks.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
2026-09-06 03:26:32 +00:00

9.3 KiB
Raw Blame History

Requirements Document

Introduction

Cluster users can already request a job spanning multiple tasks through the existing job-submission API and CLI (a minimum/maximum task count, e.g. xcecli jobs submit -n 4,6), and that requested count is correctly carried all the way into the scheduler's own job record. However, the parsed resource request the allocator dispatch actually consults is unconditionally fixed at a single task regardless of what was requested, so no job today can ever be evaluated for, or allocated across, multiple hosts -- it silently behaves as a single-task job no matter what a user asked for. This defect (bug-1770) was discovered while validating the scheduler-allocation feature, which already implements, tests, and soak-validates real multi-host allocation logic that currently has no way to ever run in production. This feature wires a job's actually-requested task count through to the resource request the allocator reads, so that a job requiring more tasks than a single host can provide is genuinely evaluated for, and where appropriate allocated across, multiple hosts -- and so that such an allocation decision results in the job actually launching and running correctly.

Boundary Context

  • In scope: standalone primary resource requests (a job's own resource request, not one nested inside a compound resource request); the existing job-level task-count submission path; fresh job ingest and job ingest from a saved snapshot loaded by the simulation/replay driver; end-to-end verification that a resulting multi-host allocation decision launches and runs correctly on the worker side.
  • Out of scope: resource-request-string-level task-count syntax (span[...]), which remains parsed and silently discarded exactly as today, unchanged by this feature; primary resource requests nested inside a compound resource request, which have a separate, already-tracked gap (bug-1768) that makes their resource checking meaningless regardless of task count; the pre-existing snapshot-restore resource-request-fingerprint gap (bug-1772), tracked separately; a sibling fingerprint gap in a job-construction code path confirmed to have no live caller in this codebase today (bug-1774, dead-code-only, no current production impact); per-host live memory-usage reconciliation for a multi-host job, which is an existing, documented limitation this feature does not change; a job's resource-request update never actually taking effect for scheduling purposes for any field, not only task count (bug-1773) — this feature's own task-count guarantee holds across an update for that same structural reason, not because this feature actively re-wires anything on update.
  • Adjacent expectations: this feature depends on the scheduler-allocation feature's already-implemented multi-host allocation and unsatisfiable-resource-request cache continuing to behave as specified; it also depends on the controller and worker's existing multi-host job-launch capability (remote task dispatch and message-passing-style launch), which predates this feature and has not previously been exercised by a real scheduler decision.

Requirements

Requirement 1: Job-level task count reaches the resource request the allocator dispatches on

Objective: As a cluster user, I want the task count I request for my job to determine how the scheduler evaluates and allocates it, so that requesting multiple tasks isn't silently ignored.

Acceptance Criteria

  1. When a job is submitted with a requested minimum and maximum task count greater than one, the Scheduler shall set the job's standalone primary resource request's minimum and maximum task count to the requested values.
  2. When a job is submitted without an explicit task count, or with a task count of one, the Scheduler shall set the job's standalone primary resource request's minimum and maximum task count to one, matching today's behavior.
  3. When a job is reconstructed from a saved snapshot loaded by the simulation/replay driver, the Scheduler shall derive the same standalone primary resource request task count it would have derived from a fresh submission of that same job.
  4. When a job's resource request is updated, the Scheduler shall continue dispatching that job using the task count it derived at ingest, regardless of the update.
  5. If a job's requested minimum task count would exceed its requested maximum task count at the point the Scheduler derives the primary resource request, the Scheduler shall not let an invalid task-count pair reach the allocator dispatch.

Requirement 2: Multi-host allocation is genuinely reachable when needed

Objective: As a cluster user submitting a job that needs more task slots than any single host can provide, I want the scheduler to actually consider spanning multiple hosts, so that my job isn't wrongly treated as unsatisfiable or restricted to a single host.

Acceptance Criteria

  1. When a standalone primary resource request's maximum task count is greater than one, the Scheduler shall evaluate candidate allocation plans ranging from a single host up to as many hosts as the requested task count could need.
  2. When a candidate allocation plan can be satisfied by available hosts, the Scheduler shall commit a job allocation spanning the hosts in that plan, with each host's own task count recorded correctly.
  3. When a standalone primary resource request's maximum task count is exactly one, the Scheduler shall dispatch the job through the same single-host allocation path used today, unaffected by this feature.

Requirement 3: No regression to existing single-task job behavior

Objective: As a cluster operator, I want every job that does not request more than one task to behave exactly as it does today, so that this fix introduces no regression to existing production traffic.

Acceptance Criteria

  1. The Scheduler shall produce identical host-selection and allocation results for a single-task job before and after this feature, given identical worker and resource state.
  2. While no job in a cluster requests more than one task, the Scheduler's observable dispatch, allocation, and pending behavior shall be unchanged by this feature.
  3. The Scheduler shall not change how a primary resource request nested inside a compound resource request is evaluated or allocated.

Requirement 4: A committed multi-host allocation results in correct job execution

Objective: As a cluster user, I want a job allocated across multiple hosts to actually launch and run correctly, so that requesting multiple tasks results in real distributed execution rather than only a scheduling decision.

Acceptance Criteria

  1. When the Scheduler commits a job allocation spanning more than one host, the Controller shall ensure every allocated host has the information it needs to start its own share of the job's tasks.
  2. When a job's allocation spans more than one host, the Worker shall start the job's tasks on every allocated host.
  3. When a multi-host job's tasks all start successfully, the Scheduler shall report the job as running through the same job-status mechanism used for single-host jobs.
  4. If a job's submission specifies a non-default launch type (for example, an MPI launch type), the Worker shall launch the job using that launch type's existing mechanism across all allocated hosts.

Requirement 5: Non-regression and end-to-end validation

Objective: As a cluster operator, I want this feature validated against the same regression standard as the scheduler-allocation feature it depends on, so that I can trust it in production, in HA failover scenarios, and in the simulation/replay tooling used for offline tuning and debugging.

Acceptance Criteria

  1. The existing scheduler, controller, and worker test suites shall continue to pass unmodified in assertions, adjusted only where they directly exercised the always-one-task-count behavior this feature replaces.
  2. A full HA soak (sequential and chaotic regression) shall show no new job loss, divergence, or liveness stall attributable to this feature. (The HA soak restarts and rejoins a scheduler through the same live ingest path fresh submission uses; it does not exercise the simulation/replay driver's snapshot-loading path, which is validated separately by AC6.)
  3. A real multi-task job shall be driven through the full path from submission to worker execution at least once as part of validating this feature, including confirmation that it reaches a running state and completes.
  4. For every job ingest path where this feature computes a resource-request fingerprint correctly, a job with a different requested task count than another otherwise-identical job shall not produce an unsatisfiable-resource-request cache hit against that other job's cached outcome. (The simulation/replay driver's snapshot-loading path has a separately-tracked, pre-existing fingerprint gap, bug-1772, that this feature does not fix; that path is explicitly excluded from this guarantee until bug-1772 is resolved.)
  5. A real multi-task job submitted as part of an array job submission shall be driven through the full path from submission to worker execution at least once, and shall receive the same per-element task count wiring as a non-array job would.
  6. A real multi-task job's saved snapshot, loaded through the simulation/replay driver, shall be shown to derive the same standalone primary resource request task count as it would through fresh submission.