Rongsong Shen
664ba565cc
New briefs: host-failure-job-recovery (jobs on a lost host are requeued or end EXIT after a grace period; stale copies fenced on rejoin; manual requeue), service-supervision (profile kind service: restart policy and liveness/readiness probes run by each replica's own worker, in-place restart), service-replica-set (controller-held service object owning independent replica jobs; event-driven replacement, scaling and rolling update; no head agent, no promotion), service-endpoint (per-replica ports, endpoints of ready replicas), job-resize (backlog, elastic jobs). application-profile brief gains the service section; roadmap lists the new specs, the existing-spec updates and the backlog item. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
66 lines
2.8 KiB
Markdown
66 lines
2.8 KiB
Markdown
# Brief: job-resize
|
|
|
|
(`/kiro-discovery`, 2026-10-10, "Long-running services"; **backlog, not on
|
|
the services path**: services use independent replica jobs instead
|
|
(`service-replica-set`), so this stays an elastic-job feature. LSF
|
|
precedent: resizable jobs, `bresize`, autoresizable jobs; Admin Guide p. 339.)
|
|
|
|
## Problem
|
|
A running parallel job's allocation is fixed at dispatch. Elastic
|
|
applications (e.g. elastic MPI or task-farm frameworks) cannot grow onto
|
|
free hosts or give hosts back without being killed and resubmitted.
|
|
|
|
## Current State
|
|
- Allocation is decided once by the scheduler and committed at dispatch
|
|
(`scheduler-dispatch-commit`); job modify (`JOB_MODIFY`) is not
|
|
implemented.
|
|
- Job arrays have a fixed size; nothing adds or removes tasks of a running
|
|
job.
|
|
|
|
## Desired Outcome
|
|
- A running parallel job can **grow** (request K more tasks; the scheduler
|
|
places them under the job's constraints, e.g. one per host, and the
|
|
controller adds them to the allocation and starts them) and **shrink**
|
|
(release named tasks or hosts; their processes are stopped first).
|
|
- A grow request pends until it can be placed and can be cancelled; the
|
|
job's min/max task range bounds every resize.
|
|
- Quotas, limits and accounting follow the current allocation.
|
|
- Requests come from the job's owner (REST/xcecli) and from the job itself
|
|
(an in-job API for elastic frameworks).
|
|
- Resizes survive controller failover (facts in the WAL) and the scheduler
|
|
simulator reproduces them.
|
|
|
|
## Approach
|
|
A resize request is a pending allocation request attached to a running job;
|
|
the scheduler treats a grow like a parallel-job placement of K tasks with
|
|
the job's span constraints (`parjob-placement`) and existing hosts excluded
|
|
or preferred per policy; the controller commits the grown/shrunk allocation
|
|
and notifies the job's workers.
|
|
|
|
## Scope
|
|
- **In**: grow/shrink requests (REST, xcecli, in-job API), scheduling of
|
|
grow requests, allocation commit and worker launch/stop,
|
|
accounting, persistence, simulator support.
|
|
- **Out**: changing per-task resource sizes (vertical resize); resizing
|
|
serial jobs; resize-driven autoscaling policies.
|
|
|
|
## Boundary Candidates
|
|
- Resize request model and API.
|
|
- Scheduler placement of grow requests.
|
|
- Controller allocation change and worker task start/stop.
|
|
|
|
## Out of Boundary
|
|
- Deciding when to resize (the user or the application).
|
|
|
|
## Upstream / Downstream
|
|
- **Upstream**: `parjob-placement` (span constraints), scheduler allocation,
|
|
`host-failure-job-recovery` (lost tasks).
|
|
- **Downstream**: elastic-job frameworks; none of the service specs.
|
|
|
|
## Existing Spec Touchpoints
|
|
- **Extends**: none.
|
|
- **Adjacent**: `parjob-placement`, `scheduler-preemption`,
|
|
`job-sandbox-multinode` (remote task launch).
|
|
|
|
## Constraints
|
|
- HA/persistence and scheduler equivalence: full 3-phase workflow.
|