docs(specs): discovery for long-running service jobs
New briefs: host-failure-job-recovery (jobs on a lost host are requeued or end EXIT after a grace period; stale copies fenced on rejoin; manual requeue), service-supervision (profile kind service: restart policy and liveness/readiness probes run by each replica's own worker, in-place restart), service-replica-set (controller-held service object owning independent replica jobs; event-driven replacement, scaling and rolling update; no head agent, no promotion), service-endpoint (per-replica ports, endpoints of ready replicas), job-resize (backlog, elastic jobs). application-profile brief gains the service section; roadmap lists the new specs, the existing-spec updates and the backlog item. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
253d92340f
commit
664ba565cc
7 changed files with 420 additions and 0 deletions
|
|
@ -34,6 +34,15 @@ decisions):**
|
|||
still taken from the profile at dispatch time; requirements decide whether
|
||||
dispatch freezes them on the job record.
|
||||
|
||||
**Update 2026-10-10 (`/kiro-discovery`, long-running services):** a profile
|
||||
may declare **kind `service`** with a service section: restart policy
|
||||
(always / on-failure / never, backoff, give-up), liveness and readiness
|
||||
probes (command / TCP / HTTP), named ports, default replica count, spread
|
||||
rule and rollout settings. The section is defined with this spec and
|
||||
consumed by `service-supervision` (worker probes and restarts),
|
||||
`service-replica-set` (replicas, scaling, rolling update) and
|
||||
`service-endpoint` (ports, endpoints).
|
||||
|
||||
## Problem
|
||||
XCE users run many kinds of applications: trusted in-house codes, untrusted
|
||||
scripts, vendor binaries, MPI solvers. Each needs a different way of running:
|
||||
|
|
|
|||
86
.kiro/specs/host-failure-job-recovery/brief.md
Normal file
86
.kiro/specs/host-failure-job-recovery/brief.md
Normal file
|
|
@ -0,0 +1,86 @@
|
|||
# Brief: host-failure-job-recovery
|
||||
|
||||
(`/kiro-discovery`, 2026-10-10, "Long-running services" phase 1. A
|
||||
prerequisite for services, and a gap for batch jobs today.)
|
||||
|
||||
## Problem
|
||||
When a host dies, the jobs running on it are never recovered. Users see them
|
||||
as RUN indefinitely, their resources stay charged, and nothing requeues or
|
||||
fails them. The only exception is a host that comes back: when its worker
|
||||
rejoins, the jobs it reports as "unrecognized" are reconciled. Every
|
||||
long-running service needs its replicas recovered after a host failure, but
|
||||
batch users are hurt by this already.
|
||||
|
||||
## Current State
|
||||
- `handle_worker_leave()` / `mark_worker_unavail()`
|
||||
(`src/controller/worker.c:897,921`) mark the host UNAVAIL, record
|
||||
`dead_since` and set its slots to 0. They do not touch the host's jobs.
|
||||
- `controller/sched.c:631` says so:
|
||||
`FIXME: need to handle job status in this worker`.
|
||||
- Reconciliation exists only when a worker rejoins (`resync.c:692-710`): a
|
||||
job it reports as "unrecognized" goes back to PEND. The rejoin check times
|
||||
out after 15 s and then simply ends, leaving the jobs as they are
|
||||
(`resync.c:259-290`).
|
||||
- Job states `RUNKWN`/`UNKWN`/`TRANSIENT` are defined (`types.h:382-405`) but
|
||||
no code sets them.
|
||||
- `JOB_REQUEUE` exists only as an enum value; the apiserver's requeue control
|
||||
is commented out (`jobs.go:2366-2382`). There is no rerunnable flag.
|
||||
|
||||
## Desired Outcome
|
||||
- After a configurable grace period with the host unreachable, each job that
|
||||
ran on it is moved to an "unknown" state and then resolved by policy:
|
||||
- requeued (PEND, new attempt) if the job is rerunnable (job option or
|
||||
application profile default);
|
||||
- otherwise EXIT with a clear reason ("host <h> lost").
|
||||
- A multi-host job that loses one of its hosts is resolved as a whole (its
|
||||
surviving tasks are stopped). A service replica is an ordinary job: it is
|
||||
resolved like any other, and `service-replica-set` replaces it.
|
||||
- If the host comes back with the job still running after the job was
|
||||
resolved, the stale copy is killed (fencing by attempt number).
|
||||
- Resources charged to the lost job are released; limits and quotas recover.
|
||||
- Users and admins see the lost-host reason and the attempt count;
|
||||
`xcecli`/REST can requeue a job manually (`JOB_REQUEUE`).
|
||||
- Behaviour is identical after controller failover (facts in the WAL).
|
||||
|
||||
## Approach
|
||||
Controller-side, primary-only, trigger-only timer (delaymgr rule:
|
||||
no blocking) that walks jobs of hosts unavailable longer than the grace
|
||||
period, commits a "job lost" fact per job and applies the policy through the
|
||||
existing commit-then-apply seam. The worker's rejoin resync gains a check
|
||||
that kills jobs whose attempt number is stale.
|
||||
|
||||
## Scope
|
||||
- **In**: grace period config; unknown state; rerunnable flag (job and
|
||||
profile default) and requeue-on-host-loss; EXIT with reason otherwise;
|
||||
multi-host job handling; attempt numbers and stale-copy fencing on rejoin;
|
||||
manual requeue (REST, `xcecli`); display; WAL/snapshot field completeness;
|
||||
sim/live scheduler consistency for released resources.
|
||||
- **Out**: requeue on job exit codes (`REQUEUE_EXIT_VALUES`, possible
|
||||
follow-up); checkpoint/restart migration; replacing service replicas
|
||||
(`service-replica-set`).
|
||||
|
||||
## Boundary Candidates
|
||||
- Host-loss detection and grace period (controller).
|
||||
- Job-lost policy (requeue / exit) and attempt numbering.
|
||||
- Stale-copy fencing on worker rejoin.
|
||||
- Manual requeue control (apiserver, xcecli).
|
||||
|
||||
## Out of Boundary
|
||||
- Restarting a process on the same host (`service-supervision`).
|
||||
- Keeping a service's replica count (`service-replica-set`).
|
||||
|
||||
## Upstream / Downstream
|
||||
- **Upstream**: worker liveness (`worker.c`), worker rejoin resync
|
||||
(`worker-rejoin-resync`), commit/apply seam (`raft-apply-decoupling`).
|
||||
- **Downstream**: `service-supervision`, `service-replica-set`,
|
||||
`job-dependency` (lost jobs end as EXIT).
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
- **Extends**: none.
|
||||
- **Adjacent**: `worker-rejoin-resync`, `controller-job-cleanup`,
|
||||
`scheduler-preemption` (requeue semantics).
|
||||
|
||||
## Constraints
|
||||
- HA/persistence: full 3-phase workflow; additive Cap'n Proto; the new facts
|
||||
round-trip through WAL and snapshot (`/xce-audit-invariants`).
|
||||
- The timer must never block the delaymgr (`ha-async-io.md`).
|
||||
66
.kiro/specs/job-resize/brief.md
Normal file
66
.kiro/specs/job-resize/brief.md
Normal file
|
|
@ -0,0 +1,66 @@
|
|||
# Brief: job-resize
|
||||
|
||||
(`/kiro-discovery`, 2026-10-10, "Long-running services"; **backlog, not on
|
||||
the services path**: services use independent replica jobs instead
|
||||
(`service-replica-set`), so this stays an elastic-job feature. LSF
|
||||
precedent: resizable jobs, `bresize`, autoresizable jobs; Admin Guide p. 339.)
|
||||
|
||||
## Problem
|
||||
A running parallel job's allocation is fixed at dispatch. Elastic
|
||||
applications (e.g. elastic MPI or task-farm frameworks) cannot grow onto
|
||||
free hosts or give hosts back without being killed and resubmitted.
|
||||
|
||||
## Current State
|
||||
- Allocation is decided once by the scheduler and committed at dispatch
|
||||
(`scheduler-dispatch-commit`); job modify (`JOB_MODIFY`) is not
|
||||
implemented.
|
||||
- Job arrays have a fixed size; nothing adds or removes tasks of a running
|
||||
job.
|
||||
|
||||
## Desired Outcome
|
||||
- A running parallel job can **grow** (request K more tasks; the scheduler
|
||||
places them under the job's constraints, e.g. one per host, and the
|
||||
controller adds them to the allocation and starts them) and **shrink**
|
||||
(release named tasks or hosts; their processes are stopped first).
|
||||
- A grow request pends until it can be placed and can be cancelled; the
|
||||
job's min/max task range bounds every resize.
|
||||
- Quotas, limits and accounting follow the current allocation.
|
||||
- Requests come from the job's owner (REST/xcecli) and from the job itself
|
||||
(an in-job API for elastic frameworks).
|
||||
- Resizes survive controller failover (facts in the WAL) and the scheduler
|
||||
simulator reproduces them.
|
||||
|
||||
## Approach
|
||||
A resize request is a pending allocation request attached to a running job;
|
||||
the scheduler treats a grow like a parallel-job placement of K tasks with
|
||||
the job's span constraints (`parjob-placement`) and existing hosts excluded
|
||||
or preferred per policy; the controller commits the grown/shrunk allocation
|
||||
and notifies the job's workers.
|
||||
|
||||
## Scope
|
||||
- **In**: grow/shrink requests (REST, xcecli, in-job API), scheduling of
|
||||
grow requests, allocation commit and worker launch/stop,
|
||||
accounting, persistence, simulator support.
|
||||
- **Out**: changing per-task resource sizes (vertical resize); resizing
|
||||
serial jobs; resize-driven autoscaling policies.
|
||||
|
||||
## Boundary Candidates
|
||||
- Resize request model and API.
|
||||
- Scheduler placement of grow requests.
|
||||
- Controller allocation change and worker task start/stop.
|
||||
|
||||
## Out of Boundary
|
||||
- Deciding when to resize (the user or the application).
|
||||
|
||||
## Upstream / Downstream
|
||||
- **Upstream**: `parjob-placement` (span constraints), scheduler allocation,
|
||||
`host-failure-job-recovery` (lost tasks).
|
||||
- **Downstream**: elastic-job frameworks; none of the service specs.
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
- **Extends**: none.
|
||||
- **Adjacent**: `parjob-placement`, `scheduler-preemption`,
|
||||
`job-sandbox-multinode` (remote task launch).
|
||||
|
||||
## Constraints
|
||||
- HA/persistence and scheduler equivalence: full 3-phase workflow.
|
||||
59
.kiro/specs/service-endpoint/brief.md
Normal file
59
.kiro/specs/service-endpoint/brief.md
Normal file
|
|
@ -0,0 +1,59 @@
|
|||
# Brief: service-endpoint
|
||||
|
||||
(`/kiro-discovery`, 2026-10-10, "Long-running services" phase 2.)
|
||||
|
||||
## Problem
|
||||
Clients of a service need a stable way to reach its ready replicas. Today
|
||||
users only see exec host names; there are no ports, no host addresses and no
|
||||
registry, so every service invents its own discovery.
|
||||
|
||||
## Current State
|
||||
- No port or network resource, endpoint, DNS or load-balancing concept;
|
||||
`xce_worker_t` has no IP field and the host API marks `ip_addr` as
|
||||
unsupported (`models/hosts.go:43`).
|
||||
- Readiness will be reported by workers (`service-supervision`).
|
||||
|
||||
## Desired Outcome
|
||||
- A service profile declares named ports (fixed or "any free"); each replica
|
||||
is allocated its ports on its host like a resource, so two replicas never
|
||||
collide; the job sees its ports in environment variables.
|
||||
- Workers report host addresses; a service has an **endpoint**: its name
|
||||
resolves to the list of `host:port` of ready replicas (readiness from
|
||||
`service-supervision`), updated as replicas become ready or not.
|
||||
- Users and clients read endpoints through REST and `xcecli` (and a simple
|
||||
query interface usable from scripts); access follows RBAC (job view
|
||||
scope).
|
||||
- Out of v1 but kept possible: DNS records, a built-in proxy or load
|
||||
balancer.
|
||||
|
||||
## Approach
|
||||
Ports become a per-host consumable resource the scheduler allocates;
|
||||
readiness changes already reach the controller; the controller (or the
|
||||
apiserver from pushed state) maintains the endpoint table per service.
|
||||
|
||||
## Scope
|
||||
- **In**: port declarations and allocation, host address reporting,
|
||||
endpoint table from readiness, REST/xcecli endpoint queries, RBAC.
|
||||
- **Out**: DNS, proxy/load balancing, TLS, ingress from outside the
|
||||
cluster.
|
||||
|
||||
## Boundary Candidates
|
||||
- Port resource and allocation (scheduler, worker).
|
||||
- Host address reporting.
|
||||
- Endpoint table and query surfaces.
|
||||
|
||||
## Out of Boundary
|
||||
- Health probing itself (`service-supervision`).
|
||||
|
||||
## Upstream / Downstream
|
||||
- **Upstream**: `service-supervision` (readiness), `application-profile`
|
||||
(port declarations), resource model (`resource-accounting.md`).
|
||||
- **Downstream**: `service-replica-set` (a service's endpoint covers all its
|
||||
replica jobs; rollouts keep it populated).
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
- **Extends**: `application-profile` (ports in the service section).
|
||||
- **Adjacent**: `resource-distribution-mapping`.
|
||||
|
||||
## Constraints
|
||||
- Endpoint updates are change-driven; no per-probe traffic.
|
||||
100
.kiro/specs/service-replica-set/brief.md
Normal file
100
.kiro/specs/service-replica-set/brief.md
Normal file
|
|
@ -0,0 +1,100 @@
|
|||
# Brief: service-replica-set
|
||||
|
||||
(`/kiro-discovery`, 2026-10-10, "Long-running services" phase 2. Replaces
|
||||
the earlier `service-head-agent` idea by user decision: no special replica,
|
||||
no head, no promotion. Kubernetes analogue: Deployment/ReplicaSet with
|
||||
independent pods.)
|
||||
|
||||
## Problem
|
||||
A service with N replicas needs N healthy replicas kept running: a replica
|
||||
that gave up or was lost with its host must be replaced elsewhere, the
|
||||
count must be changeable, and a new version must roll out without
|
||||
downtime. Per-replica supervision (`service-supervision`) cannot do this
|
||||
alone because no worker sees the other replicas.
|
||||
|
||||
## Current State
|
||||
- Phase 1 gives per-replica probes and in-place restart on each worker
|
||||
(`service-supervision`) and host-loss recovery that ends or requeues a
|
||||
job (`host-failure-job-recovery`).
|
||||
- There is no service object, no replica grouping and no rollout concept;
|
||||
job arrays are fixed-size.
|
||||
|
||||
## Desired Outcome
|
||||
- A **service** is a controller-held object: name, owner, project, queue,
|
||||
application profile (kind `service`), desired replica count, version
|
||||
(the profile/command revision it runs) and the list of its **replica
|
||||
jobs**.
|
||||
- **Replicas are independent, ordinary jobs.** Each is scheduled, limited,
|
||||
charged, recovered and RBAC-checked like any job; the scheduler spreads a
|
||||
service's replicas across hosts (anti-affinity rule from the profile,
|
||||
e.g. at most one per host, best effort or strict).
|
||||
- **No replica is special**: each replica's worker checks it
|
||||
(`service-supervision`); there is no head, no promotion, no epoch.
|
||||
- **Event-driven rules in the controller** (no reconcile loop, no new
|
||||
timers on the delaymgr), applied when a fact commits:
|
||||
- a replica job ends (gave up, lost with its host, killed outside the
|
||||
service) while the service wants more → submit a replacement replica
|
||||
job, with backoff when replacements keep failing (a crash-looping
|
||||
service pends with a reason instead of churning);
|
||||
- the desired count changes (scale through REST/xcecli) → submit or kill
|
||||
replica jobs (scale-down picks not-ready replicas first, then newest);
|
||||
- **rolling update** (new version) → replace replicas in batches limited
|
||||
by max-unavailable / max-surge, each step advancing when the new
|
||||
replicas report ready; pause, resume and rollback; a progress deadline
|
||||
marks a stuck rollout.
|
||||
- Deleting a service kills its replicas; killing one replica directly just
|
||||
triggers its replacement.
|
||||
- Users see service state: desired / running / ready counts, version,
|
||||
rollout progress, last events; `xcecli services` commands and REST.
|
||||
- Everything survives controller failover (service object and rollout state
|
||||
are WAL facts); the scheduler simulator handles replica jobs as jobs.
|
||||
|
||||
## Approach
|
||||
A new controller entity with its own WAL facts and snapshot frames; rules
|
||||
attached to the existing commit-then-apply seam (replica end, readiness
|
||||
change, desired-count change, version change). Replica jobs carry their
|
||||
service name and replica index. Considered and set aside: a head agent per
|
||||
service (needs promotion and fencing) and a separate cluster-wide service
|
||||
manager (one more HA component).
|
||||
|
||||
## Scope
|
||||
- **In**: service object and API (create, scale, update, pause/resume,
|
||||
rollback, delete, list/show) in REST, xcecli and libxce; replica jobs
|
||||
linked to the service; replacement with backoff; scaling; rolling update;
|
||||
spread rule (with the scheduler); display; persistence (WAL, snapshot,
|
||||
field completeness); RBAC on services (follows job permissions and
|
||||
project scopes).
|
||||
- **Out**: metric-based autoscaling; canary/blue-green beyond rolling
|
||||
update; endpoints (`service-endpoint`); per-replica probes
|
||||
(`service-supervision`); elastic resize of one job (`job-resize`).
|
||||
|
||||
## Boundary Candidates
|
||||
- Service entity, facts and API.
|
||||
- Replacement and scaling rules.
|
||||
- Rollout state machine.
|
||||
- Spread placement rule (scheduler).
|
||||
- Display surfaces.
|
||||
|
||||
## Out of Boundary
|
||||
- How a replica is probed or restarted on its host.
|
||||
- Endpoint table mechanics.
|
||||
|
||||
## Upstream / Downstream
|
||||
- **Upstream**: `service-supervision` (readiness, gave-up events),
|
||||
`host-failure-job-recovery` (lost replicas), `application-profile`,
|
||||
`job-config-merge`, `project-config` (quotas cap replicas).
|
||||
- **Downstream**: `service-endpoint` (the endpoint covers all replica
|
||||
jobs), future autoscaling.
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
- **Extends**: `application-profile` (default replicas, spread rule,
|
||||
rollout settings in the service section).
|
||||
- **Adjacent**: `job-array-single-wal-record` (grouped submission
|
||||
precedent), `controller-job-cleanup` (finished replica jobs).
|
||||
|
||||
## Constraints
|
||||
- Rules run inside the commit/apply path and must stay bounded and
|
||||
non-blocking (`ha-async-io.md`); replacement backoff without new delaymgr
|
||||
work beyond trigger-only timers.
|
||||
- HA/persistence: full 3-phase workflow; `/xce-audit-invariants` for the new
|
||||
entity.
|
||||
90
.kiro/specs/service-supervision/brief.md
Normal file
90
.kiro/specs/service-supervision/brief.md
Normal file
|
|
@ -0,0 +1,90 @@
|
|||
# Brief: service-supervision
|
||||
|
||||
(`/kiro-discovery`, 2026-10-10, "Long-running services" phase 1.)
|
||||
|
||||
## Problem
|
||||
Users and admins want to run long-lived services on XCE next to batch work:
|
||||
model servers, notebooks, databases and web apps for users; license servers,
|
||||
workflow engines and monitoring agents for admins. Today a job that exits or
|
||||
hangs simply ends or keeps running unnoticed: nothing restarts it and nothing
|
||||
checks that it still works.
|
||||
|
||||
## Current State
|
||||
- A job runs in a systemd transient unit `xcecd-worker-<jobid>` with no
|
||||
`Restart=` (`src/worker/src/xcecd_cgroups.c`); the `xcecd_worker` process
|
||||
forks the job and reports its exit (`xcecd_worker.c:467-523`); the
|
||||
controller ends it DONE or EXIT (`job.c:2937-2951`).
|
||||
- No health, liveness or readiness checks exist.
|
||||
- Nothing kills an indefinitely running job unless the user sets `-W`
|
||||
(`RuntimeMaxUSec` only when > 0), so running forever already works.
|
||||
- A running job writes a usage WAL record every 10 s; checkpoints keep the
|
||||
WAL bounded (no growth with job age).
|
||||
|
||||
## Desired Outcome
|
||||
- An application profile can declare **kind `service`** with:
|
||||
- a restart policy: `always` / `on-failure` / `never`, backoff (initial,
|
||||
maximum), and a give-up rule (max restarts in a window);
|
||||
- a liveness probe: command, TCP connect or HTTP GET, with interval,
|
||||
timeout, failure threshold and initial delay;
|
||||
- a readiness probe (same forms), reported but not yet used for routing
|
||||
(`service-endpoint`);
|
||||
- whether the service is rerunnable after host loss (default yes,
|
||||
`host-failure-job-recovery`).
|
||||
- The worker supervises each service task on its host: runs the probes
|
||||
locally, restarts the process in place on exit or failed liveness (keeping
|
||||
its allocation), applies backoff, and gives up per policy (the job then
|
||||
ends EXIT with the reason, or is requeued if the policy says so).
|
||||
- The worker reports only changes to the controller: ready / not ready,
|
||||
restarted (count, last reason), gave up. Users see state, restart count
|
||||
and last failure in job details and `xcecli`.
|
||||
- Stopping a service is the normal job kill; a service never ends DONE on
|
||||
its own unless its policy is `never` and it exits 0.
|
||||
- Service jobs are excluded from no-op expectations of batch jobs: no run
|
||||
limit by default, not cleaned up while running.
|
||||
|
||||
## Approach
|
||||
Supervision lives in `xcecd_worker` (already the job's parent): a restart
|
||||
loop around the launch with backoff, probe timers on the worker's event
|
||||
loop, and change-only status reports over the existing worker→controller
|
||||
channel. The profile's service section flows to the worker through the job's
|
||||
effective values (`job-config-merge`). Using systemd `Restart=` was
|
||||
considered and rejected for v1: it would rerun `xcecd_worker` itself
|
||||
(including profile pre-exec) and hide restarts from XCE.
|
||||
|
||||
## Scope
|
||||
- **In**: profile service section (kind, restart policy, probes, rerunnable);
|
||||
worker restart loop and probes; change-only health reports and their
|
||||
persistence; display (state, restarts, last failure); xcecli/REST
|
||||
submission of service jobs through profiles; tests incl. `xcecd_virtual`
|
||||
simulation of probes and restarts.
|
||||
- **Out**: replicas, scaling, replacement on other hosts, rolling update
|
||||
(`service-replica-set`); endpoints and ports
|
||||
(`service-endpoint`); host failure (`host-failure-job-recovery`).
|
||||
|
||||
## Boundary Candidates
|
||||
- Profile schema for services (with `application-profile`).
|
||||
- Worker supervision loop and probe runner.
|
||||
- Health/restart reporting and persistence (controller).
|
||||
- Display surfaces.
|
||||
|
||||
## Out of Boundary
|
||||
- Any decision that needs a cluster-wide view (more replicas, new hosts).
|
||||
- No replica is special: every replica's own worker checks it (user
|
||||
decision 2026-10-10), so there is no head and no promotion.
|
||||
|
||||
## Upstream / Downstream
|
||||
- **Upstream**: `application-profile`, `job-config-merge` (effective
|
||||
values), `host-failure-job-recovery` (host loss), worker launch path.
|
||||
- **Downstream**: `service-replica-set` (replaces replicas that gave up,
|
||||
gates rollouts on readiness), `service-endpoint` (readiness).
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
- **Extends**: `application-profile` (service section of the profile).
|
||||
- **Adjacent**: `job-sandbox-execution` (execution backends must support the
|
||||
restart loop), `controller-job-cleanup`.
|
||||
|
||||
## Constraints
|
||||
- Health reports must be change-only (no per-probe WAL traffic).
|
||||
- Real `xcecd` needs systemd/dbus to verify; `xcecd_virtual` must simulate
|
||||
probe outcomes and restarts for CI.
|
||||
- Persistence of health state: field completeness (`persistence.md`).
|
||||
|
|
@ -73,6 +73,15 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
|
|||
- [ ] job-config-merge — NEW 2026-10-09 (`/kiro-discovery`, Path E "configuration layering"; ref LSF 10.1 Admin Guide pp. 314-316, 435-437, 561-563, 643, 660). One merge function in `src/lib` turning the job's own values + application profile + queue policy + project into effective values (resreq incl. `span`, task range, limits, pre/post-exec, env). One owner per attribute (user decision; `.kiro/steering/job-config-layering.md`): job overrides a profile attribute only if it is *default* (for `select` only by adding ANDed terms); queue ceilings and project quotas only check (reject if never satisfiable, pend if full now). **Live** merge (user decision): WAL keeps inputs + names; controller/apiserver validate at submit; scheduler computes effective values and re-evaluates pending jobs on config change. Queue is policy only: queue `res_req` and `pre_exec`/`post_exec` deprecated (pre/post-exec moves to the profile), dropped queue limits (cpulimit/memlimit/runlimit/task limit) wired. Brief: `.kiro/specs/job-config-merge/brief.md`. Dependencies: none.
|
||||
- [ ] project-config — NEW 2026-10-09 (configuration layering). Projects control resource ownership, v1 = quota: project definitions in cluster config (members, priority, cluster default), default-project resolution, submit access check, scheduler `pgmgr` loaded from config so `PROJECTS`/`PER_PROJECT` limits work end to end, per-project usage display. Brief: `.kiro/specs/project-config/brief.md`. Dependencies: none.
|
||||
|
||||
- [ ] host-failure-job-recovery — NEW 2026-10-10 (`/kiro-discovery`, "Long-running services" phase 1; also a batch gap: RUN jobs on a dead host stay RUN forever, `controller/sched.c:631` FIXME). After a grace period, jobs on a lost host go to an unknown state, then are requeued (rerunnable, job option or profile default) or end EXIT "host lost"; multi-host jobs resolved as a whole; attempt numbers fence a stale copy when the host returns; manual requeue (REST/xcecli). Brief: `.kiro/specs/host-failure-job-recovery/brief.md`. Dependencies: none.
|
||||
- [ ] service-supervision — NEW 2026-10-10 (Long-running services phase 1). Application profile kind `service`: restart policy (always/on-failure/never, backoff, give-up), liveness and readiness probes (command/TCP/HTTP) run by each replica's own worker (no special replica), in-place restart, change-only health reports, display. Brief: `.kiro/specs/service-supervision/brief.md`. Dependencies: application-profile, host-failure-job-recovery.
|
||||
- [ ] service-replica-set — NEW 2026-10-10 (Long-running services phase 2; user decision: no head agent, no promotion). Controller-held service object owning independent replica jobs; event-driven rules on commit (replace ended replicas with backoff, scale, rolling update with max-unavailable/surge, pause/resume/rollback); spread rule; REST/xcecli `services`. Brief: `.kiro/specs/service-replica-set/brief.md`. Dependencies: service-supervision, host-failure-job-recovery, application-profile.
|
||||
- [ ] service-endpoint — NEW 2026-10-10 (Long-running services phase 2). Named ports allocated per replica as host resources, host addresses reported by workers, endpoint = `host:port` of ready replicas from readiness changes, REST/xcecli queries; DNS/proxy later. Brief: `.kiro/specs/service-endpoint/brief.md`. Dependencies: service-supervision, service-replica-set.
|
||||
|
||||
## Existing Spec Updates (2026-10-10 long-running services discovery)
|
||||
- [ ] application-profile — profile kind `service` section: restart policy, probes, ports, default replicas, spread rule, rollout settings (consumed by service-supervision, service-replica-set, service-endpoint).
|
||||
- [ ] job-config-merge — service attributes follow the ownership rule (profile owns them; the job may override only `default` ones, e.g. replica count).
|
||||
|
||||
## Existing Spec Updates (2026-10-09 configuration layering discovery)
|
||||
- [ ] application-profile — the profile carries the job's resource requirement (select/rusage/`span`/same) and task limits (min/default/max); "resolved and frozen at submit" replaced by the live merge of `job-config-merge` (brief updated). Dependencies: job-config-merge.
|
||||
- [ ] parjob-placement — `span` read from the effective resreq; span constraints in the fixed-N planner; range search bisects over the planner's allowed counts (block: multiples; ptile: multiples + max); LSF 10.1 findings recorded in its requirements.md notes; `span[stripe]` candidate. Dependencies: job-config-merge (for `span` from profiles; job-level `-R` span can land first).
|
||||
|
|
@ -82,6 +91,7 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
|
|||
|
||||
## Future Specs / Backlog (dependency order)
|
||||
- [ ] project-guarantee — NEW 2026-10-09 (configuration layering, backlog). Project-owned guarantees (resource amount or host pool), borrowing of idle guaranteed capacity, reclaim through preemption. Brief: `.kiro/specs/project-guarantee/brief.md`. Dependencies: project-config, scheduler-preemption.
|
||||
- [ ] job-resize — NEW 2026-10-10 (from long-running services discovery; not on the services path). Grow/shrink a running parallel job's allocation (LSF resizable jobs, `bresize`) for elastic applications. Brief: `.kiro/specs/job-resize/brief.md`. Dependencies: parjob-placement, host-failure-job-recovery.
|
||||
- [ ] project-delegated-membership — NEW 2026-10-10 (from project-config requirements, user decision: future). Project managers edit their project's (and subtree's) members at runtime without an admin config push; needs a durable store that coexists with `xceadmin` config pushes (e.g. a separate delegated member list) and audit. Dependencies: project-config, rbac-persistence.
|
||||
- [ ] lsf-config-migration — NEW 2026-10-09 (user idea, backlog; no brief yet). A tool (likely an `xceadmin` subcommand) that converts LSF configuration (`lsb.queues`, `lsb.applications`, `lsb.resources` Limit sections, `lsb.users`/`lsb.hosts` groups, `lsb.params` defaults, project settings) into XCE `configs/*.yaml`, with a report of what does not map: queue `RES_REQ` (XCE queues are policy only; move into a profile or drop), LSF's full per-section merge rules vs XCE's simplified ones, compound/alternative resreqs, unsupported parameters. Dependencies: job-config-merge, application-profile, project-config, parjob-placement (span), admin-config-tool. Promote via `/kiro-discovery` when those land.
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue