docs(specs): discovery for long-running service jobs

New briefs: host-failure-job-recovery (jobs on a lost host are requeued or
end EXIT after a grace period; stale copies fenced on rejoin; manual
requeue), service-supervision (profile kind service: restart policy and
liveness/readiness probes run by each replica's own worker, in-place
restart), service-replica-set (controller-held service object owning
independent replica jobs; event-driven replacement, scaling and rolling
update; no head agent, no promotion), service-endpoint (per-replica ports,
endpoints of ready replicas), job-resize (backlog, elastic jobs).

application-profile brief gains the service section; roadmap lists the new
specs, the existing-spec updates and the backlog item.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2026-10-10 07:15:46 +00:00

4.8 KiB

Raw Blame History

Brief: service-replica-set

(/kiro-discovery, 2026-10-10, "Long-running services" phase 2. Replaces the earlier service-head-agent idea by user decision: no special replica, no head, no promotion. Kubernetes analogue: Deployment/ReplicaSet with independent pods.)

Problem

A service with N replicas needs N healthy replicas kept running: a replica that gave up or was lost with its host must be replaced elsewhere, the count must be changeable, and a new version must roll out without downtime. Per-replica supervision (service-supervision) cannot do this alone because no worker sees the other replicas.

Current State

Phase 1 gives per-replica probes and in-place restart on each worker (service-supervision) and host-loss recovery that ends or requeues a job (host-failure-job-recovery).
There is no service object, no replica grouping and no rollout concept; job arrays are fixed-size.

Desired Outcome

A service is a controller-held object: name, owner, project, queue, application profile (kind service), desired replica count, version (the profile/command revision it runs) and the list of its replica jobs.
Replicas are independent, ordinary jobs. Each is scheduled, limited, charged, recovered and RBAC-checked like any job; the scheduler spreads a service's replicas across hosts (anti-affinity rule from the profile, e.g. at most one per host, best effort or strict).
No replica is special: each replica's worker checks it (service-supervision); there is no head, no promotion, no epoch.
Event-driven rules in the controller (no reconcile loop, no new timers on the delaymgr), applied when a fact commits:
- a replica job ends (gave up, lost with its host, killed outside the service) while the service wants more → submit a replacement replica job, with backoff when replacements keep failing (a crash-looping service pends with a reason instead of churning);
- the desired count changes (scale through REST/xcecli) → submit or kill replica jobs (scale-down picks not-ready replicas first, then newest);
- rolling update (new version) → replace replicas in batches limited by max-unavailable / max-surge, each step advancing when the new replicas report ready; pause, resume and rollback; a progress deadline marks a stuck rollout.
Deleting a service kills its replicas; killing one replica directly just triggers its replacement.
Users see service state: desired / running / ready counts, version, rollout progress, last events; xcecli services commands and REST.
Everything survives controller failover (service object and rollout state are WAL facts); the scheduler simulator handles replica jobs as jobs.

Approach

A new controller entity with its own WAL facts and snapshot frames; rules attached to the existing commit-then-apply seam (replica end, readiness change, desired-count change, version change). Replica jobs carry their service name and replica index. Considered and set aside: a head agent per service (needs promotion and fencing) and a separate cluster-wide service manager (one more HA component).

Scope

In: service object and API (create, scale, update, pause/resume, rollback, delete, list/show) in REST, xcecli and libxce; replica jobs linked to the service; replacement with backoff; scaling; rolling update; spread rule (with the scheduler); display; persistence (WAL, snapshot, field completeness); RBAC on services (follows job permissions and project scopes).
Out: metric-based autoscaling; canary/blue-green beyond rolling update; endpoints (service-endpoint); per-replica probes (service-supervision); elastic resize of one job (job-resize).

Boundary Candidates

Service entity, facts and API.
Replacement and scaling rules.
Rollout state machine.
Spread placement rule (scheduler).
Display surfaces.

Out of Boundary

How a replica is probed or restarted on its host.
Endpoint table mechanics.

Upstream / Downstream

Upstream: service-supervision (readiness, gave-up events), host-failure-job-recovery (lost replicas), application-profile, job-config-merge, project-config (quotas cap replicas).
Downstream: service-endpoint (the endpoint covers all replica jobs), future autoscaling.

Existing Spec Touchpoints

Extends: application-profile (default replicas, spread rule, rollout settings in the service section).
Adjacent: job-array-single-wal-record (grouped submission precedent), controller-job-cleanup (finished replica jobs).

Constraints

Rules run inside the commit/apply path and must stay bounded and non-blocking (ha-async-io.md); replacement backoff without new delaymgr work beyond trigger-only timers.
HA/persistence: full 3-phase workflow; /xce-audit-invariants for the new entity.

4.8 KiB Raw Blame History