Rongsong Shen
664ba565cc
New briefs: host-failure-job-recovery (jobs on a lost host are requeued or end EXIT after a grace period; stale copies fenced on rejoin; manual requeue), service-supervision (profile kind service: restart policy and liveness/readiness probes run by each replica's own worker, in-place restart), service-replica-set (controller-held service object owning independent replica jobs; event-driven replacement, scaling and rolling update; no head agent, no promotion), service-endpoint (per-replica ports, endpoints of ready replicas), job-resize (backlog, elastic jobs). application-profile brief gains the service section; roadmap lists the new specs, the existing-spec updates and the backlog item. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
100 lines
4.8 KiB
Markdown
100 lines
4.8 KiB
Markdown
# Brief: service-replica-set
|
|
|
|
(`/kiro-discovery`, 2026-10-10, "Long-running services" phase 2. Replaces
|
|
the earlier `service-head-agent` idea by user decision: no special replica,
|
|
no head, no promotion. Kubernetes analogue: Deployment/ReplicaSet with
|
|
independent pods.)
|
|
|
|
## Problem
|
|
A service with N replicas needs N healthy replicas kept running: a replica
|
|
that gave up or was lost with its host must be replaced elsewhere, the
|
|
count must be changeable, and a new version must roll out without
|
|
downtime. Per-replica supervision (`service-supervision`) cannot do this
|
|
alone because no worker sees the other replicas.
|
|
|
|
## Current State
|
|
- Phase 1 gives per-replica probes and in-place restart on each worker
|
|
(`service-supervision`) and host-loss recovery that ends or requeues a
|
|
job (`host-failure-job-recovery`).
|
|
- There is no service object, no replica grouping and no rollout concept;
|
|
job arrays are fixed-size.
|
|
|
|
## Desired Outcome
|
|
- A **service** is a controller-held object: name, owner, project, queue,
|
|
application profile (kind `service`), desired replica count, version
|
|
(the profile/command revision it runs) and the list of its **replica
|
|
jobs**.
|
|
- **Replicas are independent, ordinary jobs.** Each is scheduled, limited,
|
|
charged, recovered and RBAC-checked like any job; the scheduler spreads a
|
|
service's replicas across hosts (anti-affinity rule from the profile,
|
|
e.g. at most one per host, best effort or strict).
|
|
- **No replica is special**: each replica's worker checks it
|
|
(`service-supervision`); there is no head, no promotion, no epoch.
|
|
- **Event-driven rules in the controller** (no reconcile loop, no new
|
|
timers on the delaymgr), applied when a fact commits:
|
|
- a replica job ends (gave up, lost with its host, killed outside the
|
|
service) while the service wants more → submit a replacement replica
|
|
job, with backoff when replacements keep failing (a crash-looping
|
|
service pends with a reason instead of churning);
|
|
- the desired count changes (scale through REST/xcecli) → submit or kill
|
|
replica jobs (scale-down picks not-ready replicas first, then newest);
|
|
- **rolling update** (new version) → replace replicas in batches limited
|
|
by max-unavailable / max-surge, each step advancing when the new
|
|
replicas report ready; pause, resume and rollback; a progress deadline
|
|
marks a stuck rollout.
|
|
- Deleting a service kills its replicas; killing one replica directly just
|
|
triggers its replacement.
|
|
- Users see service state: desired / running / ready counts, version,
|
|
rollout progress, last events; `xcecli services` commands and REST.
|
|
- Everything survives controller failover (service object and rollout state
|
|
are WAL facts); the scheduler simulator handles replica jobs as jobs.
|
|
|
|
## Approach
|
|
A new controller entity with its own WAL facts and snapshot frames; rules
|
|
attached to the existing commit-then-apply seam (replica end, readiness
|
|
change, desired-count change, version change). Replica jobs carry their
|
|
service name and replica index. Considered and set aside: a head agent per
|
|
service (needs promotion and fencing) and a separate cluster-wide service
|
|
manager (one more HA component).
|
|
|
|
## Scope
|
|
- **In**: service object and API (create, scale, update, pause/resume,
|
|
rollback, delete, list/show) in REST, xcecli and libxce; replica jobs
|
|
linked to the service; replacement with backoff; scaling; rolling update;
|
|
spread rule (with the scheduler); display; persistence (WAL, snapshot,
|
|
field completeness); RBAC on services (follows job permissions and
|
|
project scopes).
|
|
- **Out**: metric-based autoscaling; canary/blue-green beyond rolling
|
|
update; endpoints (`service-endpoint`); per-replica probes
|
|
(`service-supervision`); elastic resize of one job (`job-resize`).
|
|
|
|
## Boundary Candidates
|
|
- Service entity, facts and API.
|
|
- Replacement and scaling rules.
|
|
- Rollout state machine.
|
|
- Spread placement rule (scheduler).
|
|
- Display surfaces.
|
|
|
|
## Out of Boundary
|
|
- How a replica is probed or restarted on its host.
|
|
- Endpoint table mechanics.
|
|
|
|
## Upstream / Downstream
|
|
- **Upstream**: `service-supervision` (readiness, gave-up events),
|
|
`host-failure-job-recovery` (lost replicas), `application-profile`,
|
|
`job-config-merge`, `project-config` (quotas cap replicas).
|
|
- **Downstream**: `service-endpoint` (the endpoint covers all replica
|
|
jobs), future autoscaling.
|
|
|
|
## Existing Spec Touchpoints
|
|
- **Extends**: `application-profile` (default replicas, spread rule,
|
|
rollout settings in the service section).
|
|
- **Adjacent**: `job-array-single-wal-record` (grouped submission
|
|
precedent), `controller-job-cleanup` (finished replica jobs).
|
|
|
|
## Constraints
|
|
- Rules run inside the commit/apply path and must stay bounded and
|
|
non-blocking (`ha-async-io.md`); replacement backoff without new delaymgr
|
|
work beyond trigger-only timers.
|
|
- HA/persistence: full 3-phase workflow; `/xce-audit-invariants` for the new
|
|
entity.
|