Rongsong Shen
a68aac1a2b
xce-node-manager: a "Cross-Spec Interactions" section (pending a requirements revision; approved criteria unchanged) with six points: a supervision/probe module shared with service-supervision, node-manager components vs service jobs, a lease-based provisioner role, role declaration and worker stop on decommission for provisioned hosts, a host-loss grace period longer than worker restart plus rejoin, and a future service proxy under the reverse-proxy role. design.md Open Questions points to it. The service-supervision, host-failure-job-recovery, elastic-hosts, resource-provisioner and service-endpoint briefs cross-reference the matching points; the roadmap lists the xce-node-manager revision. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
96 lines
4.9 KiB
Markdown
96 lines
4.9 KiB
Markdown
# Brief: service-supervision
|
|
|
|
(`/kiro-discovery`, 2026-10-10, "Long-running services" phase 1.)
|
|
|
|
## Problem
|
|
Users and admins want to run long-lived services on XCE next to batch work:
|
|
model servers, notebooks, databases and web apps for users; license servers,
|
|
workflow engines and monitoring agents for admins. Today a job that exits or
|
|
hangs simply ends or keeps running unnoticed: nothing restarts it and nothing
|
|
checks that it still works.
|
|
|
|
## Current State
|
|
- A job runs in a systemd transient unit `xcecd-worker-<jobid>` with no
|
|
`Restart=` (`src/worker/src/xcecd_cgroups.c`); the `xcecd_worker` process
|
|
forks the job and reports its exit (`xcecd_worker.c:467-523`); the
|
|
controller ends it DONE or EXIT (`job.c:2937-2951`).
|
|
- No health, liveness or readiness checks exist.
|
|
- Nothing kills an indefinitely running job unless the user sets `-W`
|
|
(`RuntimeMaxUSec` only when > 0), so running forever already works.
|
|
- A running job writes a usage WAL record every 10 s; checkpoints keep the
|
|
WAL bounded (no growth with job age).
|
|
|
|
## Desired Outcome
|
|
- An application profile can declare **kind `service`** with:
|
|
- a restart policy: `always` / `on-failure` / `never`, backoff (initial,
|
|
maximum), and a give-up rule (max restarts in a window);
|
|
- a liveness probe: command, TCP connect or HTTP GET, with interval,
|
|
timeout, failure threshold and initial delay;
|
|
- a readiness probe (same forms), reported but not yet used for routing
|
|
(`service-endpoint`);
|
|
- whether the service is rerunnable after host loss (default yes,
|
|
`host-failure-job-recovery`).
|
|
- The worker supervises each service task on its host: runs the probes
|
|
locally, restarts the process in place on exit or failed liveness (keeping
|
|
its allocation), applies backoff, and gives up per policy (the job then
|
|
ends EXIT with the reason, or is requeued if the policy says so).
|
|
- The worker reports only changes to the controller: ready / not ready,
|
|
restarted (count, last reason), gave up. Users see state, restart count
|
|
and last failure in job details and `xcecli`.
|
|
- Stopping a service is the normal job kill; a service never ends DONE on
|
|
its own unless its policy is `never` and it exits 0.
|
|
- Service jobs are excluded from no-op expectations of batch jobs: no run
|
|
limit by default, not cleaned up while running.
|
|
|
|
## Approach
|
|
Supervision lives in `xcecd_worker` (already the job's parent): a restart
|
|
loop around the launch with backoff, probe timers on the worker's event
|
|
loop, and change-only status reports over the existing worker→controller
|
|
channel. The profile's service section flows to the worker through the job's
|
|
effective values (`job-config-merge`). Using systemd `Restart=` was
|
|
considered and rejected for v1: it would rerun `xcecd_worker` itself
|
|
(including profile pre-exec) and hide restarts from XCE.
|
|
|
|
## Scope
|
|
- **In**: profile service section (kind, restart policy, probes, rerunnable);
|
|
worker restart loop and probes; change-only health reports and their
|
|
persistence; display (state, restarts, last failure); xcecli/REST
|
|
submission of service jobs through profiles; tests incl. `xcecd_virtual`
|
|
simulation of probes and restarts.
|
|
- **Out**: replicas, scaling, replacement on other hosts, rolling update
|
|
(`service-replica-set`); endpoints and ports
|
|
(`service-endpoint`); host failure (`host-failure-job-recovery`).
|
|
|
|
## Boundary Candidates
|
|
- Profile schema for services (with `application-profile`).
|
|
- Worker supervision loop and probe runner.
|
|
- Health/restart reporting and persistence (controller).
|
|
- Display surfaces.
|
|
|
|
## Out of Boundary
|
|
- Any decision that needs a cluster-wide view (more replicas, new hosts).
|
|
- No replica is special: every replica's own worker checks it (user
|
|
decision 2026-10-10), so there is no head and no promotion.
|
|
|
|
## Upstream / Downstream
|
|
- **Upstream**: `application-profile`, `job-config-merge` (effective
|
|
values), `host-failure-job-recovery` (host loss), worker launch path.
|
|
- **Downstream**: `service-replica-set` (replaces replicas that gave up,
|
|
gates rollouts on readiness), `service-endpoint` (readiness).
|
|
|
|
## Existing Spec Touchpoints
|
|
- **Extends**: `application-profile` (service section of the profile).
|
|
- **Adjacent**: `job-sandbox-execution` (execution backends must support the
|
|
restart loop), `controller-job-cleanup`, `xce-node-manager` (supervises
|
|
XCE's own daemons; never job processes). Restart policy, backoff, give-up
|
|
and probes (incl. the Node Manager's TCP port check) should come from one
|
|
shared supervision/probe module so both layers behave the same
|
|
(`xce-node-manager` requirements.md, "Cross-Spec Interactions" 1).
|
|
Infrastructure services that XCE does not need to run itself are service
|
|
jobs, not Node Manager roles (ibid. 2).
|
|
|
|
## Constraints
|
|
- Health reports must be change-only (no per-probe WAL traffic).
|
|
- Real `xcecd` needs systemd/dbus to verify; `xcecd_virtual` must simulate
|
|
probe outcomes and restarts for CI.
|
|
- Persistence of health state: field completeness (`persistence.md`).
|