docs(specs): record xce-node-manager interactions with service and provisioning specs

xce-node-manager: a "Cross-Spec Interactions" section (pending a
requirements revision; approved criteria unchanged) with six points: a
supervision/probe module shared with service-supervision, node-manager
components vs service jobs, a lease-based provisioner role, role
declaration and worker stop on decommission for provisioned hosts, a
host-loss grace period longer than worker restart plus rejoin, and a future
service proxy under the reverse-proxy role. design.md Open Questions points
to it.

The service-supervision, host-failure-job-recovery, elastic-hosts,
resource-provisioner and service-endpoint briefs cross-reference the
matching points; the roadmap lists the xce-node-manager revision.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2026-10-10 07:46:24 +00:00

4.9 KiB

Raw Blame History

Brief: service-supervision

(/kiro-discovery, 2026-10-10, "Long-running services" phase 1.)

Problem

Users and admins want to run long-lived services on XCE next to batch work: model servers, notebooks, databases and web apps for users; license servers, workflow engines and monitoring agents for admins. Today a job that exits or hangs simply ends or keeps running unnoticed: nothing restarts it and nothing checks that it still works.

Current State

A job runs in a systemd transient unit xcecd-worker-<jobid> with no Restart= (src/worker/src/xcecd_cgroups.c); the xcecd_worker process forks the job and reports its exit (xcecd_worker.c:467-523); the controller ends it DONE or EXIT (job.c:2937-2951).
No health, liveness or readiness checks exist.
Nothing kills an indefinitely running job unless the user sets -W (RuntimeMaxUSec only when > 0), so running forever already works.
A running job writes a usage WAL record every 10 s; checkpoints keep the WAL bounded (no growth with job age).

Desired Outcome

An application profile can declare kind service with:
- a restart policy: always / on-failure / never, backoff (initial, maximum), and a give-up rule (max restarts in a window);
- a liveness probe: command, TCP connect or HTTP GET, with interval, timeout, failure threshold and initial delay;
- a readiness probe (same forms), reported but not yet used for routing (service-endpoint);
- whether the service is rerunnable after host loss (default yes, host-failure-job-recovery).
The worker supervises each service task on its host: runs the probes locally, restarts the process in place on exit or failed liveness (keeping its allocation), applies backoff, and gives up per policy (the job then ends EXIT with the reason, or is requeued if the policy says so).
The worker reports only changes to the controller: ready / not ready, restarted (count, last reason), gave up. Users see state, restart count and last failure in job details and xcecli.
Stopping a service is the normal job kill; a service never ends DONE on its own unless its policy is never and it exits 0.
Service jobs are excluded from no-op expectations of batch jobs: no run limit by default, not cleaned up while running.

Approach

Supervision lives in xcecd_worker (already the job's parent): a restart loop around the launch with backoff, probe timers on the worker's event loop, and change-only status reports over the existing worker→controller channel. The profile's service section flows to the worker through the job's effective values (job-config-merge). Using systemd Restart= was considered and rejected for v1: it would rerun xcecd_worker itself (including profile pre-exec) and hide restarts from XCE.

Scope

In: profile service section (kind, restart policy, probes, rerunnable); worker restart loop and probes; change-only health reports and their persistence; display (state, restarts, last failure); xcecli/REST submission of service jobs through profiles; tests incl. xcecd_virtual simulation of probes and restarts.
Out: replicas, scaling, replacement on other hosts, rolling update (service-replica-set); endpoints and ports (service-endpoint); host failure (host-failure-job-recovery).

Boundary Candidates

Profile schema for services (with application-profile).
Worker supervision loop and probe runner.
Health/restart reporting and persistence (controller).
Display surfaces.

Out of Boundary

Any decision that needs a cluster-wide view (more replicas, new hosts).
No replica is special: every replica's own worker checks it (user decision 2026-10-10), so there is no head and no promotion.

Upstream / Downstream

Upstream: application-profile, job-config-merge (effective values), host-failure-job-recovery (host loss), worker launch path.
Downstream: service-replica-set (replaces replicas that gave up, gates rollouts on readiness), service-endpoint (readiness).

Existing Spec Touchpoints

Extends: application-profile (service section of the profile).
Adjacent: job-sandbox-execution (execution backends must support the restart loop), controller-job-cleanup, xce-node-manager (supervises XCE's own daemons; never job processes). Restart policy, backoff, give-up and probes (incl. the Node Manager's TCP port check) should come from one shared supervision/probe module so both layers behave the same (xce-node-manager requirements.md, "Cross-Spec Interactions" 1). Infrastructure services that XCE does not need to run itself are service jobs, not Node Manager roles (ibid. 2).

Constraints

Health reports must be change-only (no per-probe WAL traffic).
Real xcecd needs systemd/dbus to verify; xcecd_virtual must simulate probe outcomes and restarts for CI.
Persistence of health state: field completeness (persistence.md).

4.9 KiB Raw Blame History