docs(specs): record xce-node-manager interactions with service and provisioning specs

xce-node-manager: a "Cross-Spec Interactions" section (pending a
requirements revision; approved criteria unchanged) with six points: a
supervision/probe module shared with service-supervision, node-manager
components vs service jobs, a lease-based provisioner role, role
declaration and worker stop on decommission for provisioned hosts, a
host-loss grace period longer than worker restart plus rejoin, and a future
service proxy under the reverse-proxy role. design.md Open Questions points
to it.

The service-supervision, host-failure-job-recovery, elastic-hosts,
resource-provisioner and service-endpoint briefs cross-reference the
matching points; the roadmap lists the xce-node-manager revision.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-10-10 07:46:24 +00:00
parent 922188b5d7
commit a68aac1a2b
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
8 changed files with 83 additions and 5 deletions

View file

@ -63,7 +63,12 @@ provisioned hosts; template-based host config; a decommission control.
## Existing Spec Touchpoints
- **Extends**: none.
- **Adjacent**: `worker-join-refactor`, `peer-authentication-hardening`,
`xce-node-manager`.
`xce-node-manager`: a provisioned host boots with the Node Manager and the
`worker` role, declared by its host template and written into the Node
Manager's configuration by the instance's boot script; on decommission
the Node Manager stops the worker (or drops its role) instead of
restarting it, and the Controller refuses a rejoin from a decommissioned
host (`xce-node-manager` requirements.md, "Cross-Spec Interactions" 4).
## Constraints
- HA/persistence: full 3-phase workflow; the scheduler must see the same

View file

@ -78,7 +78,12 @@ that kills jobs whose attempt number is stale.
## Existing Spec Touchpoints
- **Extends**: none.
- **Adjacent**: `worker-rejoin-resync`, `controller-job-cleanup`,
`scheduler-preemption` (requeue semantics).
`scheduler-preemption` (requeue semantics), `xce-node-manager`: jobs run
in systemd units and survive a worker daemon restart, so the grace period
must exceed the Node Manager's worker restart plus rejoin time and a
worker crash must never requeue or fail jobs; optionally the Node Manager
reports "worker restarting" so the grace period does not start
(`xce-node-manager` requirements.md, "Cross-Spec Interactions" 5).
## Constraints
- HA/persistence: full 3-phase workflow; additive Cap'n Proto; the new facts

View file

@ -93,7 +93,10 @@ controller facts.
## Existing Spec Touchpoints
- **Extends**: none (`src/autoscaler` is prior art only).
- **Adjacent**: `component-failover-adoption`, `admin-config-tool`
(provider config).
(provider config), `xce-node-manager`: a `provisioner` node role keeps the
daemon running; which instance is active is decided by the
Controller-granted lease, not by the Node Manager (`xce-node-manager`
requirements.md, "Cross-Spec Interactions" 3).
## Constraints
- Provider scripts never run in the controller or scheduler process.

View file

@ -53,7 +53,10 @@ apiserver from pushed state) maintains the endpoint table per service.
## Existing Spec Touchpoints
- **Extends**: `application-profile` (ports in the service section).
- **Adjacent**: `resource-distribution-mapping`.
- **Adjacent**: `resource-distribution-mapping`, `xce-node-manager`: a
later service proxy or load balancer would run under the Node Manager's
`reverse-proxy` role, its routing fed by the endpoint table
(`xce-node-manager` requirements.md, "Cross-Spec Interactions" 6).
## Constraints
- Endpoint updates are change-driven; no per-probe traffic.

View file

@ -81,7 +81,13 @@ considered and rejected for v1: it would rerun `xcecd_worker` itself
## Existing Spec Touchpoints
- **Extends**: `application-profile` (service section of the profile).
- **Adjacent**: `job-sandbox-execution` (execution backends must support the
restart loop), `controller-job-cleanup`.
restart loop), `controller-job-cleanup`, `xce-node-manager` (supervises
XCE's own daemons; never job processes). Restart policy, backoff, give-up
and probes (incl. the Node Manager's TCP port check) should come from one
shared supervision/probe module so both layers behave the same
(`xce-node-manager` requirements.md, "Cross-Spec Interactions" 1).
Infrastructure services that XCE does not need to run itself are service
jobs, not Node Manager roles (ibid. 2).
## Constraints
- Health reports must be change-only (no per-probe WAL traffic).

View file

@ -481,6 +481,15 @@ flowchart TB
- Validation checkpoint: after Phase 2, run the soak-harness scenario in Testing Strategy's E2E section before proceeding to Phase 3/4, since it is the closest thing to a direct regression test for bug-1760 itself.
## Open Questions / Risks
- **Cross-spec interactions (2026-10-10), pending a requirements revision:**
shared supervision/probe module with `service-supervision`; a lease-based
`provisioner` role for `resource-provisioner`; stopping the worker on
decommission and role declaration for provisioned hosts (`elastic-hosts`);
host-loss grace period longer than worker restart + rejoin
(`host-failure-job-recovery`); a future service proxy under the
`reverse-proxy` role (`service-endpoint`). See requirements.md,
"Cross-Spec Interactions".
- The exact Node Role Manifest file format (YAML to match `src/configs/*.yaml`'s existing convention, vs. a simpler key-value form) is left to implementation — an additive, appropriately implementation-level decision, not a design decision, since Data Models already fixes its logical shape.
- The precise heartbeat interval and "no progress" threshold (Requirements 4.2, 4.3) are implementation-time tuning decisions, following the same posture the superseded spec already took for its own liveness thresholds.
- `tools/appimage/entry.sh` bundles `haproxy` today with no dispatch case reaching it at all — this feature is what first wires it up (via `xce-node-manager` invoking it directly for the reverse-proxy role), which is a pre-existing gap this design closes as a side effect rather than a pre-existing behavior it must preserve.

View file

@ -99,3 +99,49 @@ Open design questions to resolve during requirements/design phases (do not assum
1. Where existing deployment tooling documents starting the Controller, Scheduler Process, Worker Process, or Apiserver Process as an independent, manually-managed process, that tooling shall be updated to reflect that the Node Manager now starts these components according to each node's assigned role.
2. Where an existing spec or test tooling depends on a component being started or managed independently of the Node Manager, that dependency shall be identified and reconciled with this feature's behavior rather than left silently incompatible.
3. The observable behavior of the Controller, Scheduler Process, Worker Process, and Apiserver Process toward jobs, queues, configuration, and cluster membership shall remain unchanged by this feature while each is running.
## Cross-Spec Interactions (2026-10-10, pending requirements revision)
Recorded during the long-running services and auto resource provisioning
discoveries (`.kiro/steering/roadmap.md`). These are **not yet acceptance
criteria**: the approved requirements and design above are unchanged until a
revision (`/kiro-spec-requirements xce-node-manager`) folds them in or
rejects them.
1. **Two supervision layers, shared mechanics.** The Node Manager supervises
XCE's own components; the worker supervises user jobs, including service
replicas (`service-supervision`). The Node Manager never supervises job
processes (consistent with Requirement 5). Restart policy with backoff,
give-up rule and probes (the Node Manager's TCP port check, the worker's
command/TCP/HTTP probes) should come from one shared supervision/probe
module so their semantics match.
2. **Infrastructure services boundary.** The Node Manager runs what XCE
itself needs, or what must run before or without a Controller (XCE
components, the reverse proxy). Everything else, including admin
infrastructure services such as license servers or monitoring agents,
runs as XCE service jobs (`service-replica-set`), scheduled with quotas,
RBAC and accounting.
3. **Roles for lease-based components.** `resource-provisioner` adds a
daemon: the Node Manager needs a `provisioner` role that only keeps the
process running; which instance is active is decided by a
Controller-granted lease, not by the Node Manager (unlike the Scheduler's
primary-bound start/stop). Future XCE daemons should follow the same
"role + lease" pattern.
4. **Provisioned hosts and decommission** (`elastic-hosts`). A cloud-burst
host boots with the Node Manager and the `worker` role, declared by its
host template and written into the Node Manager's configuration by the
instance's boot script (one concrete answer to "how a node's role
assignment is declared"). On decommission, the Node Manager must stop
the worker on that node (or drop its role) instead of restarting it, and
the Controller refuses a rejoin from a decommissioned host.
5. **Worker restart vs. host loss** (`host-failure-job-recovery`). Jobs run
in systemd units and survive a worker daemon restart; the restarted
worker re-reports them on rejoin. The host-loss grace period must exceed
the Node Manager's worker restart plus rejoin time, so a worker crash
never requeues or fails jobs (service replicas included). Optional: the
Node Manager reports "worker restarting" so the Controller does not start
the grace period.
6. **Future service proxy** (`service-endpoint`). A service proxy or load
balancer, out of scope for `service-endpoint` v1, would naturally run
under the Node Manager's `reverse-proxy` role, its routing fed by the
endpoint table.

View file

@ -86,6 +86,7 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
## Existing Spec Updates (2026-10-10 auto resource provisioning discovery)
- [ ] resource-distribution-mapping — choose its shared-pool scope (option 2) so a cluster-wide pool (licenses) can be declared in `resources.yaml`; prerequisite of elastic-shared-resources.
- [ ] peer-authentication-hardening — a provisioner role with its allowed operations (provisioning facts, shared-total updates, expected hosts).
- [ ] xce-node-manager — revise requirements (approved 2026-09-03) for the cross-spec interactions recorded 2026-10-10 in its requirements.md: shared supervision/probe module with service-supervision, a lease-based `provisioner` role, worker stop on decommission and role declaration for provisioned hosts, grace period vs worker restart, future service proxy under `reverse-proxy`.
## Existing Spec Updates (2026-10-10 long-running services discovery)
- [ ] application-profile — profile kind `service` section: restart policy, probes, ports, default replicas, spread rule, rollout settings (consumed by service-supervision, service-replica-set, service-endpoint).