docs(specs): record xce-node-manager interactions with service and provisioning specs
xce-node-manager: a "Cross-Spec Interactions" section (pending a requirements revision; approved criteria unchanged) with six points: a supervision/probe module shared with service-supervision, node-manager components vs service jobs, a lease-based provisioner role, role declaration and worker stop on decommission for provisioned hosts, a host-loss grace period longer than worker restart plus rejoin, and a future service proxy under the reverse-proxy role. design.md Open Questions points to it. The service-supervision, host-failure-job-recovery, elastic-hosts, resource-provisioner and service-endpoint briefs cross-reference the matching points; the roadmap lists the xce-node-manager revision. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
922188b5d7
commit
a68aac1a2b
8 changed files with 83 additions and 5 deletions
|
|
@ -63,7 +63,12 @@ provisioned hosts; template-based host config; a decommission control.
|
|||
## Existing Spec Touchpoints
|
||||
- **Extends**: none.
|
||||
- **Adjacent**: `worker-join-refactor`, `peer-authentication-hardening`,
|
||||
`xce-node-manager`.
|
||||
`xce-node-manager`: a provisioned host boots with the Node Manager and the
|
||||
`worker` role, declared by its host template and written into the Node
|
||||
Manager's configuration by the instance's boot script; on decommission
|
||||
the Node Manager stops the worker (or drops its role) instead of
|
||||
restarting it, and the Controller refuses a rejoin from a decommissioned
|
||||
host (`xce-node-manager` requirements.md, "Cross-Spec Interactions" 4).
|
||||
|
||||
## Constraints
|
||||
- HA/persistence: full 3-phase workflow; the scheduler must see the same
|
||||
|
|
|
|||
|
|
@ -78,7 +78,12 @@ that kills jobs whose attempt number is stale.
|
|||
## Existing Spec Touchpoints
|
||||
- **Extends**: none.
|
||||
- **Adjacent**: `worker-rejoin-resync`, `controller-job-cleanup`,
|
||||
`scheduler-preemption` (requeue semantics).
|
||||
`scheduler-preemption` (requeue semantics), `xce-node-manager`: jobs run
|
||||
in systemd units and survive a worker daemon restart, so the grace period
|
||||
must exceed the Node Manager's worker restart plus rejoin time and a
|
||||
worker crash must never requeue or fail jobs; optionally the Node Manager
|
||||
reports "worker restarting" so the grace period does not start
|
||||
(`xce-node-manager` requirements.md, "Cross-Spec Interactions" 5).
|
||||
|
||||
## Constraints
|
||||
- HA/persistence: full 3-phase workflow; additive Cap'n Proto; the new facts
|
||||
|
|
|
|||
|
|
@ -93,7 +93,10 @@ controller facts.
|
|||
## Existing Spec Touchpoints
|
||||
- **Extends**: none (`src/autoscaler` is prior art only).
|
||||
- **Adjacent**: `component-failover-adoption`, `admin-config-tool`
|
||||
(provider config).
|
||||
(provider config), `xce-node-manager`: a `provisioner` node role keeps the
|
||||
daemon running; which instance is active is decided by the
|
||||
Controller-granted lease, not by the Node Manager (`xce-node-manager`
|
||||
requirements.md, "Cross-Spec Interactions" 3).
|
||||
|
||||
## Constraints
|
||||
- Provider scripts never run in the controller or scheduler process.
|
||||
|
|
|
|||
|
|
@ -53,7 +53,10 @@ apiserver from pushed state) maintains the endpoint table per service.
|
|||
|
||||
## Existing Spec Touchpoints
|
||||
- **Extends**: `application-profile` (ports in the service section).
|
||||
- **Adjacent**: `resource-distribution-mapping`.
|
||||
- **Adjacent**: `resource-distribution-mapping`, `xce-node-manager`: a
|
||||
later service proxy or load balancer would run under the Node Manager's
|
||||
`reverse-proxy` role, its routing fed by the endpoint table
|
||||
(`xce-node-manager` requirements.md, "Cross-Spec Interactions" 6).
|
||||
|
||||
## Constraints
|
||||
- Endpoint updates are change-driven; no per-probe traffic.
|
||||
|
|
|
|||
|
|
@ -81,7 +81,13 @@ considered and rejected for v1: it would rerun `xcecd_worker` itself
|
|||
## Existing Spec Touchpoints
|
||||
- **Extends**: `application-profile` (service section of the profile).
|
||||
- **Adjacent**: `job-sandbox-execution` (execution backends must support the
|
||||
restart loop), `controller-job-cleanup`.
|
||||
restart loop), `controller-job-cleanup`, `xce-node-manager` (supervises
|
||||
XCE's own daemons; never job processes). Restart policy, backoff, give-up
|
||||
and probes (incl. the Node Manager's TCP port check) should come from one
|
||||
shared supervision/probe module so both layers behave the same
|
||||
(`xce-node-manager` requirements.md, "Cross-Spec Interactions" 1).
|
||||
Infrastructure services that XCE does not need to run itself are service
|
||||
jobs, not Node Manager roles (ibid. 2).
|
||||
|
||||
## Constraints
|
||||
- Health reports must be change-only (no per-probe WAL traffic).
|
||||
|
|
|
|||
|
|
@ -481,6 +481,15 @@ flowchart TB
|
|||
- Validation checkpoint: after Phase 2, run the soak-harness scenario in Testing Strategy's E2E section before proceeding to Phase 3/4, since it is the closest thing to a direct regression test for bug-1760 itself.
|
||||
|
||||
## Open Questions / Risks
|
||||
|
||||
- **Cross-spec interactions (2026-10-10), pending a requirements revision:**
|
||||
shared supervision/probe module with `service-supervision`; a lease-based
|
||||
`provisioner` role for `resource-provisioner`; stopping the worker on
|
||||
decommission and role declaration for provisioned hosts (`elastic-hosts`);
|
||||
host-loss grace period longer than worker restart + rejoin
|
||||
(`host-failure-job-recovery`); a future service proxy under the
|
||||
`reverse-proxy` role (`service-endpoint`). See requirements.md,
|
||||
"Cross-Spec Interactions".
|
||||
- The exact Node Role Manifest file format (YAML to match `src/configs/*.yaml`'s existing convention, vs. a simpler key-value form) is left to implementation — an additive, appropriately implementation-level decision, not a design decision, since Data Models already fixes its logical shape.
|
||||
- The precise heartbeat interval and "no progress" threshold (Requirements 4.2, 4.3) are implementation-time tuning decisions, following the same posture the superseded spec already took for its own liveness thresholds.
|
||||
- `tools/appimage/entry.sh` bundles `haproxy` today with no dispatch case reaching it at all — this feature is what first wires it up (via `xce-node-manager` invoking it directly for the reverse-proxy role), which is a pre-existing gap this design closes as a side effect rather than a pre-existing behavior it must preserve.
|
||||
|
|
|
|||
|
|
@ -99,3 +99,49 @@ Open design questions to resolve during requirements/design phases (do not assum
|
|||
1. Where existing deployment tooling documents starting the Controller, Scheduler Process, Worker Process, or Apiserver Process as an independent, manually-managed process, that tooling shall be updated to reflect that the Node Manager now starts these components according to each node's assigned role.
|
||||
2. Where an existing spec or test tooling depends on a component being started or managed independently of the Node Manager, that dependency shall be identified and reconciled with this feature's behavior rather than left silently incompatible.
|
||||
3. The observable behavior of the Controller, Scheduler Process, Worker Process, and Apiserver Process toward jobs, queues, configuration, and cluster membership shall remain unchanged by this feature while each is running.
|
||||
|
||||
## Cross-Spec Interactions (2026-10-10, pending requirements revision)
|
||||
|
||||
Recorded during the long-running services and auto resource provisioning
|
||||
discoveries (`.kiro/steering/roadmap.md`). These are **not yet acceptance
|
||||
criteria**: the approved requirements and design above are unchanged until a
|
||||
revision (`/kiro-spec-requirements xce-node-manager`) folds them in or
|
||||
rejects them.
|
||||
|
||||
1. **Two supervision layers, shared mechanics.** The Node Manager supervises
|
||||
XCE's own components; the worker supervises user jobs, including service
|
||||
replicas (`service-supervision`). The Node Manager never supervises job
|
||||
processes (consistent with Requirement 5). Restart policy with backoff,
|
||||
give-up rule and probes (the Node Manager's TCP port check, the worker's
|
||||
command/TCP/HTTP probes) should come from one shared supervision/probe
|
||||
module so their semantics match.
|
||||
2. **Infrastructure services boundary.** The Node Manager runs what XCE
|
||||
itself needs, or what must run before or without a Controller (XCE
|
||||
components, the reverse proxy). Everything else, including admin
|
||||
infrastructure services such as license servers or monitoring agents,
|
||||
runs as XCE service jobs (`service-replica-set`), scheduled with quotas,
|
||||
RBAC and accounting.
|
||||
3. **Roles for lease-based components.** `resource-provisioner` adds a
|
||||
daemon: the Node Manager needs a `provisioner` role that only keeps the
|
||||
process running; which instance is active is decided by a
|
||||
Controller-granted lease, not by the Node Manager (unlike the Scheduler's
|
||||
primary-bound start/stop). Future XCE daemons should follow the same
|
||||
"role + lease" pattern.
|
||||
4. **Provisioned hosts and decommission** (`elastic-hosts`). A cloud-burst
|
||||
host boots with the Node Manager and the `worker` role, declared by its
|
||||
host template and written into the Node Manager's configuration by the
|
||||
instance's boot script (one concrete answer to "how a node's role
|
||||
assignment is declared"). On decommission, the Node Manager must stop
|
||||
the worker on that node (or drop its role) instead of restarting it, and
|
||||
the Controller refuses a rejoin from a decommissioned host.
|
||||
5. **Worker restart vs. host loss** (`host-failure-job-recovery`). Jobs run
|
||||
in systemd units and survive a worker daemon restart; the restarted
|
||||
worker re-reports them on rejoin. The host-loss grace period must exceed
|
||||
the Node Manager's worker restart plus rejoin time, so a worker crash
|
||||
never requeues or fails jobs (service replicas included). Optional: the
|
||||
Node Manager reports "worker restarting" so the Controller does not start
|
||||
the grace period.
|
||||
6. **Future service proxy** (`service-endpoint`). A service proxy or load
|
||||
balancer, out of scope for `service-endpoint` v1, would naturally run
|
||||
under the Node Manager's `reverse-proxy` role, its routing fed by the
|
||||
endpoint table.
|
||||
|
|
|
|||
|
|
@ -86,6 +86,7 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
|
|||
## Existing Spec Updates (2026-10-10 auto resource provisioning discovery)
|
||||
- [ ] resource-distribution-mapping — choose its shared-pool scope (option 2) so a cluster-wide pool (licenses) can be declared in `resources.yaml`; prerequisite of elastic-shared-resources.
|
||||
- [ ] peer-authentication-hardening — a provisioner role with its allowed operations (provisioning facts, shared-total updates, expected hosts).
|
||||
- [ ] xce-node-manager — revise requirements (approved 2026-09-03) for the cross-spec interactions recorded 2026-10-10 in its requirements.md: shared supervision/probe module with service-supervision, a lease-based `provisioner` role, worker stop on decommission and role declaration for provisioned hosts, grace period vs worker restart, future service proxy under `reverse-proxy`.
|
||||
|
||||
## Existing Spec Updates (2026-10-10 long-running services discovery)
|
||||
- [ ] application-profile — profile kind `service` section: restart policy, probes, ports, default replicas, spread rule, rollout settings (consumed by service-supervision, service-replica-set, service-endpoint).
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue