diff --git a/.kiro/specs/elastic-hosts/brief.md b/.kiro/specs/elastic-hosts/brief.md index 774c46db..8daa7509 100644 --- a/.kiro/specs/elastic-hosts/brief.md +++ b/.kiro/specs/elastic-hosts/brief.md @@ -63,7 +63,12 @@ provisioned hosts; template-based host config; a decommission control. ## Existing Spec Touchpoints - **Extends**: none. - **Adjacent**: `worker-join-refactor`, `peer-authentication-hardening`, - `xce-node-manager`. + `xce-node-manager`: a provisioned host boots with the Node Manager and the + `worker` role, declared by its host template and written into the Node + Manager's configuration by the instance's boot script; on decommission + the Node Manager stops the worker (or drops its role) instead of + restarting it, and the Controller refuses a rejoin from a decommissioned + host (`xce-node-manager` requirements.md, "Cross-Spec Interactions" 4). ## Constraints - HA/persistence: full 3-phase workflow; the scheduler must see the same diff --git a/.kiro/specs/host-failure-job-recovery/brief.md b/.kiro/specs/host-failure-job-recovery/brief.md index 6025a34f..c790a722 100644 --- a/.kiro/specs/host-failure-job-recovery/brief.md +++ b/.kiro/specs/host-failure-job-recovery/brief.md @@ -78,7 +78,12 @@ that kills jobs whose attempt number is stale. ## Existing Spec Touchpoints - **Extends**: none. - **Adjacent**: `worker-rejoin-resync`, `controller-job-cleanup`, - `scheduler-preemption` (requeue semantics). + `scheduler-preemption` (requeue semantics), `xce-node-manager`: jobs run + in systemd units and survive a worker daemon restart, so the grace period + must exceed the Node Manager's worker restart plus rejoin time and a + worker crash must never requeue or fail jobs; optionally the Node Manager + reports "worker restarting" so the grace period does not start + (`xce-node-manager` requirements.md, "Cross-Spec Interactions" 5). ## Constraints - HA/persistence: full 3-phase workflow; additive Cap'n Proto; the new facts diff --git a/.kiro/specs/resource-provisioner/brief.md b/.kiro/specs/resource-provisioner/brief.md index 7117251d..3515a1ff 100644 --- a/.kiro/specs/resource-provisioner/brief.md +++ b/.kiro/specs/resource-provisioner/brief.md @@ -93,7 +93,10 @@ controller facts. ## Existing Spec Touchpoints - **Extends**: none (`src/autoscaler` is prior art only). - **Adjacent**: `component-failover-adoption`, `admin-config-tool` - (provider config). + (provider config), `xce-node-manager`: a `provisioner` node role keeps the + daemon running; which instance is active is decided by the + Controller-granted lease, not by the Node Manager (`xce-node-manager` + requirements.md, "Cross-Spec Interactions" 3). ## Constraints - Provider scripts never run in the controller or scheduler process. diff --git a/.kiro/specs/service-endpoint/brief.md b/.kiro/specs/service-endpoint/brief.md index 797958d2..c9c63b8d 100644 --- a/.kiro/specs/service-endpoint/brief.md +++ b/.kiro/specs/service-endpoint/brief.md @@ -53,7 +53,10 @@ apiserver from pushed state) maintains the endpoint table per service. ## Existing Spec Touchpoints - **Extends**: `application-profile` (ports in the service section). -- **Adjacent**: `resource-distribution-mapping`. +- **Adjacent**: `resource-distribution-mapping`, `xce-node-manager`: a + later service proxy or load balancer would run under the Node Manager's + `reverse-proxy` role, its routing fed by the endpoint table + (`xce-node-manager` requirements.md, "Cross-Spec Interactions" 6). ## Constraints - Endpoint updates are change-driven; no per-probe traffic. diff --git a/.kiro/specs/service-supervision/brief.md b/.kiro/specs/service-supervision/brief.md index 029a888a..e7d28d6a 100644 --- a/.kiro/specs/service-supervision/brief.md +++ b/.kiro/specs/service-supervision/brief.md @@ -81,7 +81,13 @@ considered and rejected for v1: it would rerun `xcecd_worker` itself ## Existing Spec Touchpoints - **Extends**: `application-profile` (service section of the profile). - **Adjacent**: `job-sandbox-execution` (execution backends must support the - restart loop), `controller-job-cleanup`. + restart loop), `controller-job-cleanup`, `xce-node-manager` (supervises + XCE's own daemons; never job processes). Restart policy, backoff, give-up + and probes (incl. the Node Manager's TCP port check) should come from one + shared supervision/probe module so both layers behave the same + (`xce-node-manager` requirements.md, "Cross-Spec Interactions" 1). + Infrastructure services that XCE does not need to run itself are service + jobs, not Node Manager roles (ibid. 2). ## Constraints - Health reports must be change-only (no per-probe WAL traffic). diff --git a/.kiro/specs/xce-node-manager/design.md b/.kiro/specs/xce-node-manager/design.md index 45f93452..7a535ec3 100644 --- a/.kiro/specs/xce-node-manager/design.md +++ b/.kiro/specs/xce-node-manager/design.md @@ -481,6 +481,15 @@ flowchart TB - Validation checkpoint: after Phase 2, run the soak-harness scenario in Testing Strategy's E2E section before proceeding to Phase 3/4, since it is the closest thing to a direct regression test for bug-1760 itself. ## Open Questions / Risks + +- **Cross-spec interactions (2026-10-10), pending a requirements revision:** + shared supervision/probe module with `service-supervision`; a lease-based + `provisioner` role for `resource-provisioner`; stopping the worker on + decommission and role declaration for provisioned hosts (`elastic-hosts`); + host-loss grace period longer than worker restart + rejoin + (`host-failure-job-recovery`); a future service proxy under the + `reverse-proxy` role (`service-endpoint`). See requirements.md, + "Cross-Spec Interactions". - The exact Node Role Manifest file format (YAML to match `src/configs/*.yaml`'s existing convention, vs. a simpler key-value form) is left to implementation — an additive, appropriately implementation-level decision, not a design decision, since Data Models already fixes its logical shape. - The precise heartbeat interval and "no progress" threshold (Requirements 4.2, 4.3) are implementation-time tuning decisions, following the same posture the superseded spec already took for its own liveness thresholds. - `tools/appimage/entry.sh` bundles `haproxy` today with no dispatch case reaching it at all — this feature is what first wires it up (via `xce-node-manager` invoking it directly for the reverse-proxy role), which is a pre-existing gap this design closes as a side effect rather than a pre-existing behavior it must preserve. diff --git a/.kiro/specs/xce-node-manager/requirements.md b/.kiro/specs/xce-node-manager/requirements.md index a6c7b2ec..d9aca308 100644 --- a/.kiro/specs/xce-node-manager/requirements.md +++ b/.kiro/specs/xce-node-manager/requirements.md @@ -99,3 +99,49 @@ Open design questions to resolve during requirements/design phases (do not assum 1. Where existing deployment tooling documents starting the Controller, Scheduler Process, Worker Process, or Apiserver Process as an independent, manually-managed process, that tooling shall be updated to reflect that the Node Manager now starts these components according to each node's assigned role. 2. Where an existing spec or test tooling depends on a component being started or managed independently of the Node Manager, that dependency shall be identified and reconciled with this feature's behavior rather than left silently incompatible. 3. The observable behavior of the Controller, Scheduler Process, Worker Process, and Apiserver Process toward jobs, queues, configuration, and cluster membership shall remain unchanged by this feature while each is running. + +## Cross-Spec Interactions (2026-10-10, pending requirements revision) + +Recorded during the long-running services and auto resource provisioning +discoveries (`.kiro/steering/roadmap.md`). These are **not yet acceptance +criteria**: the approved requirements and design above are unchanged until a +revision (`/kiro-spec-requirements xce-node-manager`) folds them in or +rejects them. + +1. **Two supervision layers, shared mechanics.** The Node Manager supervises + XCE's own components; the worker supervises user jobs, including service + replicas (`service-supervision`). The Node Manager never supervises job + processes (consistent with Requirement 5). Restart policy with backoff, + give-up rule and probes (the Node Manager's TCP port check, the worker's + command/TCP/HTTP probes) should come from one shared supervision/probe + module so their semantics match. +2. **Infrastructure services boundary.** The Node Manager runs what XCE + itself needs, or what must run before or without a Controller (XCE + components, the reverse proxy). Everything else, including admin + infrastructure services such as license servers or monitoring agents, + runs as XCE service jobs (`service-replica-set`), scheduled with quotas, + RBAC and accounting. +3. **Roles for lease-based components.** `resource-provisioner` adds a + daemon: the Node Manager needs a `provisioner` role that only keeps the + process running; which instance is active is decided by a + Controller-granted lease, not by the Node Manager (unlike the Scheduler's + primary-bound start/stop). Future XCE daemons should follow the same + "role + lease" pattern. +4. **Provisioned hosts and decommission** (`elastic-hosts`). A cloud-burst + host boots with the Node Manager and the `worker` role, declared by its + host template and written into the Node Manager's configuration by the + instance's boot script (one concrete answer to "how a node's role + assignment is declared"). On decommission, the Node Manager must stop + the worker on that node (or drop its role) instead of restarting it, and + the Controller refuses a rejoin from a decommissioned host. +5. **Worker restart vs. host loss** (`host-failure-job-recovery`). Jobs run + in systemd units and survive a worker daemon restart; the restarted + worker re-reports them on rejoin. The host-loss grace period must exceed + the Node Manager's worker restart plus rejoin time, so a worker crash + never requeues or fails jobs (service replicas included). Optional: the + Node Manager reports "worker restarting" so the Controller does not start + the grace period. +6. **Future service proxy** (`service-endpoint`). A service proxy or load + balancer, out of scope for `service-endpoint` v1, would naturally run + under the Node Manager's `reverse-proxy` role, its routing fed by the + endpoint table. diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index e3d280d8..45622b3c 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -86,6 +86,7 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han ## Existing Spec Updates (2026-10-10 auto resource provisioning discovery) - [ ] resource-distribution-mapping — choose its shared-pool scope (option 2) so a cluster-wide pool (licenses) can be declared in `resources.yaml`; prerequisite of elastic-shared-resources. - [ ] peer-authentication-hardening — a provisioner role with its allowed operations (provisioning facts, shared-total updates, expected hosts). +- [ ] xce-node-manager — revise requirements (approved 2026-09-03) for the cross-spec interactions recorded 2026-10-10 in its requirements.md: shared supervision/probe module with service-supervision, a lease-based `provisioner` role, worker stop on decommission and role declaration for provisioned hosts, grace period vs worker restart, future service proxy under `reverse-proxy`. ## Existing Spec Updates (2026-10-10 long-running services discovery) - [ ] application-profile — profile kind `service` section: restart policy, probes, ports, default replicas, spread rule, rollout settings (consumed by service-supervision, service-replica-set, service-endpoint).