docs(spec): peer-authentication-hardening: decide SSH-style ed25519 keys with temporary keys and revocation

Revocation and expiry close live connections authenticated with the key,
since authentication happens only at join. Lists the design questions:
where the trusted list is changed, keeping revoked keys revoked, bootstrap
keys, token registration for elastic hosts.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.kiro/specs/peer-authentication-hardening/brief.md

@@ -6,12 +6,12 @@ every user request goes through the apiserver, so the controller must reject
 user requests from anything else. Open decision: secret type and key
 distribution, below.)
 
-**Status (2026-10-12): waiting on a decision.** The user deferred the
-secret-type and key-distribution choice to a later discussion. Do not run
-`/kiro-spec-init` until it is made. The options, trade-offs and the
-recommendation are in "Open decision: secret type", "Open decision: key
-distribution" and "Recommendation" below; the rest of this brief (modes,
-role allowlist, `admin` role, apiserver-only user requests) is agreed.
+**Status (2026-10-12): secret type decided, D (SSH-style ed25519 key
+pairs)**, with temporary keys and revocation (user: "it just like auth of
+ssh, we also can create some tempo keys and revoke them when it's
+necessary"). See "Decision" below. Modes, role allowlist, `admin` role and
+apiserver-only user requests were already agreed. Ready for
+`/kiro-spec-init` when the user wants to continue.
 
 ## Problem
 The controller cannot tell who is connected to it. Any host that reaches the
@@ -83,7 +83,7 @@ accident too: pointed at a production cluster, they bypass the apiserver.
 - Rotation without downtime, and no secret ever in the WAL, a snapshot,
   `cluster.bin`, logs or the pushed config.
 
-## Open decision: secret type
+## Secret type (options considered)
 Four candidates (D added 2026-10-12 from the user's question "or just
 rsa/ed25519?"). All are verified at join only; afterwards the
 long-lived connection is the identity (connections are never re-used
@@ -162,7 +162,7 @@ each with its role (and, for a worker, its host name), like SSH's
   exporter value: `ssl` doesn't expose it, pyOpenSSL does (new xceadmin
   dependency).
 
-## Open decision: key distribution
+## Key distribution (options considered)
 | | 1. Operator copies files | 2. Bootstrap token, then enrolment | 3. xce-node-manager places them |
 |---|---|---|---|
 | How | config management (Ansible, Nix, images) installs the key file (root, 0600) on each host, as sites do for `munge.key` | admin runs `xceadmin token create --role worker --ttl 1h`; a new host presents the one-time token once (pinning the controller's cert hash, like `kubeadm join --discovery-token-ca-cert-hash`) and receives its long-lived credential (a per-host key or a signed cert), stored locally | the local node manager (`xce-node-manager`) reads key paths from its node role manifest and passes them to each component; it does not obtain them |
@@ -171,7 +171,46 @@ each with its role (and, for a worker, its host name), like SSH's
 | Air-gapped sites (`product.md`) | works | works (no external CA) | works |
 | HA | each controller gets the same files | the issuing secret (CA key or root key) must be on every controller to enrol through any of them; it is a file on each controller, never replicated through the WAL | n/a |
 
-## Recommendation (for decision)
+## Decision (2026-10-12)
+**D: ed25519 key pairs, authenticated like SSH.**
+- Each component generates its own key pair; only public keys are
+  collected (config management for fixed hosts, one-time token
+  registration for elastic hosts).
+- The controller's trusted list maps public key → role (+ host for a
+  worker, + a comment naming the owner), like `authorized_keys`.
+- **Temporary keys:** any entry may carry an expiry (absolute time, by the
+  controller's clock), like OpenSSH's `expiry-time=`. Uses: an admin's
+  key for a maintenance window, a CI or test runner, a contractor, an
+  elastic host's lease. `xceadmin key add --role admin --expires 8h
+  <pubkey>`.
+- **Revocation:** removing an entry or marking it revoked takes effect at
+  once: new joins with the key fail, and **live connections authenticated
+  with it are closed** (authentication happens only at join and
+  connections are long-lived, so a revocation that left them open would
+  do nothing). The same applies when a temporary key expires: a timer
+  queues the close (the delaymgr only triggers it; it never blocks). For
+  a worker this looks like the host going away; that is the intent.
+- Every add, expiry, revocation and rejected join is logged with the key's
+  fingerprint and comment, never the key itself.
+
+Design questions this raises:
+1. **Where the trusted list is changed.** Through the cluster config only
+   (a `keys.yaml`, replaced by `xceadmin config push`), or through
+   dedicated admin requests (`keyAdd` / `keyRevoke`) recorded as WAL
+   facts (public keys are not secret, so the WAL may hold them)? Temporary
+   keys and urgent revocation favour dedicated requests; a single source
+   favours config. If both, decide which wins (the same problem as
+   `rbac-persistence` and `project-delegated-membership`).
+2. **Revoked stays revoked.** Re-pushing an older config must not bring a
+   revoked key back: keep a revoked-fingerprint list (OpenSSH
+   `RevokedKeys`) that always wins over any entry.
+3. **Bootstrap keys** (controllers' own keys, the first admin key) live in
+   each controller's local startup config; how they are rotated.
+4. **Token registration** for elastic hosts: token lifetime, single use,
+   which roles a token may register (worker only?), and the key's expiry
+   tied to the host's lease.
+
+## Recommendation (superseded by the decision above)
 **Revised 2026-10-12: D (ed25519 key pairs) is the better fit.** It gives
 C's per-host identity without a CA, certificates or expiry outages; no
 secret ever moves between hosts; the trusted list rides on the existing

.kiro/steering/roadmap.md

@@ -158,7 +158,7 @@ Smaller items that likely don't warrant a full spec; pick up opportunistically o
 - [ ] **job_ctrl direct-mutate path — migrate to commit-then-apply** — **recorded 2026-07-13, investigated 2026-08-19 (finding: real risk, wider scope than originally framed).** `raft-apply-decoupling` deliberately scoped `job_ctrl`'s `mark_job_update`/`mark_job_fail` (admin SIGCONT/SIGSTOP) OUT-OF-BOUNDARY (task 4.1/6.1 Implementation Notes). The 2026-08-19 investigation found the original framing wrong in two ways: (1) **NOT the only remaining call site** — `on_job_stop`/`on_job_resume`/`on_job_ckpt` (`job.c:2067-2222`) share the identical unconverted mutate-then-notify-then-WAL-append pattern, fed by ordinary worker/scheduler traffic, not "rare, admin-only" as originally described; (2) **blast radius (a) is confirmed real, not theoretical** — the pre-commit mutation is synchronously pushed live to BOTH the scheduler (`sched_job_update_fun`) AND every connected apiserver/client (`apiserver_job_update_fun`) before the commit barrier resolves; on barrier timeout the admin gets a failure reply but `job->job.state` is never rolled back and no retraction is ever sent — matching the bug-795/828/851/1060 shape. Live-apply vs replay-apply (b) is narrower than feared for `job_ctrl` specifically (the `allocs != NULL` gate mutually excludes it from the converted deferred-apply fact types), but the investigation surfaced a SEPARATE bug underpinning that safety argument: `job_ctrl` (`job.c:999`) holds only a read lock while mutating job state — every other mutator in this file takes a write lock — so two concurrent admin commands on the same job aren't mutually excluded, undermining the "same lock, no staleness window" claim the replay-safety argument depends on (already flagged by a `raft-apply-decoupling` task 2.1 reviewer, never fixed). **Recommendation**: promote to a small follow-up spec covering all four call sites together (bounded, well-precedented fix — same shape as the fact-receipt conversion `raft-apply-decoupling` already built for job submit); the lock-type bug is independently fixable and low-risk regardless of when/whether the larger migration happens. Not yet promoted — user decision 2026-08-19 to correct scope/framing here first, promote later.
 - [x] **WAL-trim cursor-bound guard (bug-1147; surfaced 2026-06-25 by raft-drive-async design review)** — **IMPLEMENTED** as part of `raft-drive-async` task 3.2 (`raft-drive-async` is IMPLEMENTED, validate-impl GO 2026-07-18; see above). `trim_wal_segments` (`persist.c` ~2597-2644) now computes `floor = repl_trim_lsn(&p->repl, active_seg_lsn, p->config.snap_interval_s)` — folding in active-standby `acked_lsn` and disconnected-slot `last_disconnected_lsn` — and only unlinks segments below that floor; the code comment at `persist.c:2634` explicitly labels this the "(bug-1147 guard)". Far-behind/empty join (cursor=0) is additionally covered now that `entity-bearing snapshots` (strategy B) has landed as `ha-entity-snapshots` (see above), with a live decision path (`ha.c:4413`, "< oldest_wal_lsn → requesting snapshot transfer"). Related: bug-795, bug-828, bug-1060.
 - [ ] OP_heartbeat peers cross-validation — include a hash of the local `--peers` string in heartbeat requests; log ERROR on mismatch between candidates. Detects misconfigured nodes at election time instead of silently risking dual-primary split-brain. _(FUTURE_WORK.md §3.)_
-- [ ] **peer-authentication-hardening** — **NEW 2026-07-06 from admin-config-tool discovery** (`/kiro-discovery`). **Brief 2026-10-12: `.kiro/specs/peer-authentication-hardening/brief.md`** (user decisions: production/test mode on the controller, production by default; in production user requests are accepted only from an authenticated `apiserver` connection, harness/direct libxce are test tools; `xceadmin` joins as a new `admin` role; open decision: secret type (cluster key / per-role keys / PKI / SSH-style ed25519 key pairs) and key distribution (files / token enrolment), current recommendation ed25519 key pairs with the trusted public-key list in the cluster config; **decision deferred by the user for a later discussion, do not start the spec before it**). Correction: TLS is `SSL_VERIFY_NONE` everywhere, so not even a CA-signed certificate is required. Original text: XCE currently operates under a **trusted-LAN model** — any CA-signed SSL peer can connect to the controller and perform ANY operation. Three gaps identified: **(a) Admin→Controller**: `validate_host()` is a stub returning success for all peers (roadmap § below). `OP_upgradeConfig` has no real authorization gate today. **Same gap confirmed 2026-08-02 for `OPCode_schedAdmin`** (`scheduler-core` design, `cluster.c`'s `case OPCode_upgradeConfig` was checked as the closest precedent and has only a primary/standby routing gate, no RBAC/auth check) — `scheduler-core` ships `OPCode_schedAdmin` at parity with that existing gap (project owner decision: not a `scheduler-core` blocker), so this item now covers both opcodes; when `validate_host()` is implemented, wire it to both `OP_upgradeConfig` and `OPCode_schedAdmin` in the same pass. **(b) Worker/Scheduler→Controller**: workers and schedulers connect via `controller_connector.c` using the same TLS context as controller peers, with no role differentiation — a compromised worker could send `OP_upgradeConfig` or read full config via `OP_replConfig`. **(c) Cross-component trust**: no distinction between controller peers, admin tools, workers, and schedulers — they share the same X.509 identity pool. **(d) xceadmin login (future)**: xceadmin currently uses zero-config TLS (same as xce_test). Production deployments likely need an admin login requirement — token-based session (`xceadmin login → token → config push`) or password-per-command. X.509 CN alone proves identity but not intent; a login step provides audit trail, session expiry, and operator accountability. **Needs design**: role-based X.509 CN validation (controller/admin/worker/scheduler roles), per-role OP code allowlists (now including `OPCode_schedAdmin`), admin login/session model, audit logging for admin operations. Separate from the existing `validate_host()` hardening (below) which is the minimal gate for `OP_upgradeConfig`/`OPCode_schedAdmin` specifically. Discuss + design before implementing; likely promotes to a spec via `/kiro-discovery`.
+- [ ] **peer-authentication-hardening** — **NEW 2026-07-06 from admin-config-tool discovery** (`/kiro-discovery`). **Brief 2026-10-12: `.kiro/specs/peer-authentication-hardening/brief.md`** (user decisions: production/test mode on the controller, production by default; in production user requests are accepted only from an authenticated `apiserver` connection, harness/direct libxce are test tools; `xceadmin` joins as a new `admin` role; secret type decided 2026-10-12: SSH-style ed25519 key pairs, a trusted public-key list (role, host, optional expiry), temporary keys, revocation that also closes live connections; public keys collected by config management, token registration for elastic hosts; ready for `/kiro-spec-init`). Correction: TLS is `SSL_VERIFY_NONE` everywhere, so not even a CA-signed certificate is required. Original text: XCE currently operates under a **trusted-LAN model** — any CA-signed SSL peer can connect to the controller and perform ANY operation. Three gaps identified: **(a) Admin→Controller**: `validate_host()` is a stub returning success for all peers (roadmap § below). `OP_upgradeConfig` has no real authorization gate today. **Same gap confirmed 2026-08-02 for `OPCode_schedAdmin`** (`scheduler-core` design, `cluster.c`'s `case OPCode_upgradeConfig` was checked as the closest precedent and has only a primary/standby routing gate, no RBAC/auth check) — `scheduler-core` ships `OPCode_schedAdmin` at parity with that existing gap (project owner decision: not a `scheduler-core` blocker), so this item now covers both opcodes; when `validate_host()` is implemented, wire it to both `OP_upgradeConfig` and `OPCode_schedAdmin` in the same pass. **(b) Worker/Scheduler→Controller**: workers and schedulers connect via `controller_connector.c` using the same TLS context as controller peers, with no role differentiation — a compromised worker could send `OP_upgradeConfig` or read full config via `OP_replConfig`. **(c) Cross-component trust**: no distinction between controller peers, admin tools, workers, and schedulers — they share the same X.509 identity pool. **(d) xceadmin login (future)**: xceadmin currently uses zero-config TLS (same as xce_test). Production deployments likely need an admin login requirement — token-based session (`xceadmin login → token → config push`) or password-per-command. X.509 CN alone proves identity but not intent; a login step provides audit trail, session expiry, and operator accountability. **Needs design**: role-based X.509 CN validation (controller/admin/worker/scheduler roles), per-role OP code allowlists (now including `OPCode_schedAdmin`), admin login/session model, audit logging for admin operations. Separate from the existing `validate_host()` hardening (below) which is the minimal gate for `OP_upgradeConfig`/`OPCode_schedAdmin` specifically. Discuss + design before implementing; likely promotes to a spec via `/kiro-discovery`.
 - [ ] validate_host() real authorization — currently a stub returning success for all peers (`join.c:13-16`, `return 1;` unconditionally). **`CONF_ENABLE_UPGRADE_CONFIG` is already `1` in the standard build** (`config_repl.h:27-28`, no build-system override found) — the precondition this item was originally framed to guard against has already been crossed; `OP_upgradeConfig` (and `OPCode_schedAdmin`, added 2026-08-02, `scheduler-core`) are live and unauthenticated TODAY, not a future risk to prevent. Minimum fix: verify peer X.509 CN against a configured allowlist (`--admin-cn` flag or `admin_cn_list` config), reject non-allowlisted `OP_upgradeConfig` and `OPCode_schedAdmin` with clear error, WARN-log rejected attempts with CN+IP. Future: IP allowlist, role-based access, admin-specific CA. _(FUTURE_WORK.md §6; security-critical.)_ **Part of the broader `peer-authentication-hardening` item above.**
 - [ ] OP_replConfig access control hardening — currently serves the full config blob to any CA-signed SSL peer (trusted-LAN model). Consider: verify requester is a known `--peers` controller, rate-limit per peer, `minimum_role` check (controllers only). _(FUTURE_WORK.md §7.)_ **Part of the broader `peer-authentication-hardening` item above.**
 - [ ] lost-and-found drain command — `xcecli lost-and-found drain`: reassign or cancel sidelined jobs, reset `lost_and_found_job_count`. Automates operator cleanup after config upgrade. _(FUTURE_WORK.md §4.)_