From a2bf160cc3447c9df0fca236cdb7c068b0020586 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Tue, 13 Oct 2026 04:25:17 +0000 Subject: [PATCH] docs(spec): peer-authentication-hardening brief; production mode accepts user requests only from the apiserver Promotes the roadmap item to a brief: production/test mode on the controller, authenticated roles at join, per-role request allowlist with user requests only from the apiserver, an admin role for xceadmin. Lays out the open decisions on secret type (cluster key / per-role keys / PKI) and key distribution (files / token enrolment), with a phased recommendation. project-config Req 3.13 applies to test mode only. Co-Authored-By: Claude Opus 5.5 (1M context) --- .../peer-authentication-hardening/brief.md | 172 ++++++++++++++++++ .kiro/specs/project-config/brief.md | 4 +- .kiro/specs/project-config/requirements.md | 3 +- .kiro/steering/roadmap.md | 4 +- 4 files changed, 179 insertions(+), 4 deletions(-) create mode 100644 .kiro/specs/peer-authentication-hardening/brief.md diff --git a/.kiro/specs/peer-authentication-hardening/brief.md b/.kiro/specs/peer-authentication-hardening/brief.md new file mode 100644 index 00000000..c2410ab8 --- /dev/null +++ b/.kiro/specs/peer-authentication-hardening/brief.md @@ -0,0 +1,172 @@ +# Brief: peer-authentication-hardening + +(Roadmap backlog item since 2026-07-06, admin-config-tool discovery. Promoted +to a brief 2026-10-12 from the project-config RBAC discussion: in production +every user request goes through the apiserver, so the controller must reject +user requests from anything else. Open decision: secret type and key +distribution, below.) + +## Problem +The controller cannot tell who is connected to it. Any host that reaches the +controller port can join under any role and send any request: +- join as `apiserver` and submit, kill or modify jobs as any user (the + submit carries the user name and the controller believes it), bypassing + all RBAC, project authorization (`project-config` Req 3.12) and audit; +- push a cluster config (`upgradeConfig`) or run `schedAdmin`; +- read the full config (`replConfig`), or act as a worker for any host + (fake usage, job state, resync answers). + +Test tools (harness, direct libxce clients) make this easy to hit by +accident too: pointed at a production cluster, they bypass the apiserver. + +## Current State +- Role is self-declared at join (`JoinData.role`, `controller.capnp:151`) + and stored per connection (`priv->role`, `controller/join.c:275`); a + connection that never joins counts as `client` (`cluster.c:1413`). + Nothing checks a role against the request it sends, except a few + role-specific paths (`opupd.c:26`). +- `validate_host()` returns 1 for everyone (`join.c:13-16`), and is only + called for worker joins. +- The peer key is the literal `"keepSecret"` in the controller + (`cluster.c:41`), worker (`worker/src/controller_connector.c:175`) and + harness (`harness/test.c:272`). +- TLS is server-side only with `SSL_VERIFY_NONE` everywhere: no client + certificates, no server verification by clients, by decision + (`security.md` "TLS", bug-1453). (The roadmap said "any CA-signed SSL + peer"; in fact no certificate is verified at all, so a MITM can sit + between any two components.) Deploy scripts generate a self-signed pair + per deploy (`tools/deploy-ha.sh:246`). +- Non-joined direct client: `xceadmin config push` sends `upgradeConfig` + without joining (`xceadmin/push.py:81`), through pycapnp, not libxce. +- The apiserver joins through libxce via cgo (`backend/clib.go:57`, + rejoin `backend/reconnect.go:107`); schedulers and workers through + libxce (`xcecli_join`, `lib/client.c:300`). + +## Desired Outcome +- **Production and test mode** (user decision 2026-10-12). The controller + runs in production mode unless its local startup config says `test` + (never settable through the pushed cluster config). Test deployments + (harness, simulator, `deploy-ha.sh`, `/xce-deploy-ha`, `/xce-test-*`) + set test mode explicitly. +- **Authenticated roles.** A peer proves its role at join with a secret; + a failed proof closes the connection with `-EPERM`, logged with peer + address and claimed role (never the secret). +- **Per-role request allowlist** in production mode: + + | Role | Allowed requests | + |---|---| + | `apiserver` | all user requests: submit, kill/signal/modify, queries, queue control, ... | + | `worker` | worker requests: join, heartbeat, status and usage reports, job events, resync answers | + | `scheduler` | scheduler requests: decisions, `schedAdmin` replies, config fetch | + | `controller` | replication, votes, `replConfig` | + | `admin` (new) | `upgradeConfig`, `schedAdmin`, read-only status | + | `client` (unauthenticated) | nothing in production; everything in test mode | + + A request outside its role's list is rejected with `-EPERM` and logged. + Later roles (reporter, claim provider, provisioner; roadmap 101/106) + are added to the same table. +- **User requests only from the apiserver.** The controller trusts the + user name and authorization basis on a request only because it came + from an authenticated `apiserver` connection (`project-config` Req + 3.12, 3.13). +- **`xceadmin` joins as `admin`** with its secret, so config push keeps + working without the apiserver (bootstrap, apiserver down). +- **Clients authenticate the controller too**, so a worker or apiserver + cannot be fooled by a fake controller (mutual proof). +- Rotation without downtime, and no secret ever in the WAL, a snapshot, + `cluster.bin`, logs or the pushed config. + +## Open decision: secret type +Three candidates. All are verified at join only; afterwards the +long-lived connection is the identity (connections are never re-used +across roles). + +| | A. One cluster key (shared, symmetric) | B. One key per role (shared, symmetric) | C. PKI: cert per host from a cluster CA (mTLS) | +|---|---|---|---| +| Proof at join | HMAC challenge-response, both directions | same, key chosen by role | TLS client cert; role from the cert (e.g. OU), host from CN/SAN | +| A stolen worker key lets you be | anything, including `apiserver` and `admin` | any worker host, nothing else | that one host, as a worker | +| Per-host identity (worker A can't report for host B) | no | no | **yes** | +| Revoke one host | rotate the key everywhere | rotate the worker key on all workers | revoke that cert (CRL / short-lived certs) | +| Fixes MITM | yes, if the proof is bound to the TLS session (exporter, RFC 5705) | same | yes (both sides verify certs) | +| Cost | lowest: one file | low: 4-5 files, each on the hosts of its role | CA management, issuing and renewing a cert per host, expiry outages; reverses `security.md`'s no-mTLS decision (this spec would be that design doc) | +| Implementations to write | C (libxce), Python (`xceadmin`) | same | OpenSSL config in C, `ssl` in Python; mostly configuration, little custom crypto | +| HPC precedent | Slurm MUNGE (`munge.key` on every node) | Slurm `auth/slurm` / JWT keys per use | Kubernetes (cluster CA, kubelet certs) | + +Notes: +- A and B need a custom handshake. The proof must be bound to the TLS + session (`SSL_export_keying_material`); otherwise a MITM, which + `SSL_VERIFY_NONE` allows, just relays the challenge and the answer. + With binding, A/B also gives clients server authentication without + certificate checks. +- C gives per-host identity, which A and B never can. That matters most + for workers (thousands of hosts, run user code) and for elastic hosts. +- B can be extended later to per-host worker keys derived from the worker + key and the hostname (HKDF(worker_key, host)): the controller derives the + expected key; a stolen per-host key lets you be only that host. Each + host then still needs its own file, so it shares C's distribution + problem without C's revocation. + +## Open decision: key distribution +| | 1. Operator copies files | 2. Bootstrap token, then enrolment | 3. xce-node-manager places them | +|---|---|---|---| +| How | config management (Ansible, Nix, images) installs the key file (root, 0600) on each host, as sites do for `munge.key` | admin runs `xceadmin token create --role worker --ttl 1h`; a new host presents the one-time token once (pinning the controller's cert hash, like `kubeadm join --discovery-token-ca-cert-hash`) and receives its long-lived credential (a per-host key or a signed cert), stored locally | the local node manager (`xce-node-manager`) reads key paths from its node role manifest and passes them to each component; it does not obtain them | +| Fits | A, B | B (per-host keys), C | any (it is where the files are referenced, not how they arrive) | +| Elastic hosts (`elastic-hosts`, cloud burst) | the key must be baked into the image or user data: long-lived secret in cloud metadata | short-lived token in user data; nothing long-lived leaves the cluster | n/a | +| Air-gapped sites (`product.md`) | works | works (no external CA) | works | +| HA | each controller gets the same files | the issuing secret (CA key or root key) must be on every controller to enrol through any of them; it is a file on each controller, never replicated through the WAL | n/a | + +## Recommendation (for decision) +Phase it, with the authenticator pluggable like the apiserver auth backends: +1. **Now: B (per-role keys) + distribution 1 (files), referenced by the + node manager (3).** Smallest change that delivers everything in + "Desired Outcome": apiserver-only user requests, the `admin` role, + mutual authentication, MITM fixed through session binding, a stolen + worker key unable to become `apiserver`. Matches HPC practice. Key + files hold a list of keys with ids, so rotation is: add the new key on + controllers, roll peers, remove the old key. +2. **Later, when elastic hosts or per-host worker identity are needed:** + C with distribution 2 (token enrolment, controller-issued certs with + short lifetimes, so renewal replaces revocation lists). Per-role keys + can stay as the enrolment trust for the apiserver and admin. + +Going straight to C is the alternative if per-host worker identity is a +v1 requirement; it costs a CA, enrolment and renewal up front. + +## Scope +- **In**: production/test mode; join authentication (mutual, session + bound); per-role request allowlist (including `upgradeConfig`, + `schedAdmin`, `replConfig`); `admin` role for `xceadmin`; key files, + rotation, startup validation; libxce (C), apiserver (cgo), `xceadmin` + (Python), harness/test-mode support; deploy scripts and skills; logs of + rejected joins and requests. +- **Out**: `xceadmin` login and audit sessions (roadmap item (d), later); + per-host identity and token enrolment (phase 2, unless chosen now); + apiserver HTTPS posture (`security.md` HAProxy follow-up). + +## Upstream / Downstream +- **Upstream**: libxce join (`lib/client.c`, `lib/request.c:1414`), + controller join and dispatch (`join.c`, `cluster.c`), TLS context + (`lib/xcectx.c`), `xceadmin push.py`. +- **Downstream**: `project-config` (Req 3.12/3.13 rely on authenticated + apiserver connections), `elastic-hosts`, `resource-provisioner` and + reporter/claim-provider roles (roadmap 101/106), `xce-node-manager` + (key file paths in the node manifest). + +## Existing Spec Touchpoints +- **Adjacent**: `xce-node-manager`, `elastic-hosts`, `admin-config-tool`, + `libxce-failover` / rejoin paths (a rejoin re-authenticates), + `ha-peer-connection-reuse` (controller peer connections authenticate as + `controller`). + +## Constraints +- HA: a full spec workflow (security + every component's join). Join + authentication runs on the I/O path: HMAC and session-key export are + cheap and non-blocking; nothing on the delaymgr; no file reads after + startup except an explicit key reload. +- Connections stay long-lived; a failed authentication closes only the + unauthenticated connection. A controller restarted with rotated keys + must still accept peers holding any listed key, so a rolling rotation + never triggers failover. +- Secrets never in WAL, snapshot, `cluster.bin`, logs or replies. +- Test mode must keep the existing harness, simulator and skills working + unchanged apart from setting the mode. diff --git a/.kiro/specs/project-config/brief.md b/.kiro/specs/project-config/brief.md index b8a3eb47..549aa52f 100644 --- a/.kiro/specs/project-config/brief.md +++ b/.kiro/specs/project-config/brief.md @@ -106,6 +106,7 @@ projects: ## Upstream / Downstream - **Upstream**: config pipeline (`admin-config-tool`, `config-upgrade-snapshot`), + `peer-authentication-hardening` (authenticated apiserver connections), limit plugin (`plugins/limit.c`, `lib/xcelimit.c`), RBAC (`rbac_check.go`). - **Downstream**: `job-config-merge` (project level), `project-guarantee`, `limits-to-owners`, `lsf-config-migration`, `rbac-persistence` (if access uses RBAC scopes). @@ -121,7 +122,8 @@ projects: apiserver resolves the project and checks `project:submit` (membership and role grants); the controller records the project and authorization basis it is given and never re-checks RBAC. A direct submit is recorded - as given with basis `direct` (Req 3.13). + as given with basis `direct` in test mode (Req 3.13); in production mode + `peer-authentication-hardening` rejects it. - The job's authorization basis (Req 3.11) is a new job field: WAL fact and scheduler snapshot must carry it (field completeness, `persistence.md`). diff --git a/.kiro/specs/project-config/requirements.md b/.kiro/specs/project-config/requirements.md index ef42d720..03dd54d7 100644 --- a/.kiro/specs/project-config/requirements.md +++ b/.kiro/specs/project-config/requirements.md @@ -78,6 +78,7 @@ This feature makes projects first-class cluster configuration entities that cont - aggregate caps on queues (dropped by decision 2026-10-12, not deferred). - **Adjacent expectations**: - Project quotas are written on the project (decision 2026-10-12, reversing "`limits.yaml` stays the place where project quotas are written"). The scheduler's limit plugin enforces them as generated limits; admins never write those limits. + - `peer-authentication-hardening` makes the controller accept user requests in production mode only from an authenticated apiserver connection; Requirement 3.12 relies on it, and until it lands the controller's trust in the apiserver's authorization is a trusted-network assumption. - Quotas count what the scheduler allocated or reserved, not measured use; a job without a memory reservation can exceed its project's memory quota at run time (accepted; profile default reservations close the gap). - User groups come from the existing user-group configuration. - Grants made through the role API are subject to the existing `rbac-persistence` gap (not durable across apiserver restart) until that spec lands. @@ -126,7 +127,7 @@ This feature makes projects first-class cluster configuration entities that cont 10. The Cluster shall check `project:submit` in addition to, not instead of, the existing queue permission. 11. The Cluster shall record on the job whether its project was authorized by membership, by a role grant or by the cluster administrator role. 12. The apiserver shall be the only component that resolves the project and checks `project:submit`; the controller shall record the project and authorization basis it receives with the submission without checking them again. -13. When a job is submitted directly to the controller, bypassing the apiserver (harness and client-library test paths, not supported for production), the Cluster shall record the project name as given (an empty name as `default`) with the authorization basis `direct`, without project resolution or authorization; Requirement 5 still applies to the pending job, except the membership hold of 5.2. +13. While the controller runs in test mode, when a job is submitted directly to the controller, bypassing the apiserver (harness and client-library test paths), the Cluster shall record the project name as given (an empty name as `default`) with the authorization basis `direct`, without project resolution or authorization; Requirement 5 still applies to the pending job, except the membership hold of 5.2. (In production mode such a submit is rejected by `peer-authentication-hardening`.) ### Requirement 4: Open mode when no projects are defined **Objective:** As an operator of an existing cluster, I want clusters without project definitions to keep working as today, so that upgrading does not break existing submissions. diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index d6da1a54..d27d3e80 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -73,7 +73,7 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han - [ ] controller-job-cleanup — NEW 2026-09-29 (`/kiro-discovery`, Path C). Primary-only periodic sweep (configurable `job_cleanup_period`, default 300 s) removes DONE/EXIT jobs finished longer than configurable `job_retention` (default 3600 s, 0 = off) with no dependents (stub; XCE has no job dependencies yet), via a NEW replicated WAL job-removal fact applied on every node (first-ever `jobmgr` delete; refcount-safe `xkvmgr` delete), propagated to scheduler and apiserver (`OnJobDeleted`). Retires the node-local snapshot-only `job_should_retain` finished-job filter (hard-coded 3600 s; causes snapshot/memory job-set drift). HA/persistence → full 3-phase workflow. Brief: `.kiro/specs/controller-job-cleanup/brief.md`. Dependencies: none (extends ha-entity-snapshots). - [ ] job-status-detail-display — NEW 2026-09-28 (`/kiro-discovery`, Path E). End users (REST/`xcecli`) see NO pend reason for ANY job today: the apiserver cgo conversion (`backend/convert.go`) never maps `xce_job_t.pend_reasons`, so `pending_reason` is always empty. Maps it to a structured JSON (including `alternatives` from day one) using libxce's code→text mapping, and renders it in `xcecli` job detail. Open for requirements: also map `run_usage` → `JobRusage` (same gap; reserved vs. used for RUN jobs). Brief: `.kiro/specs/job-status-detail-display/brief.md`. Dependencies: none for flat reasons; scheduler-alter-pend-reasons for per-alternative rendering. - [ ] job-config-merge — NEW 2026-10-09 (`/kiro-discovery`, Path E "configuration layering"; ref LSF 10.1 Admin Guide pp. 314-316, 435-437, 561-563, 643, 660). One merge function in `src/lib` turning the job's own values + application profile + queue policy + project into effective values (resreq incl. `span`, task range, limits, pre/post-exec, env). One owner per attribute (user decision; `.kiro/steering/job-config-layering.md`): the job may add to a profile's values but never change or remove them (additive rule, 2026-10-12, replacing *default*/*fixed* markers); queue ceilings and project quotas only check (reject if never satisfiable, pend if full now). **Live** merge (user decision): WAL keeps inputs + names; controller/apiserver validate at submit; scheduler computes effective values and re-evaluates pending jobs on config change. Queue is policy only: queue `res_req` and `pre_exec`/`post_exec` deprecated (pre/post-exec moves to the profile), dropped queue limits (cpulimit/memlimit/runlimit/task limit) wired. Brief: `.kiro/specs/job-config-merge/brief.md`. Dependencies: none. -- [ ] project-config — NEW 2026-10-09 (configuration layering). Projects control resource ownership, v1 = quota: project definitions in cluster config (path identity, members, cluster default), default-project resolution, submit access check, scheduler `pgmgr` loaded from config, quotas written on the project (`quota`, `member_quota` over the subtree; user decision 2026-10-12, replacing `PROJECTS`/`PER_PROJECT` limits in `limits.yaml`) and compiled into the limit plugin, per-project usage display. Brief: `.kiro/specs/project-config/brief.md`. Dependencies: none. +- [ ] project-config — NEW 2026-10-09 (configuration layering). Projects control resource ownership, v1 = quota: project definitions in cluster config (path identity, members, cluster default), default-project resolution, submit access check, scheduler `pgmgr` loaded from config, quotas written on the project (`quota`, `member_quota` over the subtree; user decision 2026-10-12, replacing `PROJECTS`/`PER_PROJECT` limits in `limits.yaml`) and compiled into the limit plugin, per-project usage display; `project:submit` RBAC implied by membership, project admins, authorization in the apiserver only. Brief: `.kiro/specs/project-config/brief.md`. Dependencies: none to ship; production rejection of direct submits comes from peer-authentication-hardening. - [ ] host-failure-job-recovery — NEW 2026-10-10 (`/kiro-discovery`, "Long-running services" phase 1; also a batch gap: RUN jobs on a dead host stay RUN forever, `controller/sched.c:631` FIXME). After a grace period, jobs on a lost host go to an unknown state, then are requeued (rerunnable, job option or profile default) or end EXIT "host lost"; multi-host jobs resolved as a whole; attempt numbers fence a stale copy when the host returns; manual requeue (REST/xcecli). Brief: `.kiro/specs/host-failure-job-recovery/brief.md`. Dependencies: none. - [ ] service-supervision — NEW 2026-10-10 (Long-running services phase 1). Application profile kind `service`: restart policy (always/on-failure/never, backoff, give-up), liveness and readiness probes (command/TCP/HTTP) run by each replica's own worker (no special replica), in-place restart, change-only health reports, display. Brief: `.kiro/specs/service-supervision/brief.md`. Dependencies: application-profile, host-failure-job-recovery. @@ -158,7 +158,7 @@ Smaller items that likely don't warrant a full spec; pick up opportunistically o - [ ] **job_ctrl direct-mutate path — migrate to commit-then-apply** — **recorded 2026-07-13, investigated 2026-08-19 (finding: real risk, wider scope than originally framed).** `raft-apply-decoupling` deliberately scoped `job_ctrl`'s `mark_job_update`/`mark_job_fail` (admin SIGCONT/SIGSTOP) OUT-OF-BOUNDARY (task 4.1/6.1 Implementation Notes). The 2026-08-19 investigation found the original framing wrong in two ways: (1) **NOT the only remaining call site** — `on_job_stop`/`on_job_resume`/`on_job_ckpt` (`job.c:2067-2222`) share the identical unconverted mutate-then-notify-then-WAL-append pattern, fed by ordinary worker/scheduler traffic, not "rare, admin-only" as originally described; (2) **blast radius (a) is confirmed real, not theoretical** — the pre-commit mutation is synchronously pushed live to BOTH the scheduler (`sched_job_update_fun`) AND every connected apiserver/client (`apiserver_job_update_fun`) before the commit barrier resolves; on barrier timeout the admin gets a failure reply but `job->job.state` is never rolled back and no retraction is ever sent — matching the bug-795/828/851/1060 shape. Live-apply vs replay-apply (b) is narrower than feared for `job_ctrl` specifically (the `allocs != NULL` gate mutually excludes it from the converted deferred-apply fact types), but the investigation surfaced a SEPARATE bug underpinning that safety argument: `job_ctrl` (`job.c:999`) holds only a read lock while mutating job state — every other mutator in this file takes a write lock — so two concurrent admin commands on the same job aren't mutually excluded, undermining the "same lock, no staleness window" claim the replay-safety argument depends on (already flagged by a `raft-apply-decoupling` task 2.1 reviewer, never fixed). **Recommendation**: promote to a small follow-up spec covering all four call sites together (bounded, well-precedented fix — same shape as the fact-receipt conversion `raft-apply-decoupling` already built for job submit); the lock-type bug is independently fixable and low-risk regardless of when/whether the larger migration happens. Not yet promoted — user decision 2026-08-19 to correct scope/framing here first, promote later. - [x] **WAL-trim cursor-bound guard (bug-1147; surfaced 2026-06-25 by raft-drive-async design review)** — **IMPLEMENTED** as part of `raft-drive-async` task 3.2 (`raft-drive-async` is IMPLEMENTED, validate-impl GO 2026-07-18; see above). `trim_wal_segments` (`persist.c` ~2597-2644) now computes `floor = repl_trim_lsn(&p->repl, active_seg_lsn, p->config.snap_interval_s)` — folding in active-standby `acked_lsn` and disconnected-slot `last_disconnected_lsn` — and only unlinks segments below that floor; the code comment at `persist.c:2634` explicitly labels this the "(bug-1147 guard)". Far-behind/empty join (cursor=0) is additionally covered now that `entity-bearing snapshots` (strategy B) has landed as `ha-entity-snapshots` (see above), with a live decision path (`ha.c:4413`, "< oldest_wal_lsn → requesting snapshot transfer"). Related: bug-795, bug-828, bug-1060. - [ ] OP_heartbeat peers cross-validation — include a hash of the local `--peers` string in heartbeat requests; log ERROR on mismatch between candidates. Detects misconfigured nodes at election time instead of silently risking dual-primary split-brain. _(FUTURE_WORK.md §3.)_ -- [ ] **peer-authentication-hardening** — **NEW 2026-07-06 from admin-config-tool discovery** (`/kiro-discovery`). XCE currently operates under a **trusted-LAN model** — any CA-signed SSL peer can connect to the controller and perform ANY operation. Three gaps identified: **(a) Admin→Controller**: `validate_host()` is a stub returning success for all peers (roadmap § below). `OP_upgradeConfig` has no real authorization gate today. **Same gap confirmed 2026-08-02 for `OPCode_schedAdmin`** (`scheduler-core` design, `cluster.c`'s `case OPCode_upgradeConfig` was checked as the closest precedent and has only a primary/standby routing gate, no RBAC/auth check) — `scheduler-core` ships `OPCode_schedAdmin` at parity with that existing gap (project owner decision: not a `scheduler-core` blocker), so this item now covers both opcodes; when `validate_host()` is implemented, wire it to both `OP_upgradeConfig` and `OPCode_schedAdmin` in the same pass. **(b) Worker/Scheduler→Controller**: workers and schedulers connect via `controller_connector.c` using the same TLS context as controller peers, with no role differentiation — a compromised worker could send `OP_upgradeConfig` or read full config via `OP_replConfig`. **(c) Cross-component trust**: no distinction between controller peers, admin tools, workers, and schedulers — they share the same X.509 identity pool. **(d) xceadmin login (future)**: xceadmin currently uses zero-config TLS (same as xce_test). Production deployments likely need an admin login requirement — token-based session (`xceadmin login → token → config push`) or password-per-command. X.509 CN alone proves identity but not intent; a login step provides audit trail, session expiry, and operator accountability. **Needs design**: role-based X.509 CN validation (controller/admin/worker/scheduler roles), per-role OP code allowlists (now including `OPCode_schedAdmin`), admin login/session model, audit logging for admin operations. Separate from the existing `validate_host()` hardening (below) which is the minimal gate for `OP_upgradeConfig`/`OPCode_schedAdmin` specifically. Discuss + design before implementing; likely promotes to a spec via `/kiro-discovery`. +- [ ] **peer-authentication-hardening** — **NEW 2026-07-06 from admin-config-tool discovery** (`/kiro-discovery`). **Brief 2026-10-12: `.kiro/specs/peer-authentication-hardening/brief.md`** (user decisions: production/test mode on the controller, production by default; in production user requests are accepted only from an authenticated `apiserver` connection, harness/direct libxce are test tools; `xceadmin` joins as a new `admin` role; open decision: secret type (cluster key / per-role keys / PKI) and key distribution (files / token enrolment), recommendation per-role keys + files now, PKI with enrolment later). Correction: TLS is `SSL_VERIFY_NONE` everywhere, so not even a CA-signed certificate is required. Original text: XCE currently operates under a **trusted-LAN model** — any CA-signed SSL peer can connect to the controller and perform ANY operation. Three gaps identified: **(a) Admin→Controller**: `validate_host()` is a stub returning success for all peers (roadmap § below). `OP_upgradeConfig` has no real authorization gate today. **Same gap confirmed 2026-08-02 for `OPCode_schedAdmin`** (`scheduler-core` design, `cluster.c`'s `case OPCode_upgradeConfig` was checked as the closest precedent and has only a primary/standby routing gate, no RBAC/auth check) — `scheduler-core` ships `OPCode_schedAdmin` at parity with that existing gap (project owner decision: not a `scheduler-core` blocker), so this item now covers both opcodes; when `validate_host()` is implemented, wire it to both `OP_upgradeConfig` and `OPCode_schedAdmin` in the same pass. **(b) Worker/Scheduler→Controller**: workers and schedulers connect via `controller_connector.c` using the same TLS context as controller peers, with no role differentiation — a compromised worker could send `OP_upgradeConfig` or read full config via `OP_replConfig`. **(c) Cross-component trust**: no distinction between controller peers, admin tools, workers, and schedulers — they share the same X.509 identity pool. **(d) xceadmin login (future)**: xceadmin currently uses zero-config TLS (same as xce_test). Production deployments likely need an admin login requirement — token-based session (`xceadmin login → token → config push`) or password-per-command. X.509 CN alone proves identity but not intent; a login step provides audit trail, session expiry, and operator accountability. **Needs design**: role-based X.509 CN validation (controller/admin/worker/scheduler roles), per-role OP code allowlists (now including `OPCode_schedAdmin`), admin login/session model, audit logging for admin operations. Separate from the existing `validate_host()` hardening (below) which is the minimal gate for `OP_upgradeConfig`/`OPCode_schedAdmin` specifically. Discuss + design before implementing; likely promotes to a spec via `/kiro-discovery`. - [ ] validate_host() real authorization — currently a stub returning success for all peers (`join.c:13-16`, `return 1;` unconditionally). **`CONF_ENABLE_UPGRADE_CONFIG` is already `1` in the standard build** (`config_repl.h:27-28`, no build-system override found) — the precondition this item was originally framed to guard against has already been crossed; `OP_upgradeConfig` (and `OPCode_schedAdmin`, added 2026-08-02, `scheduler-core`) are live and unauthenticated TODAY, not a future risk to prevent. Minimum fix: verify peer X.509 CN against a configured allowlist (`--admin-cn` flag or `admin_cn_list` config), reject non-allowlisted `OP_upgradeConfig` and `OPCode_schedAdmin` with clear error, WARN-log rejected attempts with CN+IP. Future: IP allowlist, role-based access, admin-specific CA. _(FUTURE_WORK.md §6; security-critical.)_ **Part of the broader `peer-authentication-hardening` item above.** - [ ] OP_replConfig access control hardening — currently serves the full config blob to any CA-signed SSL peer (trusted-LAN model). Consider: verify requester is a known `--peers` controller, rate-limit per peer, `minimum_role` check (controllers only). _(FUTURE_WORK.md §7.)_ **Part of the broader `peer-authentication-hardening` item above.** - [ ] lost-and-found drain command — `xcecli lost-and-found drain`: reassign or cancel sidelined jobs, reset `lost_and_found_job_count`. Automates operator cleanup after config upgrade. _(FUTURE_WORK.md §4.)_