docs(spec): peer-authentication-hardening brief; production mode accepts user requests only from the apiserver

Promotes the roadmap item to a brief: production/test mode on the
controller, authenticated roles at join, per-role request allowlist with
user requests only from the apiserver, an admin role for xceadmin. Lays
out the open decisions on secret type (cluster key / per-role keys / PKI)
and key distribution (files / token enrolment), with a phased
recommendation. project-config Req 3.13 applies to test mode only.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.kiro/specs/peer-authentication-hardening/brief.md

@@ -0,0 +1,172 @@
+# Brief: peer-authentication-hardening
+
+(Roadmap backlog item since 2026-07-06, admin-config-tool discovery. Promoted
+to a brief 2026-10-12 from the project-config RBAC discussion: in production
+every user request goes through the apiserver, so the controller must reject
+user requests from anything else. Open decision: secret type and key
+distribution, below.)
+
+## Problem
+The controller cannot tell who is connected to it. Any host that reaches the
+controller port can join under any role and send any request:
+- join as `apiserver` and submit, kill or modify jobs as any user (the
+  submit carries the user name and the controller believes it), bypassing
+  all RBAC, project authorization (`project-config` Req 3.12) and audit;
+- push a cluster config (`upgradeConfig`) or run `schedAdmin`;
+- read the full config (`replConfig`), or act as a worker for any host
+  (fake usage, job state, resync answers).
+
+Test tools (harness, direct libxce clients) make this easy to hit by
+accident too: pointed at a production cluster, they bypass the apiserver.
+
+## Current State
+- Role is self-declared at join (`JoinData.role`, `controller.capnp:151`)
+  and stored per connection (`priv->role`, `controller/join.c:275`); a
+  connection that never joins counts as `client` (`cluster.c:1413`).
+  Nothing checks a role against the request it sends, except a few
+  role-specific paths (`opupd.c:26`).
+- `validate_host()` returns 1 for everyone (`join.c:13-16`), and is only
+  called for worker joins.
+- The peer key is the literal `"keepSecret"` in the controller
+  (`cluster.c:41`), worker (`worker/src/controller_connector.c:175`) and
+  harness (`harness/test.c:272`).
+- TLS is server-side only with `SSL_VERIFY_NONE` everywhere: no client
+  certificates, no server verification by clients, by decision
+  (`security.md` "TLS", bug-1453). (The roadmap said "any CA-signed SSL
+  peer"; in fact no certificate is verified at all, so a MITM can sit
+  between any two components.) Deploy scripts generate a self-signed pair
+  per deploy (`tools/deploy-ha.sh:246`).
+- Non-joined direct client: `xceadmin config push` sends `upgradeConfig`
+  without joining (`xceadmin/push.py:81`), through pycapnp, not libxce.
+- The apiserver joins through libxce via cgo (`backend/clib.go:57`,
+  rejoin `backend/reconnect.go:107`); schedulers and workers through
+  libxce (`xcecli_join`, `lib/client.c:300`).
+
+## Desired Outcome
+- **Production and test mode** (user decision 2026-10-12). The controller
+  runs in production mode unless its local startup config says `test`
+  (never settable through the pushed cluster config). Test deployments
+  (harness, simulator, `deploy-ha.sh`, `/xce-deploy-ha`, `/xce-test-*`)
+  set test mode explicitly.
+- **Authenticated roles.** A peer proves its role at join with a secret;
+  a failed proof closes the connection with `-EPERM`, logged with peer
+  address and claimed role (never the secret).
+- **Per-role request allowlist** in production mode:
+
+  | Role | Allowed requests |
+  |---|---|
+  | `apiserver` | all user requests: submit, kill/signal/modify, queries, queue control, ... |
+  | `worker` | worker requests: join, heartbeat, status and usage reports, job events, resync answers |
+  | `scheduler` | scheduler requests: decisions, `schedAdmin` replies, config fetch |
+  | `controller` | replication, votes, `replConfig` |
+  | `admin` (new) | `upgradeConfig`, `schedAdmin`, read-only status |
+  | `client` (unauthenticated) | nothing in production; everything in test mode |
+
+  A request outside its role's list is rejected with `-EPERM` and logged.
+  Later roles (reporter, claim provider, provisioner; roadmap 101/106)
+  are added to the same table.
+- **User requests only from the apiserver.** The controller trusts the
+  user name and authorization basis on a request only because it came
+  from an authenticated `apiserver` connection (`project-config` Req
+  3.12, 3.13).
+- **`xceadmin` joins as `admin`** with its secret, so config push keeps
+  working without the apiserver (bootstrap, apiserver down).
+- **Clients authenticate the controller too**, so a worker or apiserver
+  cannot be fooled by a fake controller (mutual proof).
+- Rotation without downtime, and no secret ever in the WAL, a snapshot,
+  `cluster.bin`, logs or the pushed config.
+
+## Open decision: secret type
+Three candidates. All are verified at join only; afterwards the
+long-lived connection is the identity (connections are never re-used
+across roles).
+
+| | A. One cluster key (shared, symmetric) | B. One key per role (shared, symmetric) | C. PKI: cert per host from a cluster CA (mTLS) |
+|---|---|---|---|
+| Proof at join | HMAC challenge-response, both directions | same, key chosen by role | TLS client cert; role from the cert (e.g. OU), host from CN/SAN |
+| A stolen worker key lets you be | anything, including `apiserver` and `admin` | any worker host, nothing else | that one host, as a worker |
+| Per-host identity (worker A can't report for host B) | no | no | **yes** |
+| Revoke one host | rotate the key everywhere | rotate the worker key on all workers | revoke that cert (CRL / short-lived certs) |
+| Fixes MITM | yes, if the proof is bound to the TLS session (exporter, RFC 5705) | same | yes (both sides verify certs) |
+| Cost | lowest: one file | low: 4-5 files, each on the hosts of its role | CA management, issuing and renewing a cert per host, expiry outages; reverses `security.md`'s no-mTLS decision (this spec would be that design doc) |
+| Implementations to write | C (libxce), Python (`xceadmin`) | same | OpenSSL config in C, `ssl` in Python; mostly configuration, little custom crypto |
+| HPC precedent | Slurm MUNGE (`munge.key` on every node) | Slurm `auth/slurm` / JWT keys per use | Kubernetes (cluster CA, kubelet certs) |
+
+Notes:
+- A and B need a custom handshake. The proof must be bound to the TLS
+  session (`SSL_export_keying_material`); otherwise a MITM, which
+  `SSL_VERIFY_NONE` allows, just relays the challenge and the answer.
+  With binding, A/B also gives clients server authentication without
+  certificate checks.
+- C gives per-host identity, which A and B never can. That matters most
+  for workers (thousands of hosts, run user code) and for elastic hosts.
+- B can be extended later to per-host worker keys derived from the worker
+  key and the hostname (HKDF(worker_key, host)): the controller derives the
+  expected key; a stolen per-host key lets you be only that host. Each
+  host then still needs its own file, so it shares C's distribution
+  problem without C's revocation.
+
+## Open decision: key distribution
+| | 1. Operator copies files | 2. Bootstrap token, then enrolment | 3. xce-node-manager places them |
+|---|---|---|---|
+| How | config management (Ansible, Nix, images) installs the key file (root, 0600) on each host, as sites do for `munge.key` | admin runs `xceadmin token create --role worker --ttl 1h`; a new host presents the one-time token once (pinning the controller's cert hash, like `kubeadm join --discovery-token-ca-cert-hash`) and receives its long-lived credential (a per-host key or a signed cert), stored locally | the local node manager (`xce-node-manager`) reads key paths from its node role manifest and passes them to each component; it does not obtain them |
+| Fits | A, B | B (per-host keys), C | any (it is where the files are referenced, not how they arrive) |
+| Elastic hosts (`elastic-hosts`, cloud burst) | the key must be baked into the image or user data: long-lived secret in cloud metadata | short-lived token in user data; nothing long-lived leaves the cluster | n/a |
+| Air-gapped sites (`product.md`) | works | works (no external CA) | works |
+| HA | each controller gets the same files | the issuing secret (CA key or root key) must be on every controller to enrol through any of them; it is a file on each controller, never replicated through the WAL | n/a |
+
+## Recommendation (for decision)
+Phase it, with the authenticator pluggable like the apiserver auth backends:
+1. **Now: B (per-role keys) + distribution 1 (files), referenced by the
+   node manager (3).** Smallest change that delivers everything in
+   "Desired Outcome": apiserver-only user requests, the `admin` role,
+   mutual authentication, MITM fixed through session binding, a stolen
+   worker key unable to become `apiserver`. Matches HPC practice. Key
+   files hold a list of keys with ids, so rotation is: add the new key on
+   controllers, roll peers, remove the old key.
+2. **Later, when elastic hosts or per-host worker identity are needed:**
+   C with distribution 2 (token enrolment, controller-issued certs with
+   short lifetimes, so renewal replaces revocation lists). Per-role keys
+   can stay as the enrolment trust for the apiserver and admin.
+
+Going straight to C is the alternative if per-host worker identity is a
+v1 requirement; it costs a CA, enrolment and renewal up front.
+
+## Scope
+- **In**: production/test mode; join authentication (mutual, session
+  bound); per-role request allowlist (including `upgradeConfig`,
+  `schedAdmin`, `replConfig`); `admin` role for `xceadmin`; key files,
+  rotation, startup validation; libxce (C), apiserver (cgo), `xceadmin`
+  (Python), harness/test-mode support; deploy scripts and skills; logs of
+  rejected joins and requests.
+- **Out**: `xceadmin` login and audit sessions (roadmap item (d), later);
+  per-host identity and token enrolment (phase 2, unless chosen now);
+  apiserver HTTPS posture (`security.md` HAProxy follow-up).
+
+## Upstream / Downstream
+- **Upstream**: libxce join (`lib/client.c`, `lib/request.c:1414`),
+  controller join and dispatch (`join.c`, `cluster.c`), TLS context
+  (`lib/xcectx.c`), `xceadmin push.py`.
+- **Downstream**: `project-config` (Req 3.12/3.13 rely on authenticated
+  apiserver connections), `elastic-hosts`, `resource-provisioner` and
+  reporter/claim-provider roles (roadmap 101/106), `xce-node-manager`
+  (key file paths in the node manifest).
+
+## Existing Spec Touchpoints
+- **Adjacent**: `xce-node-manager`, `elastic-hosts`, `admin-config-tool`,
+  `libxce-failover` / rejoin paths (a rejoin re-authenticates),
+  `ha-peer-connection-reuse` (controller peer connections authenticate as
+  `controller`).
+
+## Constraints
+- HA: a full spec workflow (security + every component's join). Join
+  authentication runs on the I/O path: HMAC and session-key export are
+  cheap and non-blocking; nothing on the delaymgr; no file reads after
+  startup except an explicit key reload.
+- Connections stay long-lived; a failed authentication closes only the
+  unauthenticated connection. A controller restarted with rotated keys
+  must still accept peers holding any listed key, so a rolling rotation
+  never triggers failover.
+- Secrets never in WAL, snapshot, `cluster.bin`, logs or replies.
+- Test mode must keep the existing harness, simulator and skills working
+  unchanged apart from setting the mode.

.kiro/specs/project-config/brief.md

@@ -106,6 +106,7 @@ projects:
 
 ## Upstream / Downstream
 - **Upstream**: config pipeline (`admin-config-tool`, `config-upgrade-snapshot`),
+  `peer-authentication-hardening` (authenticated apiserver connections),
   limit plugin (`plugins/limit.c`, `lib/xcelimit.c`), RBAC (`rbac_check.go`).
 - **Downstream**: `job-config-merge` (project level), `project-guarantee`,
   `limits-to-owners`, `lsf-config-migration`, `rbac-persistence` (if access uses RBAC scopes).
@@ -121,7 +122,8 @@ projects:
   apiserver resolves the project and checks `project:submit` (membership
   and role grants); the controller records the project and authorization
   basis it is given and never re-checks RBAC. A direct submit is recorded
-  as given with basis `direct` (Req 3.13).
+  as given with basis `direct` in test mode (Req 3.13); in production mode
+  `peer-authentication-hardening` rejects it.
 - The job's authorization basis (Req 3.11) is a new job field: WAL fact and
   scheduler snapshot must carry it (field completeness, `persistence.md`).
 

.kiro/specs/project-config/requirements.md

@@ -78,6 +78,7 @@ This feature makes projects first-class cluster configuration entities that cont
   - aggregate caps on queues (dropped by decision 2026-10-12, not deferred).
 - **Adjacent expectations**:
   - Project quotas are written on the project (decision 2026-10-12, reversing "`limits.yaml` stays the place where project quotas are written"). The scheduler's limit plugin enforces them as generated limits; admins never write those limits.
+  - `peer-authentication-hardening` makes the controller accept user requests in production mode only from an authenticated apiserver connection; Requirement 3.12 relies on it, and until it lands the controller's trust in the apiserver's authorization is a trusted-network assumption.
   - Quotas count what the scheduler allocated or reserved, not measured use; a job without a memory reservation can exceed its project's memory quota at run time (accepted; profile default reservations close the gap).
   - User groups come from the existing user-group configuration.
   - Grants made through the role API are subject to the existing `rbac-persistence` gap (not durable across apiserver restart) until that spec lands.
@@ -126,7 +127,7 @@ This feature makes projects first-class cluster configuration entities that cont
 10. The Cluster shall check `project:submit` in addition to, not instead of, the existing queue permission.
 11. The Cluster shall record on the job whether its project was authorized by membership, by a role grant or by the cluster administrator role.
 12. The apiserver shall be the only component that resolves the project and checks `project:submit`; the controller shall record the project and authorization basis it receives with the submission without checking them again.
-13. When a job is submitted directly to the controller, bypassing the apiserver (harness and client-library test paths, not supported for production), the Cluster shall record the project name as given (an empty name as `default`) with the authorization basis `direct`, without project resolution or authorization; Requirement 5 still applies to the pending job, except the membership hold of 5.2.
+13. While the controller runs in test mode, when a job is submitted directly to the controller, bypassing the apiserver (harness and client-library test paths), the Cluster shall record the project name as given (an empty name as `default`) with the authorization basis `direct`, without project resolution or authorization; Requirement 5 still applies to the pending job, except the membership hold of 5.2. (In production mode such a submit is rejected by `peer-authentication-hardening`.)
 
 ### Requirement 4: Open mode when no projects are defined
 **Objective:** As an operator of an existing cluster, I want clusters without project definitions to keep working as today, so that upgrading does not break existing submissions.

.kiro/steering/roadmap.md

@@ -73,7 +73,7 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
 - [ ] controller-job-cleanup — NEW 2026-09-29 (`/kiro-discovery`, Path C). Primary-only periodic sweep (configurable `job_cleanup_period`, default 300 s) removes DONE/EXIT jobs finished longer than configurable `job_retention` (default 3600 s, 0 = off) with no dependents (stub; XCE has no job dependencies yet), via a NEW replicated WAL job-removal fact applied on every node (first-ever `jobmgr` delete; refcount-safe `xkvmgr` delete), propagated to scheduler and apiserver (`OnJobDeleted`). Retires the node-local snapshot-only `job_should_retain` finished-job filter (hard-coded 3600 s; causes snapshot/memory job-set drift). HA/persistence → full 3-phase workflow. Brief: `.kiro/specs/controller-job-cleanup/brief.md`. Dependencies: none (extends ha-entity-snapshots).
 - [ ] job-status-detail-display — NEW 2026-09-28 (`/kiro-discovery`, Path E). End users (REST/`xcecli`) see NO pend reason for ANY job today: the apiserver cgo conversion (`backend/convert.go`) never maps `xce_job_t.pend_reasons`, so `pending_reason` is always empty. Maps it to a structured JSON (including `alternatives` from day one) using libxce's code→text mapping, and renders it in `xcecli` job detail. Open for requirements: also map `run_usage` → `JobRusage` (same gap; reserved vs. used for RUN jobs). Brief: `.kiro/specs/job-status-detail-display/brief.md`. Dependencies: none for flat reasons; scheduler-alter-pend-reasons for per-alternative rendering.
 - [ ] job-config-merge — NEW 2026-10-09 (`/kiro-discovery`, Path E "configuration layering"; ref LSF 10.1 Admin Guide pp. 314-316, 435-437, 561-563, 643, 660). One merge function in `src/lib` turning the job's own values + application profile + queue policy + project into effective values (resreq incl. `span`, task range, limits, pre/post-exec, env). One owner per attribute (user decision; `.kiro/steering/job-config-layering.md`): the job may add to a profile's values but never change or remove them (additive rule, 2026-10-12, replacing *default*/*fixed* markers); queue ceilings and project quotas only check (reject if never satisfiable, pend if full now). **Live** merge (user decision): WAL keeps inputs + names; controller/apiserver validate at submit; scheduler computes effective values and re-evaluates pending jobs on config change. Queue is policy only: queue `res_req` and `pre_exec`/`post_exec` deprecated (pre/post-exec moves to the profile), dropped queue limits (cpulimit/memlimit/runlimit/task limit) wired. Brief: `.kiro/specs/job-config-merge/brief.md`. Dependencies: none.
-- [ ] project-config — NEW 2026-10-09 (configuration layering). Projects control resource ownership, v1 = quota: project definitions in cluster config (path identity, members, cluster default), default-project resolution, submit access check, scheduler `pgmgr` loaded from config, quotas written on the project (`quota`, `member_quota` over the subtree; user decision 2026-10-12, replacing `PROJECTS`/`PER_PROJECT` limits in `limits.yaml`) and compiled into the limit plugin, per-project usage display. Brief: `.kiro/specs/project-config/brief.md`. Dependencies: none.
+- [ ] project-config — NEW 2026-10-09 (configuration layering). Projects control resource ownership, v1 = quota: project definitions in cluster config (path identity, members, cluster default), default-project resolution, submit access check, scheduler `pgmgr` loaded from config, quotas written on the project (`quota`, `member_quota` over the subtree; user decision 2026-10-12, replacing `PROJECTS`/`PER_PROJECT` limits in `limits.yaml`) and compiled into the limit plugin, per-project usage display; `project:submit` RBAC implied by membership, project admins, authorization in the apiserver only. Brief: `.kiro/specs/project-config/brief.md`. Dependencies: none to ship; production rejection of direct submits comes from peer-authentication-hardening.
 
 - [ ] host-failure-job-recovery — NEW 2026-10-10 (`/kiro-discovery`, "Long-running services" phase 1; also a batch gap: RUN jobs on a dead host stay RUN forever, `controller/sched.c:631` FIXME). After a grace period, jobs on a lost host go to an unknown state, then are requeued (rerunnable, job option or profile default) or end EXIT "host lost"; multi-host jobs resolved as a whole; attempt numbers fence a stale copy when the host returns; manual requeue (REST/xcecli). Brief: `.kiro/specs/host-failure-job-recovery/brief.md`. Dependencies: none.
 - [ ] service-supervision — NEW 2026-10-10 (Long-running services phase 1). Application profile kind `service`: restart policy (always/on-failure/never, backoff, give-up), liveness and readiness probes (command/TCP/HTTP) run by each replica's own worker (no special replica), in-place restart, change-only health reports, display. Brief: `.kiro/specs/service-supervision/brief.md`. Dependencies: application-profile, host-failure-job-recovery.
@@ -158,7 +158,7 @@ Smaller items that likely don't warrant a full spec; pick up opportunistically o
 - [ ] **job_ctrl direct-mutate path — migrate to commit-then-apply** — **recorded 2026-07-13, investigated 2026-08-19 (finding: real risk, wider scope than originally framed).** `raft-apply-decoupling` deliberately scoped `job_ctrl`'s `mark_job_update`/`mark_job_fail` (admin SIGCONT/SIGSTOP) OUT-OF-BOUNDARY (task 4.1/6.1 Implementation Notes). The 2026-08-19 investigation found the original framing wrong in two ways: (1) **NOT the only remaining call site** — `on_job_stop`/`on_job_resume`/`on_job_ckpt` (`job.c:2067-2222`) share the identical unconverted mutate-then-notify-then-WAL-append pattern, fed by ordinary worker/scheduler traffic, not "rare, admin-only" as originally described; (2) **blast radius (a) is confirmed real, not theoretical** — the pre-commit mutation is synchronously pushed live to BOTH the scheduler (`sched_job_update_fun`) AND every connected apiserver/client (`apiserver_job_update_fun`) before the commit barrier resolves; on barrier timeout the admin gets a failure reply but `job->job.state` is never rolled back and no retraction is ever sent — matching the bug-795/828/851/1060 shape. Live-apply vs replay-apply (b) is narrower than feared for `job_ctrl` specifically (the `allocs != NULL` gate mutually excludes it from the converted deferred-apply fact types), but the investigation surfaced a SEPARATE bug underpinning that safety argument: `job_ctrl` (`job.c:999`) holds only a read lock while mutating job state — every other mutator in this file takes a write lock — so two concurrent admin commands on the same job aren't mutually excluded, undermining the "same lock, no staleness window" claim the replay-safety argument depends on (already flagged by a `raft-apply-decoupling` task 2.1 reviewer, never fixed). **Recommendation**: promote to a small follow-up spec covering all four call sites together (bounded, well-precedented fix — same shape as the fact-receipt conversion `raft-apply-decoupling` already built for job submit); the lock-type bug is independently fixable and low-risk regardless of when/whether the larger migration happens. Not yet promoted — user decision 2026-08-19 to correct scope/framing here first, promote later.
 - [x] **WAL-trim cursor-bound guard (bug-1147; surfaced 2026-06-25 by raft-drive-async design review)** — **IMPLEMENTED** as part of `raft-drive-async` task 3.2 (`raft-drive-async` is IMPLEMENTED, validate-impl GO 2026-07-18; see above). `trim_wal_segments` (`persist.c` ~2597-2644) now computes `floor = repl_trim_lsn(&p->repl, active_seg_lsn, p->config.snap_interval_s)` — folding in active-standby `acked_lsn` and disconnected-slot `last_disconnected_lsn` — and only unlinks segments below that floor; the code comment at `persist.c:2634` explicitly labels this the "(bug-1147 guard)". Far-behind/empty join (cursor=0) is additionally covered now that `entity-bearing snapshots` (strategy B) has landed as `ha-entity-snapshots` (see above), with a live decision path (`ha.c:4413`, "< oldest_wal_lsn → requesting snapshot transfer"). Related: bug-795, bug-828, bug-1060.
 - [ ] OP_heartbeat peers cross-validation — include a hash of the local `--peers` string in heartbeat requests; log ERROR on mismatch between candidates. Detects misconfigured nodes at election time instead of silently risking dual-primary split-brain. _(FUTURE_WORK.md §3.)_
-- [ ] **peer-authentication-hardening** — **NEW 2026-07-06 from admin-config-tool discovery** (`/kiro-discovery`). XCE currently operates under a **trusted-LAN model** — any CA-signed SSL peer can connect to the controller and perform ANY operation. Three gaps identified: **(a) Admin→Controller**: `validate_host()` is a stub returning success for all peers (roadmap § below). `OP_upgradeConfig` has no real authorization gate today. **Same gap confirmed 2026-08-02 for `OPCode_schedAdmin`** (`scheduler-core` design, `cluster.c`'s `case OPCode_upgradeConfig` was checked as the closest precedent and has only a primary/standby routing gate, no RBAC/auth check) — `scheduler-core` ships `OPCode_schedAdmin` at parity with that existing gap (project owner decision: not a `scheduler-core` blocker), so this item now covers both opcodes; when `validate_host()` is implemented, wire it to both `OP_upgradeConfig` and `OPCode_schedAdmin` in the same pass. **(b) Worker/Scheduler→Controller**: workers and schedulers connect via `controller_connector.c` using the same TLS context as controller peers, with no role differentiation — a compromised worker could send `OP_upgradeConfig` or read full config via `OP_replConfig`. **(c) Cross-component trust**: no distinction between controller peers, admin tools, workers, and schedulers — they share the same X.509 identity pool. **(d) xceadmin login (future)**: xceadmin currently uses zero-config TLS (same as xce_test). Production deployments likely need an admin login requirement — token-based session (`xceadmin login → token → config push`) or password-per-command. X.509 CN alone proves identity but not intent; a login step provides audit trail, session expiry, and operator accountability. **Needs design**: role-based X.509 CN validation (controller/admin/worker/scheduler roles), per-role OP code allowlists (now including `OPCode_schedAdmin`), admin login/session model, audit logging for admin operations. Separate from the existing `validate_host()` hardening (below) which is the minimal gate for `OP_upgradeConfig`/`OPCode_schedAdmin` specifically. Discuss + design before implementing; likely promotes to a spec via `/kiro-discovery`.
+- [ ] **peer-authentication-hardening** — **NEW 2026-07-06 from admin-config-tool discovery** (`/kiro-discovery`). **Brief 2026-10-12: `.kiro/specs/peer-authentication-hardening/brief.md`** (user decisions: production/test mode on the controller, production by default; in production user requests are accepted only from an authenticated `apiserver` connection, harness/direct libxce are test tools; `xceadmin` joins as a new `admin` role; open decision: secret type (cluster key / per-role keys / PKI) and key distribution (files / token enrolment), recommendation per-role keys + files now, PKI with enrolment later). Correction: TLS is `SSL_VERIFY_NONE` everywhere, so not even a CA-signed certificate is required. Original text: XCE currently operates under a **trusted-LAN model** — any CA-signed SSL peer can connect to the controller and perform ANY operation. Three gaps identified: **(a) Admin→Controller**: `validate_host()` is a stub returning success for all peers (roadmap § below). `OP_upgradeConfig` has no real authorization gate today. **Same gap confirmed 2026-08-02 for `OPCode_schedAdmin`** (`scheduler-core` design, `cluster.c`'s `case OPCode_upgradeConfig` was checked as the closest precedent and has only a primary/standby routing gate, no RBAC/auth check) — `scheduler-core` ships `OPCode_schedAdmin` at parity with that existing gap (project owner decision: not a `scheduler-core` blocker), so this item now covers both opcodes; when `validate_host()` is implemented, wire it to both `OP_upgradeConfig` and `OPCode_schedAdmin` in the same pass. **(b) Worker/Scheduler→Controller**: workers and schedulers connect via `controller_connector.c` using the same TLS context as controller peers, with no role differentiation — a compromised worker could send `OP_upgradeConfig` or read full config via `OP_replConfig`. **(c) Cross-component trust**: no distinction between controller peers, admin tools, workers, and schedulers — they share the same X.509 identity pool. **(d) xceadmin login (future)**: xceadmin currently uses zero-config TLS (same as xce_test). Production deployments likely need an admin login requirement — token-based session (`xceadmin login → token → config push`) or password-per-command. X.509 CN alone proves identity but not intent; a login step provides audit trail, session expiry, and operator accountability. **Needs design**: role-based X.509 CN validation (controller/admin/worker/scheduler roles), per-role OP code allowlists (now including `OPCode_schedAdmin`), admin login/session model, audit logging for admin operations. Separate from the existing `validate_host()` hardening (below) which is the minimal gate for `OP_upgradeConfig`/`OPCode_schedAdmin` specifically. Discuss + design before implementing; likely promotes to a spec via `/kiro-discovery`.
 - [ ] validate_host() real authorization — currently a stub returning success for all peers (`join.c:13-16`, `return 1;` unconditionally). **`CONF_ENABLE_UPGRADE_CONFIG` is already `1` in the standard build** (`config_repl.h:27-28`, no build-system override found) — the precondition this item was originally framed to guard against has already been crossed; `OP_upgradeConfig` (and `OPCode_schedAdmin`, added 2026-08-02, `scheduler-core`) are live and unauthenticated TODAY, not a future risk to prevent. Minimum fix: verify peer X.509 CN against a configured allowlist (`--admin-cn` flag or `admin_cn_list` config), reject non-allowlisted `OP_upgradeConfig` and `OPCode_schedAdmin` with clear error, WARN-log rejected attempts with CN+IP. Future: IP allowlist, role-based access, admin-specific CA. _(FUTURE_WORK.md §6; security-critical.)_ **Part of the broader `peer-authentication-hardening` item above.**
 - [ ] OP_replConfig access control hardening — currently serves the full config blob to any CA-signed SSL peer (trusted-LAN model). Consider: verify requester is a known `--peers` controller, rate-limit per peer, `minimum_role` check (controllers only). _(FUTURE_WORK.md §7.)_ **Part of the broader `peer-authentication-hardening` item above.**
 - [ ] lost-and-found drain command — `xcecli lost-and-found drain`: reassign or cancel sidelined jobs, reset `lost_and_found_job_count`. Automates operator cleanup after config upgrade. _(FUTURE_WORK.md §4.)_