lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(spec): peer-authentication-hardening brief; production mode accepts user requests only from the apiserver

Browse source 
Promotes the roadmap item to a brief: production/test mode on the
controller, authenticated roles at join, per-role request allowlist with
user requests only from the apiserver, an admin role for xceadmin. Lays
out the open decisions on secret type (cluster key / per-role keys / PKI)
and key distribution (files / token enrolment), with a phased
recommendation. project-config Req 3.13 applies to test mode only.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-10-13 04:25:17 +00:00
parent 33edbeeaf0
commit a2bf160cc3
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
4 changed files with 179 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

172
.kiro/specs/peer-authentication-hardening/brief.md Normal file
View file

@ -0,0 +1,172 @@
# Brief: peer-authentication-hardening
(Roadmap backlog item since 2026-07-06, admin-config-tool discovery. Promoted
to a brief 2026-10-12 from the project-config RBAC discussion: in production
every user request goes through the apiserver, so the controller must reject
user requests from anything else. Open decision: secret type and key
distribution, below.)
## Problem
The controller cannot tell who is connected to it. Any host that reaches the
controller port can join under any role and send any request:
- join as `apiserver` and submit, kill or modify jobs as any user (the
submit carries the user name and the controller believes it), bypassing
all RBAC, project authorization (`project-config` Req 3.12) and audit;
- push a cluster config (`upgradeConfig`) or run `schedAdmin`;
- read the full config (`replConfig`), or act as a worker for any host
(fake usage, job state, resync answers).
Test tools (harness, direct libxce clients) make this easy to hit by
accident too: pointed at a production cluster, they bypass the apiserver.
## Current State
- Role is self-declared at join (`JoinData.role`, `controller.capnp:151`)
and stored per connection (`priv->role`, `controller/join.c:275`); a
connection that never joins counts as `client` (`cluster.c:1413`).
Nothing checks a role against the request it sends, except a few
role-specific paths (`opupd.c:26`).
- `validate_host()` returns 1 for everyone (`join.c:13-16`), and is only
called for worker joins.
- The peer key is the literal `"keepSecret"` in the controller
(`cluster.c:41`), worker (`worker/src/controller_connector.c:175`) and
harness (`harness/test.c:272`).
- TLS is server-side only with `SSL_VERIFY_NONE` everywhere: no client
certificates, no server verification by clients, by decision
(`security.md` "TLS", bug-1453). (The roadmap said "any CA-signed SSL
peer"; in fact no certificate is verified at all, so a MITM can sit
between any two components.) Deploy scripts generate a self-signed pair
per deploy (`tools/deploy-ha.sh:246`).
- Non-joined direct client: `xceadmin config push` sends `upgradeConfig`
without joining (`xceadmin/push.py:81`), through pycapnp, not libxce.
- The apiserver joins through libxce via cgo (`backend/clib.go:57`,
rejoin `backend/reconnect.go:107`); schedulers and workers through
libxce (`xcecli_join`, `lib/client.c:300`).
## Desired Outcome
- **Production and test mode** (user decision 2026-10-12). The controller
runs in production mode unless its local startup config says `test`
(never settable through the pushed cluster config). Test deployments
(harness, simulator, `deploy-ha.sh`, `/xce-deploy-ha`, `/xce-test-*`)
set test mode explicitly.
- **Authenticated roles.** A peer proves its role at join with a secret;
a failed proof closes the connection with `-EPERM`, logged with peer
address and claimed role (never the secret).
- **Per-role request allowlist** in production mode:
| Role | Allowed requests |
|---|---|
| `apiserver` | all user requests: submit, kill/signal/modify, queries, queue control, ... |
| `worker` | worker requests: join, heartbeat, status and usage reports, job events, resync answers |
| `scheduler` | scheduler requests: decisions, `schedAdmin` replies, config fetch |
| `controller` | replication, votes, `replConfig` |
| `admin` (new) | `upgradeConfig`, `schedAdmin`, read-only status |
| `client` (unauthenticated) | nothing in production; everything in test mode |
A request outside its role's list is rejected with `-EPERM` and logged.
Later roles (reporter, claim provider, provisioner; roadmap 101/106)
are added to the same table.
- **User requests only from the apiserver.** The controller trusts the
user name and authorization basis on a request only because it came
from an authenticated `apiserver` connection (`project-config` Req
3.12, 3.13).
- **`xceadmin` joins as `admin`** with its secret, so config push keeps
working without the apiserver (bootstrap, apiserver down).
- **Clients authenticate the controller too**, so a worker or apiserver
cannot be fooled by a fake controller (mutual proof).
- Rotation without downtime, and no secret ever in the WAL, a snapshot,
`cluster.bin`, logs or the pushed config.
## Open decision: secret type
Three candidates. All are verified at join only; afterwards the
long-lived connection is the identity (connections are never re-used
across roles).
| | A. One cluster key (shared, symmetric) | B. One key per role (shared, symmetric) | C. PKI: cert per host from a cluster CA (mTLS) |
|---|---|---|---|
| Proof at join | HMAC challenge-response, both directions | same, key chosen by role | TLS client cert; role from the cert (e.g. OU), host from CN/SAN |
| A stolen worker key lets you be | anything, including `apiserver` and `admin` | any worker host, nothing else | that one host, as a worker |
| Per-host identity (worker A can't report for host B) | no | no | **yes** |
| Revoke one host | rotate the key everywhere | rotate the worker key on all workers | revoke that cert (CRL / short-lived certs) |
| Fixes MITM | yes, if the proof is bound to the TLS session (exporter, RFC 5705) | same | yes (both sides verify certs) |
| Cost | lowest: one file | low: 4-5 files, each on the hosts of its role | CA management, issuing and renewing a cert per host, expiry outages; reverses `security.md`'s no-mTLS decision (this spec would be that design doc) |
| Implementations to write | C (libxce), Python (`xceadmin`) | same | OpenSSL config in C, `ssl` in Python; mostly configuration, little custom crypto |
| HPC precedent | Slurm MUNGE (`munge.key` on every node) | Slurm `auth/slurm` / JWT keys per use | Kubernetes (cluster CA, kubelet certs) |
Notes:
- A and B need a custom handshake. The proof must be bound to the TLS
session (`SSL_export_keying_material`); otherwise a MITM, which
`SSL_VERIFY_NONE` allows, just relays the challenge and the answer.
With binding, A/B also gives clients server authentication without
certificate checks.
- C gives per-host identity, which A and B never can. That matters most
for workers (thousands of hosts, run user code) and for elastic hosts.
- B can be extended later to per-host worker keys derived from the worker
key and the hostname (HKDF(worker_key, host)): the controller derives the
expected key; a stolen per-host key lets you be only that host. Each
host then still needs its own file, so it shares C's distribution
problem without C's revocation.
## Open decision: key distribution
| | 1. Operator copies files | 2. Bootstrap token, then enrolment | 3. xce-node-manager places them |
|---|---|---|---|
| How | config management (Ansible, Nix, images) installs the key file (root, 0600) on each host, as sites do for `munge.key` | admin runs `xceadmin token create --role worker --ttl 1h`; a new host presents the one-time token once (pinning the controller's cert hash, like `kubeadm join --discovery-token-ca-cert-hash`) and receives its long-lived credential (a per-host key or a signed cert), stored locally | the local node manager (`xce-node-manager`) reads key paths from its node role manifest and passes them to each component; it does not obtain them |
| Fits | A, B | B (per-host keys), C | any (it is where the files are referenced, not how they arrive) |
| Elastic hosts (`elastic-hosts`, cloud burst) | the key must be baked into the image or user data: long-lived secret in cloud metadata | short-lived token in user data; nothing long-lived leaves the cluster | n/a |
| Air-gapped sites (`product.md`) | works | works (no external CA) | works |
| HA | each controller gets the same files | the issuing secret (CA key or root key) must be on every controller to enrol through any of them; it is a file on each controller, never replicated through the WAL | n/a |
## Recommendation (for decision)
Phase it, with the authenticator pluggable like the apiserver auth backends:
1. **Now: B (per-role keys) + distribution 1 (files), referenced by the
node manager (3).** Smallest change that delivers everything in
"Desired Outcome": apiserver-only user requests, the `admin` role,
mutual authentication, MITM fixed through session binding, a stolen
worker key unable to become `apiserver`. Matches HPC practice. Key
files hold a list of keys with ids, so rotation is: add the new key on
controllers, roll peers, remove the old key.
2. **Later, when elastic hosts or per-host worker identity are needed:**
C with distribution 2 (token enrolment, controller-issued certs with
short lifetimes, so renewal replaces revocation lists). Per-role keys
can stay as the enrolment trust for the apiserver and admin.
Going straight to C is the alternative if per-host worker identity is a
v1 requirement; it costs a CA, enrolment and renewal up front.
## Scope
- **In**: production/test mode; join authentication (mutual, session
bound); per-role request allowlist (including `upgradeConfig`,
`schedAdmin`, `replConfig`); `admin` role for `xceadmin`; key files,
rotation, startup validation; libxce (C), apiserver (cgo), `xceadmin`
(Python), harness/test-mode support; deploy scripts and skills; logs of
rejected joins and requests.
- **Out**: `xceadmin` login and audit sessions (roadmap item (d), later);
per-host identity and token enrolment (phase 2, unless chosen now);
apiserver HTTPS posture (`security.md` HAProxy follow-up).
## Upstream / Downstream
- **Upstream**: libxce join (`lib/client.c`, `lib/request.c:1414`),
controller join and dispatch (`join.c`, `cluster.c`), TLS context
(`lib/xcectx.c`), `xceadmin push.py`.
- **Downstream**: `project-config` (Req 3.12/3.13 rely on authenticated
apiserver connections), `elastic-hosts`, `resource-provisioner` and
reporter/claim-provider roles (roadmap 101/106), `xce-node-manager`
(key file paths in the node manifest).
## Existing Spec Touchpoints
- **Adjacent**: `xce-node-manager`, `elastic-hosts`, `admin-config-tool`,
`libxce-failover` / rejoin paths (a rejoin re-authenticates),
`ha-peer-connection-reuse` (controller peer connections authenticate as
`controller`).
## Constraints
- HA: a full spec workflow (security + every component's join). Join
authentication runs on the I/O path: HMAC and session-key export are
cheap and non-blocking; nothing on the delaymgr; no file reads after
startup except an explicit key reload.
- Connections stay long-lived; a failed authentication closes only the
unauthenticated connection. A controller restarted with rotated keys
must still accept peers holding any listed key, so a rolling rotation
never triggers failover.
- Secrets never in WAL, snapshot, `cluster.bin`, logs or replies.
- Test mode must keep the existing harness, simulator and skills working
unchanged apart from setting the mode.

4
.kiro/specs/project-config/brief.md
View file

@ -106,6 +106,7 @@ projects:
## Upstream / Downstream
- **Upstream**: config pipeline (`admin-config-tool`, `config-upgrade-snapshot`),
`peer-authentication-hardening` (authenticated apiserver connections),
limit plugin (`plugins/limit.c`, `lib/xcelimit.c`), RBAC (`rbac_check.go`).
- **Downstream**: `job-config-merge` (project level), `project-guarantee`,
`limits-to-owners`, `lsf-config-migration`, `rbac-persistence` (if access uses RBAC scopes).
@ -121,7 +122,8 @@ projects:
apiserver resolves the project and checks `project:submit` (membership
and role grants); the controller records the project and authorization
basis it is given and never re-checks RBAC. A direct submit is recorded
as given with basis `direct` (Req 3.13).
as given with basis `direct` in test mode (Req 3.13); in production mode
`peer-authentication-hardening` rejects it.
- The job's authorization basis (Req 3.11) is a new job field: WAL fact and
scheduler snapshot must carry it (field completeness, `persistence.md`).

3
.kiro/specs/project-config/requirements.md
View file

@ -78,6 +78,7 @@ This feature makes projects first-class cluster configuration entities that cont
- aggregate caps on queues (dropped by decision 2026-10-12, not deferred).
- **Adjacent expectations**:
- Project quotas are written on the project (decision 2026-10-12, reversing "`limits.yaml` stays the place where project quotas are written"). The scheduler's limit plugin enforces them as generated limits; admins never write those limits.
- `peer-authentication-hardening` makes the controller accept user requests in production mode only from an authenticated apiserver connection; Requirement 3.12 relies on it, and until it lands the controller's trust in the apiserver's authorization is a trusted-network assumption.
- Quotas count what the scheduler allocated or reserved, not measured use; a job without a memory reservation can exceed its project's memory quota at run time (accepted; profile default reservations close the gap).
- User groups come from the existing user-group configuration.
- Grants made through the role API are subject to the existing `rbac-persistence` gap (not durable across apiserver restart) until that spec lands.
@ -126,7 +127,7 @@ This feature makes projects first-class cluster configuration entities that cont
10. The Cluster shall check `project:submit` in addition to, not instead of, the existing queue permission.
11. The Cluster shall record on the job whether its project was authorized by membership, by a role grant or by the cluster administrator role.
12. The apiserver shall be the only component that resolves the project and checks `project:submit`; the controller shall record the project and authorization basis it receives with the submission without checking them again.
13. When a job is submitted directly to the controller, bypassing the apiserver (harness and client-library test paths, not supported for production), the Cluster shall record the project name as given (an empty name as `default`) with the authorization basis `direct`, without project resolution or authorization; Requirement 5 still applies to the pending job, except the membership hold of 5.2.
13. While the controller runs in test mode, when a job is submitted directly to the controller, bypassing the apiserver (harness and client-library test paths), the Cluster shall record the project name as given (an empty name as `default`) with the authorization basis `direct`, without project resolution or authorization; Requirement 5 still applies to the pending job, except the membership hold of 5.2. (In production mode such a submit is rejected by `peer-authentication-hardening`.)
### Requirement 4: Open mode when no projects are defined
**Objective:** As an operator of an existing cluster, I want clusters without project definitions to keep working as today, so that upgrading does not break existing submissions.