# Brief: rbac-persistence

## Problem
Cluster admins manage roles and permissions through the apiserver's REST API:
- create and delete roles;
- add permissions to a role;
- assign roles to users and remove them.

Those changes live only in apiserver memory. When the apiserver restarts, every
change made through the API since startup **silently reverts** to whatever
`config/rbac.yaml` says. There is no error, warning or audit trail. An admin who
granted a user access (or, worse, **revoked** it) finds the change gone after a
routine restart or upgrade. A revoked permission silently coming back is a
security problem.

This matters more now:
- `application-profile` adds `app:use`/`app:view` grants, which admins will
  manage per profile.
- The job-sandbox initiative relies on RBAC to decide who may run which
  applications, including sandbox versus direct execution.

## Current State
Documented in `.kiro/steering/security.md` as a known, deferred gap (flagged
2026-08-27 during the `scheduler-snapshot-job-allocation` persistence survey).

- **Loading.** `store.Roles` and `store.UserRoles` (`apiserver/store/stores.go`)
  are loaded from `config/rbac.yaml` at startup (`LoadRBACConfig()`,
  `controllers/rbac_check.go`).
- **Changes made at runtime with no write-back:**
  - `RoleController.CreateRole` / `AddPermissionsForRole` /
    `DeleteRoleAndPermissions` (`controllers/roles.go:499`, `:540`, `:594`);
  - `store.AddRoleForUser` / `DeleteRoleForUser`, called from the automatic role
    assignment in `AuthController.Logon` and from the user-role endpoints in
    `controllers/users.go`.
- **Model.** Permission strings are `action:scope` with domain types `all`,
  `self`, `self_ug`, `queue` (with values) and `hostgroup` (with values).
  `cluster_admin` bypasses checks through one early return.
- **Failure class.** The same save/restore field-completeness failure that
  `persistence.md` describes, but in the apiserver's in-memory RBAC store.
- **HA.** The apiserver is a separate process fed by controller push. Nothing
  covers how several apiserver instances would share RBAC state; the design must
  settle this.

## Desired Outcome
- Every RBAC change accepted by the API is **durable**. It survives an apiserver
  restart, or it is rejected; never silently lost.
- Revocations are durable exactly like grants.
- Every RBAC change is **audited**: who changed what, when, and old versus new
  value.
- On startup, the effective RBAC state equals the last successfully persisted
  state. Conflicts between the shipped `rbac.yaml` defaults and API changes are
  resolved by a documented, predictable rule.
- If there are several apiserver instances, they converge on the same RBAC state
  (the design decides how).

## Approach
Design decides. Candidates noted at discovery:
- **(a) Write-through to `rbac.yaml`.** Atomic write (temp file plus rename),
  preserving the builtin roles. Simple, but a single node's file, and it merges
  badly with shipped defaults.
- **(b) Store RBAC state in the controller's replicated state.** A new cluster
  entity persisted through the WAL and snapshots. HA-consistent and shared by
  every apiserver, but a much larger HA/persistence change.
- **(c) A small embedded store** (for example SQLite or bbolt) in the apiserver.
  Durable, but local to the node.
- **(d) Declare API changes runtime-only and reject them.** Make `rbac.yaml` (or
  xceadmin-managed config) the only source, and turn the mutating endpoints into
  errors or read-only. The smallest change, but it removes a capability.

## Scope
- **In**:
  - Durability of every mutating RBAC API: roles, permissions, user-role
    assignments, and automatic role assignment at login.
  - The audit trail.
  - The rule that reconciles shipped defaults with API changes at startup.
  - Multi-instance consistency (whatever the design chooses).
  - Tests, including restart round-trips of grants and revocations.
- **Out**:
  - New permission types (`application-profile` owns `app:*`).
  - Auth backends and JWT.
  - Changing the middleware chain or the `cluster_admin` bypass.

## Boundary Candidates
- RBAC state storage and load (the persistence mechanism).
- Mutating-API write path (durability, error on failure).
- Audit log.
- Reconciling defaults with runtime changes.

## Out of Boundary
- The permission model's semantics (actions, domain types).
- Profile, queue and other resource-specific permission checks.

## Upstream / Downstream
- **Upstream**: the existing RBAC store and APIs. If approach (b) is chosen, the
  controller WAL/snapshot machinery.
- **Downstream**: `application-profile` (durable `app:use` grants) and any
  future RBAC-governed feature.

## Existing Spec Touchpoints
- **Extends**: none (closes a gap recorded in `security.md`).
- **Adjacent**: `application-profile`, `admin-config-tool` (if RBAC moves into
  xceadmin-managed config).

## Constraints
- Middleware chain unchanged: WhitelistURL → WhitelistIP → RateLimit → JWT →
  CORS → RBAC.
- Fail closed. If a change cannot be persisted, the API returns an error and the
  in-memory state is left unchanged.
- Never log secrets (`security.md`).
- If approach (b): HA/persistence rules, and the full 3-phase workflow.
- When done, update `security.md` to remove the "Known gap" note.