API keys were a pure in-memory map (store/apikey.go) — every apiserver
restart silently revoked every service-account key (advisor, MCP
server) until re-issued, with only an inexplicable 401 to show for it.
Keys now persist to <top>/state/apikeys.json: hash → metadata only
(never the raw key, preserving the existing "a dump cannot leak usable
credentials" property), written atomically (temp file + fsync + rename,
0600) on each issuance, loaded at InitStores. The load also restores
the user→role mapping, without which a reloaded key would authenticate
but be denied every permission check. A missing file is a normal fresh
install; a corrupt one fails fast at startup instead of quietly
starting with zero keys. A failed save fails the issuance request —
the raw key is shown exactly once, so we must not hand out a key we
could not persist.
The memory-only path (no SetAPIKeyStatePath) is kept for unit tests.
Cluster-verified on t1-t3: issue key → restart apiserver → key still
authenticates (hosts OK, advisor:report POST published 1) — previously
impossible. Verify job submitted, ran, exited; all daemons healthy.
Co-Authored-By: Claude Code <noreply@anthropic.com>
9dbb75adb3