# Design Document — controller-status (`ctrlstatus`)

## Overview

**Purpose:** add a single, cheap, read-only client→controller query — `ctrlstatus`, an XCE analog of LSF `lsid` — that returns basic cluster/controller identity and health: who is the current primary, the answering node's identity and role, the configured controllers (peers), the committed configuration version and content checksum, and a few HA health/progress fields (term, commit/apply position, fenced state, cluster name).

**Users:** operators inspecting a running cluster, and test/E2E authors orchestrating failover (including `wait-primary`, which switches its probe to this query).

**Impact:** today there is no status query — callers infer "is there a serving primary" indirectly (e.g. by listing jobs). This adds a new opcode by cloning the existing **opcode-only read** pattern of `clusterconf`, reads only already-maintained HA/config state (no behavior change), and crucially makes the query answerable on **any** node by adding it to the HA dispatch-gate allow-lists.

### Goals
- A new read-only opcode returning the status fields of Requirements 2–6 in one request/response (1.1, 1.4).
- Answerable on **any** node — a standby reports a hint view (the primary it knows); an **authoritative** answer comes only from the primary, reached by following the redirect / rotating the master list (2.3, 7.1–7.3).
- Report the loaded config version + content checksum (and the committed version) so same-version divergence is detectable (5.2, 5.3).
- A `ctrlstatus` test-client command (8.x) and a `wait-primary` probe switched to an authoritative (primary-only) status query (9.x).

### Non-Goals
- No operator-facing CLI/REST `lsid` surface (Go tier) — later pass.
- No change to HA election, replication, or commit behavior — the query only **reads** existing state.
- No new configuration checksum computation — the committed-config SHA-256 already exists; this only exposes it.

## Boundary Commitments

### This Spec Owns
- A new Cap'n Proto opcode `ctrlstatus` and its reply payload (`CtrlStatus`, `CtrlPeer`).
- The controller handler `handle_ctrlstatus` that snapshots local HA/config state into the reply.
- The **dispatch-gate allow-list entries** that let `ctrlstatus` answer in every controller role (incl. no-quorum).
- The libxce client API (`xce_ctrlstatus_req` / `xce_ctrlstatus` / `xcecli_ctrlstatus` + free helper).
- The `ctrlstatus` test-client command and the `wait-primary` probe switch.

### Out of Boundary
- The Go CLI/REST `lsid` surface (later spec).
- Any change to how HA state (role, term, primary identity, committed config tag, commit/apply LSN, fence) is **produced or maintained** — this spec only reads it.
- The existing `clusterconf` opcode and its resource/queue/worker payload (left unchanged).

### Allowed Dependencies
- The Cap'n Proto schema + build-time codegen (`src/schema/controller.capnp` → generated `controller.capnp.h/.c`).
- The existing request/reply plumbing (`init_reply_by_req`, `xce_reply`, opcode-only `*_req` senders) and client failover wrapper (`XCECLI_FAILOVER_CALL`, `is_write=false`).
- Read access to existing HA state (`ha_state_t`), config (`myconf_t`: `committed_config_tag`, `config_sha256`, `peers_cfg`), and persist accessors (`persist_recovery_unrecoverable`, `persist_durable_lsn`).

### Revalidation Triggers
- Any change to the `CtrlStatus`/`CtrlPeer` reply shape or the `ctrlstatus` opcode ordinal → re-check libxce decode + the test command + any future consumer.
- Any change to the dispatch-gate role allow-lists → re-check that `ctrlstatus` still answers in every role and that no other opcode's gating widened.
- A change to the meaning of `committed_config_tag` / `config_sha256` → re-check the version/checksum fields.

## Architecture

### Existing Architecture Analysis
- **Opcode-only read precedent:** `clusterconf` is sent with no request body (`Request_body_nomsg`), dispatched in `cluster.c` `switch(request.opcode)`, handled by `handle_clusterconf` (`reply.c:10`), and returned via `xce_reply`. The client side is `xce_clusterconf` → `xcecli_clusterconf` (`XCECLI_FAILOVER_CALL`, `is_write=false`). **`ctrlstatus` clones this pattern.**
- **No-alloc state-report precedent:** `ha_nop_reply_state_report` (`ha.c:1784`) reads HA atomics into a stack struct and points the reply union at it without heap allocation. `handle_ctrlstatus` follows this model (a small bounded peer list is the only allocation).
- **Dispatch gate (the critical seam):** `ha_dispatch_gate` (`ha.c:2059`) is an allow-list keyed by role. The `CTRL_STANDBY` branch defaults to `-XCE_ERR_STANDBY`; `CTRL_INIT`/`CTRL_ELECTING` default to `-XCE_ERR_NOT_READY`; the `CTRL_PRIMARY` no-quorum branch allow-lists only specific reads. `clusterconf` is **not** in any standby allow-list — it is implicitly primary-only-when-serving. A status query must **not** inherit that; it must be explicitly allow-listed in every role branch.
- **Config content checksum already exists:** `config_version_tag` = first 8 bytes of SHA-256 over the committed config blob; full digest in `config_sha256[32]`, round-trip-verified on load (`config_repl.c`).

### Pattern & dependency direction
Extend-in-place along the existing request path. Dependency direction (unchanged): `schema → libxce (request/reply/client) → controller handler`; the test client depends on libxce. No upward dependencies introduced.

```mermaid
graph LR
    TestClient[xce_test ctrlstatus and wait_primary] --> Libxce[libxce xcecli_ctrlstatus]
    Libxce --> Failover[XCECLI_FAILOVER_CALL is_write false]
    Failover --> Gate[ha_dispatch_gate allow_list all roles]
    Gate --> Handler[handle_ctrlstatus snapshots local state]
    Handler --> HAState[ha_state_t and myconf_t]
    Handler --> Reply[xce_reply CtrlStatus]
```

## File Structure Plan

### Modified Files
- `src/schema/controller.capnp` — add `OPCode ctrlstatus @40`; new `struct CtrlStatus $C.mapname("xce_ctrlstatus_t")` and `struct CtrlPeer $C.mapname("xce_ctrlpeer_t")`; reply-union member `ctrlstatus @17: CtrlStatus`. (Build regenerates `controller.capnp.h/.c` → `xce_ctrlstatus_t`, `RequestReply_data_ctrlstatus`, `free_CtrlStatus`.)
- `src/controller/reply.c` — add `handle_ctrlstatus` (snapshot local HA/config state into a `CtrlStatus` reply; small peer list; `xce_reply`). Mirrors `handle_clusterconf`/`ha_nop_reply_state_report`.
- `src/controller/cluster.c` — add `case OPCode_ctrlstatus: rc = handle_ctrlstatus(...)` to the opcode switch.
- `src/controller/ha.c` — add `OPCode_ctrlstatus` to the `ha_dispatch_gate` allow-lists for `CTRL_PRIMARY` (incl. the no-quorum branch), `CTRL_STANDBY`, `CTRL_ELECTING`, `CTRL_INIT`. **(the requirements-critical change)**
- `src/lib/request.c` + `src/include/request.h` — add `xce_ctrlstatus_req` (opcode-only sender, clone of `xce_clusterconf_req`).
- `src/lib/xceapi.c` — add `xce_ctrlstatus` (send req, `cbuf_get` status, `MOVE_PTR` reply union).
- `src/lib/client.c` + `src/include/client.h` — add `xcecli_ctrlstatus` (`XCECLI_FAILOVER_CALL`, `is_write=false`), the authoritative `xcecli_ctrlstatus_primary` (follows redirect / rotates the master list until `responderRole == PRIMARY`), and the `xcecli_free_ctrlstatus` macro.
- `src/controller/testsuite/test.c` — add `cmd_ctrlstatus` + table registration (uses `xcecli_ctrlstatus`, shows the responder's view); switch `cmd_wait_primary`'s probe (currently `joblist_query(...)` at the deadline-loop call site) to `xcecli_ctrlstatus_primary` (authoritative, primary-only success).

### New Files
- None (capnp C types are generated; all handlers/APIs slot into existing TUs).

## System Flows

### Status query: any-node hint then authoritative-from-primary
```mermaid
sequenceDiagram
    participant C as Client
    participant S as Standby node
    participant P as Primary node
    C->>S: ctrlstatus request
    S-->>C: reply responderRole STANDBY plus known primary hint
    Note over C,S: Standby answer is a non authoritative hint view
    C->>P: re-query targeting the reported primary
    P-->>C: authoritative reply responderRole PRIMARY
```
The dispatch gate allow-lists `ctrlstatus` in every role (incl. no-quorum), so any node answers with its local view (satisfying any-node visibility). An **authoritative** answer comes only from the node whose `responderRole == PRIMARY`. When the first reply is from a non-primary, the client follows the redirect — it maps the reply's `primaryName` to an address via the reply's `peers` list (or rotates the configured master list) and re-queries — until it obtains the primary's authoritative reply or exhausts its budget. The plain `ctrlstatus` command may display the first (hint) reply with its "answered by" framing; `wait-primary` and any authoritative use require the primary's reply.
Gate decision: `ctrlstatus` is allow-listed in every role branch and does not require quorum or a committed read — it returns the node's local view even when fenced or without quorum (precisely when an operator most wants it).

### wait-primary probe (Req 9)
`wait-primary` keeps its existing monotonic-deadline poll loop and timeout/exit semantics; only the probe changes from `joblist_query` to `xcecli_ctrlstatus`. Success predicate: the status reply reports a **non-`"unknown"`** current primary. On success it prints the served primary's identity. (Rationale + caveat below.)

## Requirements Traceability

| Requirement | Summary | Components | Interfaces |
|-------------|---------|------------|------------|
| 1.1, 1.2, 1.4 | Single-call read-only focused status | `handle_ctrlstatus`, `CtrlStatus` schema | opcode `ctrlstatus`, reply `CtrlStatus` |
| 1.3 | "unknown" for undeterminable fields | `handle_ctrlstatus` | empty Text/sentinel mapping |
| 2.1, 2.2, 2.3 | Current primary identity (unknown when unelected; standby's known primary) | `handle_ctrlstatus` (reads `ha->primary_host` under `ha->lock`) | `CtrlStatus.primaryName` |
| 3.1, 3.2, 3.3 | Answered-by node + role, distinct from primary | `handle_ctrlstatus` | `CtrlStatus.responderName`, `responderRole` |
| 4.1, 4.2 | Configured controllers + count | `handle_ctrlstatus` (reads `peers_cfg`) | `CtrlStatus.peers`, `peerCount` |
| 5.1, 5.2, 5.3 | Config version + content checksum (loaded pair + committed) | `handle_ctrlstatus` (reads loaded `config_version_tag`/`config_sha256` + `committed_config_tag`) | `CtrlStatus.configVersion`, `configChecksum`, `committedConfigVersion` |
| 6.1–6.4 | Term, commit/apply, fenced, cluster name | `handle_ctrlstatus` | `CtrlStatus.term`, `committedLsn`, `appliedLsn`, `fenced`, `clusterName` |
| 7.1, 7.2, 7.3 | Failover-aware any-node delivery | `xcecli_ctrlstatus`, dispatch-gate allow-lists | `XCECLI_FAILOVER_CALL` (is_write=false), gate entries |
| 8.1, 8.2, 8.3 | Test-client `ctrlstatus` command | `cmd_ctrlstatus` | client command + exit code |
| 9.1–9.4 | wait-primary uses an authoritative (primary-only) status probe | `cmd_wait_primary` probe swap, `xcecli_ctrlstatus_primary` | redirect-to-primary, `responderRole == PRIMARY` success |

## Components and Interfaces

| Component | Layer | Intent | Req Coverage | Key Dependencies | Contracts |
|-----------|-------|--------|--------------|------------------|-----------|
| `CtrlStatus`/`CtrlPeer` schema | Schema | Reply payload for the status query | 2–6 | capnp codegen (P0) | API (reply) |
| `handle_ctrlstatus` | Controller | Snapshot local HA/config state into the reply | 1–6 | `ha_state_t`, `myconf_t`, persist accessors (P0) | Service |
| dispatch-gate allow-lists | Controller HA | Permit `ctrlstatus` in every role | 7 | `ha_dispatch_gate` (P0) | Service |
| `xcecli_ctrlstatus` / `xcecli_ctrlstatus_primary` | libxce | Failover-aware read (any-node) + authoritative reach-primary read + free | 1, 7, 9 | failover wrapper, redirect hint (P0) | Service |
| `cmd_ctrlstatus` | Test client | Print the status (responder view) | 8 | `xcecli_ctrlstatus` (P0) | State |
| `cmd_wait_primary` probe | Test client | Authoritative serving-primary probe | 9 | `xcecli_ctrlstatus_primary` (P0) | State |

### Schema — reply payload (Cap'n Proto)
```capnp
struct CtrlPeer $C.mapname("xce_ctrlpeer_t") {
  name @0: Text;
  host @1: Text;
  port @2: UInt32;
}

struct CtrlStatus $C.mapname("xce_ctrlstatus_t") {
  clusterName       @0:  Text;
  primaryName       @1:  Text;    # empty ⇒ "unknown" (no elected/known primary)
  responderName     @2:  Text;    # the node that answered
  responderRole     @3:  UInt8;   # ctrl_role_t: INIT/ELECTING/PRIMARY/STANDBY
  term              @4:  UInt64;
  # Loaded config (consistent pair: version is the 8-byte prefix of the checksum)
  configVersion     @5:  UInt64;  # loaded config_version_tag (8-byte SHA-256 prefix)
  configChecksum    @6:  Data;    # loaded config_sha256 (full 32 bytes)
  # Committed config version (may differ from loaded mid config-change)
  committedConfigVersion @7: UInt64;  # committed_config_tag
  committedLsn      @8:  UInt64;
  appliedLsn        @9:  UInt64;
  fenced            @10: Bool;    # in_catch_up OR recovery_unrecoverable
  peers             @11: List(CtrlPeer) $C.mapname("peers.list") $C.maplistcount("peers.count");
  peerCount         @12: UInt32;
}
```
- **Config fields (Req 5):** `configVersion`/`configChecksum` are the **loaded** config's tag and full SHA-256 — a self-consistent pair (the version is the first 8 bytes of the checksum), so two nodes with identical loaded content match on both and divergent content differs on both (5.3). `committedConfigVersion` is the separately-tracked committed tag, exposing "what I run" vs "what is committed"; in steady state all three agree.
- New OPCode ordinal: `ctrlstatus @40` (next free after `preVoteReply @39`). New reply-union member: `ctrlstatus @17` (next free after `walackreplymsg @16`). Field ordinals are append-only — never reuse/reorder.

#### `handle_ctrlstatus` — Service contract
| Field | Detail |
|-------|--------|
| Intent | Build a `CtrlStatus` from the responding node's local HA/config state and reply |
| Requirements | 1.1–1.4, 2.1–2.3, 3.1–3.3, 4.1–4.2, 5.1–5.3, 6.1–6.4 |

**Responsibilities & Constraints**
- Read-only: must not modify any HA, persist, or config state (1.2).
- Field sourcing: `clusterName`/`responderName` from `c->myconf`/own peerinfo; `responderRole`, `term`, `appliedLsn`, `committedLsn`, `committedConfigVersion` (`committed_config_tag`) via `atomic_load_explicit(..., acquire)`; `configVersion` from the loaded `config_version_tag` and `configChecksum` from the loaded `config_sha256[32]` (a self-consistent pair — Req 5.3); `primaryName`/peers as below; `fenced = in_catch_up || persist_recovery_unrecoverable(...)`.
- **Locking:** `primary_host`/`primary_port` are guarded by `ha->lock` — take it for a consistent snapshot, copy out, release. All other fields are atomics or immutable config.
- **"unknown" mapping (1.3, 2.2, 2.3):** if `primary_host[0] == '\0'` (no known primary), emit an empty `primaryName`, which the client renders as `"unknown"`.
- **Peers (4.x):** iterate `c->myconf->peers_cfg.candidates[0..count-1]` → one `CtrlPeer{name,host,port}` each; set `peerCount`.
- Answerable in every role; returns the local view even when fenced/no-quorum.

**Dependencies**
- Inbound: `cluster.c` opcode switch — routes the request (P0).
- Outbound: `ha_state_t`, `myconf_t`, persist accessors — read-only snapshot (P0).
- Outbound: `init_reply_by_req`, `xce_reply` — reply encode/send (P0).

#### Dispatch-gate allow-lists — Service contract
- Add `OPCode_ctrlstatus` to the allow-list in `ha_dispatch_gate` for `CTRL_PRIMARY` (including the no-quorum branch), `CTRL_STANDBY`, `CTRL_ELECTING`, and `CTRL_INIT` (7.1, 7.2).
- Constraint: the change is scoped strictly to `ctrlstatus`; no other opcode's gating may widen. `ctrlstatus` never requires quorum or a committed read.

#### libxce client API — Service contract
```c
/* opcode-only request sender (clone of xce_clusterconf_req) */
int  xce_ctrlstatus_req(endpoint_t* ep);
/* send + await reply; on success MOVE_PTR the reply union out */
int  xce_ctrlstatus(xce_ctx_t* ctx, xce_ctrlstatus_t** status);
/* failover-aware read (is_write=false): returns whatever node answers (may be a
 * standby hint view). Use for the operator command. */
int  xcecli_ctrlstatus(xcecli_ctx_t c, xce_ctrlstatus_t** status);
/* AUTHORITATIVE read: returns only a reply whose responderRole == PRIMARY. If the
 * first answer is from a non-primary, follow the redirect — target the reported
 * primaryName via the reply's peers list, else rotate the configured master list
 * — until the primary answers or the failover budget is exhausted. */
int  xcecli_ctrlstatus_primary(xcecli_ctx_t c, xce_ctrlstatus_t** status);
#define xcecli_free_ctrlstatus(s)  xce_free_ctrlstatus(s)
```
- Preconditions: valid client context.
- Postconditions: on success `*status` owns a decoded `CtrlStatus` (caller frees); on failure returns a typed negative (7.3) and `*status` untouched. For `xcecli_ctrlstatus_primary`, success additionally guarantees `responderRole == PRIMARY` (an authoritative answer).
- Invariants: `is_write=false` — retryable/rotatable across the configured controllers.

#### `cmd_ctrlstatus` (test client) — State contract
- Calls `xcecli_ctrlstatus`; on success prints the fields in human-readable form (8.1) — `Primary`, `answered by <responderName> (<role>)` distinct from `Primary:` (3.3), peers + count, `configVersion` and `committedConfigVersion`, `configChecksum` as hex, term, commit/apply, fenced, cluster name; renders empty `primaryName` as `unknown`. Returns 0 on success, `<0` on failure with the error printed (8.2, 8.3). Frees the reply.

#### `cmd_wait_primary` probe swap — State contract
- Replace the single `joblist_query(...)` probe call in the deadline loop with `xcecli_ctrlstatus_primary` (the authoritative variant).
- **Success predicate (9.1, 9.2):** the probe obtains an authoritative reply with `responderRole == PRIMARY` (the answer came from the serving primary). On success, complete and print that primary's identity. A standby-only/hint answer is NOT success — the probe follows the redirect to the primary (or keeps polling) until the primary answers or the deadline elapses.
- **Decision (resolves review Issue 1):** require the answer to come **from the primary**, not merely "a primary name was reported by some node." This avoids a false success on a standby's stale `primary_host` after a primary death (the false-positive class removed from the durability soak) and faithfully meets Req 9.1's "**serving** primary." The any-node hint view remains available for the operator `ctrlstatus` command via `xcecli_ctrlstatus`.
- **Preserved (9.4):** the timeout-argument parsing, monotonic deadline, poll interval, log-level quieting, and success/failure exit codes are unchanged; only the probe call and the success message change.

## Data Models

The only new data is the reply payload above. `CtrlStatus` is a flat value object (no persistence, no aggregate); `CtrlPeer` is a bounded value list (≤ configured controller count, typically 3). `configChecksum` is the loaded config's 32-byte SHA-256 and `configVersion` is its own 8-byte prefix (a self-consistent pair); `committedConfigVersion` is the separately-tracked committed tag. Two nodes with identical loaded config content produce identical `configVersion` + `configChecksum`; differing content differs in both (5.3). A node mid config-change may show `configVersion != committedConfigVersion`.

## Error Handling

- **No reachable controller (7.3):** `xcecli_ctrlstatus` returns the failover wrapper's typed negative (e.g. no-master/timeout); `cmd_ctrlstatus` prints it and returns non-zero (8.3); `wait-primary` keeps polling until its deadline (9.3).
- **Undeterminable field (1.3):** emit the defined "unknown"/empty value, never uninitialized data.
- **Standby with no known primary (2.3):** `primaryName` empty → "unknown"; the call still succeeds (the status itself is valid).
- **Read-only guarantee (1.2):** the handler takes only read locks/atomic loads; no state mutation on any path.

## Testing Strategy

### Unit / handler
- `handle_ctrlstatus` on a PRIMARY returns `primaryName==self`, `responderRole==PRIMARY`, populated peers/version/checksum (2.1, 3.x, 4.x, 5.x).
- `handle_ctrlstatus` on a STANDBY with a known primary returns that primary and `responderRole==STANDBY`; with no known primary returns empty `primaryName` (2.2, 2.3).
- `fenced` reflects `in_catch_up`/`recovery_unrecoverable` (6.3); the handler mutates no state (1.2).
- `configVersion`/`configChecksum` are a self-consistent loaded pair — equal across two nodes with identical loaded config, differ when content differs; `committedConfigVersion` is exposed separately (5.1, 5.2, 5.3).

### Integration
- Dispatch-gate: a `ctrlstatus` request is answered (not `-XCE_ERR_STANDBY`/`-XCE_ERR_NOT_READY`) on STANDBY, ELECTING, INIT, and PRIMARY-no-quorum nodes; no other opcode's gating changed (7.1, 7.2).
- libxce round-trip: `xcecli_ctrlstatus` decodes all fields; `is_write=false` rotates across configured controllers; no-reachable-controller returns a typed failure (7.3).
- Authoritative read: `xcecli_ctrlstatus_primary`, when the first answer is from a standby, follows the redirect to the primary and returns a reply with `responderRole == PRIMARY`; it never returns a standby/hint reply as the authoritative result (9.1, 9.2).

### E2E (live `/xce-deploy-ha`)
- `ctrlstatus` against a healthy 3-node cluster prints the primary, peers, version+checksum; querying a standby still reports the primary with `answered by <standby>` (2.3, 8.1).
- `wait-primary` (authoritative probe) returns immediately on a healthy cluster and names the primary; after a primary kill, blocks until the **new primary itself** answers, then names it; does NOT false-succeed on a standby's stale primary view; times out when no primary serves (9.1–9.4). Directly useful to the `ha-commit-durability` Phase 6 liveness/>60s-election investigation.

## Security Considerations

`ctrlstatus` exposes cluster topology, role, term, and config version/checksum — the same trust level as the existing `clusterconf`/state-report reads (no secrets, no config content, no job data). It is read-only and goes through the same authenticated client path as other client ops; no new authorization surface is introduced. Broadening the dispatch gate is scoped strictly to this one read opcode.