myxce/.kiro/specs/controller-status/design.md

239 lines
22 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Design Document — controller-status (`ctrlstatus`)
## Overview
**Purpose:** add a single, cheap, read-only client→controller query — `ctrlstatus`, an XCE analog of LSF `lsid` — that returns basic cluster/controller identity and health: who is the current primary, the answering node's identity and role, the configured controllers (peers), the committed configuration version and content checksum, and a few HA health/progress fields (term, commit/apply position, fenced state, cluster name).
**Users:** operators inspecting a running cluster, and test/E2E authors orchestrating failover (including `wait-primary`, which switches its probe to this query).
**Impact:** today there is no status query — callers infer "is there a serving primary" indirectly (e.g. by listing jobs). This adds a new opcode by cloning the existing **opcode-only read** pattern of `clusterconf`, reads only already-maintained HA/config state (no behavior change), and crucially makes the query answerable on **any** node by adding it to the HA dispatch-gate allow-lists.
### Goals
- A new read-only opcode returning the status fields of Requirements 2–6 in one request/response (1.1, 1.4).
- Answerable on **any** node — a standby reports a hint view (the primary it knows); an **authoritative** answer comes only from the primary, reached by following the redirect / rotating the master list (2.3, 7.1–7.3).
- Report the loaded config version + content checksum (and the committed version) so same-version divergence is detectable (5.2, 5.3).
- A `ctrlstatus` test-client command (8.x) and a `wait-primary` probe switched to an authoritative (primary-only) status query (9.x).
### Non-Goals
- No operator-facing CLI/REST `lsid` surface (Go tier) — later pass.
- No change to HA election, replication, or commit behavior — the query only **reads** existing state.
- No new configuration checksum computation — the committed-config SHA-256 already exists; this only exposes it.
## Boundary Commitments
### This Spec Owns
- A new Cap'n Proto opcode `ctrlstatus` and its reply payload (`CtrlStatus`, `CtrlPeer`).
- The controller handler `handle_ctrlstatus` that snapshots local HA/config state into the reply.
- The **dispatch-gate allow-list entries** that let `ctrlstatus` answer in every controller role (incl. no-quorum).
- The libxce client API (`xce_ctrlstatus_req` / `xce_ctrlstatus` / `xcecli_ctrlstatus` + free helper).
- The `ctrlstatus` test-client command and the `wait-primary` probe switch.
### Out of Boundary
- The Go CLI/REST `lsid` surface (later spec).
- Any change to how HA state (role, term, primary identity, committed config tag, commit/apply LSN, fence) is **produced or maintained** — this spec only reads it.
- The existing `clusterconf` opcode and its resource/queue/worker payload (left unchanged).
### Allowed Dependencies
- The Cap'n Proto schema + build-time codegen (`src/schema/controller.capnp` → generated `controller.capnp.h/.c`).
- The existing request/reply plumbing (`init_reply_by_req`, `xce_reply`, opcode-only `*_req` senders) and client failover wrapper (`XCECLI_FAILOVER_CALL`, `is_write=false`).
- Read access to existing HA state (`ha_state_t`), config (`myconf_t`: `committed_config_tag`, `config_sha256`, `peers_cfg`), and persist accessors (`persist_recovery_unrecoverable`, `persist_durable_lsn`).
### Revalidation Triggers
- Any change to the `CtrlStatus`/`CtrlPeer` reply shape or the `ctrlstatus` opcode ordinal → re-check libxce decode + the test command + any future consumer.
- Any change to the dispatch-gate role allow-lists → re-check that `ctrlstatus` still answers in every role and that no other opcode's gating widened.
- A change to the meaning of `committed_config_tag` / `config_sha256` → re-check the version/checksum fields.
## Architecture
### Existing Architecture Analysis
- **Opcode-only read precedent:** `clusterconf` is sent with no request body (`Request_body_nomsg`), dispatched in `cluster.c` `switch(request.opcode)`, handled by `handle_clusterconf` (`reply.c:10`), and returned via `xce_reply`. The client side is `xce_clusterconf` → `xcecli_clusterconf` (`XCECLI_FAILOVER_CALL`, `is_write=false`). **`ctrlstatus` clones this pattern.**
- **No-alloc state-report precedent:** `ha_nop_reply_state_report` (`ha.c:1784`) reads HA atomics into a stack struct and points the reply union at it without heap allocation. `handle_ctrlstatus` follows this model (a small bounded peer list is the only allocation).
- **Dispatch gate (the critical seam):** `ha_dispatch_gate` (`ha.c:2059`) is an allow-list keyed by role. The `CTRL_STANDBY` branch defaults to `-XCE_ERR_STANDBY`; `CTRL_INIT`/`CTRL_ELECTING` default to `-XCE_ERR_NOT_READY`; the `CTRL_PRIMARY` no-quorum branch allow-lists only specific reads. `clusterconf` is **not** in any standby allow-list — it is implicitly primary-only-when-serving. A status query must **not** inherit that; it must be explicitly allow-listed in every role branch.
- **Config content checksum already exists:** `config_version_tag` = first 8 bytes of SHA-256 over the committed config blob; full digest in `config_sha256[32]`, round-trip-verified on load (`config_repl.c`).
### Pattern & dependency direction
Extend-in-place along the existing request path. Dependency direction (unchanged): `schema → libxce (request/reply/client) → controller handler`; the test client depends on libxce. No upward dependencies introduced.
```mermaid
graph LR
TestClient[xce_test ctrlstatus and wait_primary] --> Libxce[libxce xcecli_ctrlstatus]
Libxce --> Failover[XCECLI_FAILOVER_CALL is_write false]
Failover --> Gate[ha_dispatch_gate allow_list all roles]
Gate --> Handler[handle_ctrlstatus snapshots local state]
Handler --> HAState[ha_state_t and myconf_t]
Handler --> Reply[xce_reply CtrlStatus]
```
## File Structure Plan
### Modified Files
- `src/schema/controller.capnp` — add `OPCode ctrlstatus @40`; new `struct CtrlStatus $C.mapname("xce_ctrlstatus_t")` and `struct CtrlPeer $C.mapname("xce_ctrlpeer_t")`; reply-union member `ctrlstatus @17: CtrlStatus`. (Build regenerates `controller.capnp.h/.c` → `xce_ctrlstatus_t`, `RequestReply_data_ctrlstatus`, `free_CtrlStatus`.)
- `src/controller/reply.c` — add `handle_ctrlstatus` (snapshot local HA/config state into a `CtrlStatus` reply; small peer list; `xce_reply`). Mirrors `handle_clusterconf`/`ha_nop_reply_state_report`.
- `src/controller/cluster.c` — add `case OPCode_ctrlstatus: rc = handle_ctrlstatus(...)` to the opcode switch.
- `src/controller/ha.c` — add `OPCode_ctrlstatus` to the `ha_dispatch_gate` allow-lists for `CTRL_PRIMARY` (incl. the no-quorum branch), `CTRL_STANDBY`, `CTRL_ELECTING`, `CTRL_INIT`. **(the requirements-critical change)**
- `src/lib/request.c` + `src/include/request.h` — add `xce_ctrlstatus_req` (opcode-only sender, clone of `xce_clusterconf_req`).
- `src/lib/xceapi.c` — add `xce_ctrlstatus` (send req, `cbuf_get` status, `MOVE_PTR` reply union).
- `src/lib/client.c` + `src/include/client.h` — add `xcecli_ctrlstatus` (`XCECLI_FAILOVER_CALL`, `is_write=false`), the authoritative `xcecli_ctrlstatus_primary` (follows redirect / rotates the master list until `responderRole == PRIMARY`), and the `xcecli_free_ctrlstatus` macro.
- `src/controller/testsuite/test.c` — add `cmd_ctrlstatus` + table registration (uses `xcecli_ctrlstatus`, shows the responder's view); switch `cmd_wait_primary`'s probe (currently `joblist_query(...)` at the deadline-loop call site) to `xcecli_ctrlstatus_primary` (authoritative, primary-only success).
### New Files
- None (capnp C types are generated; all handlers/APIs slot into existing TUs).
## System Flows
### Status query: any-node hint then authoritative-from-primary
```mermaid
sequenceDiagram
participant C as Client
participant S as Standby node
participant P as Primary node
C->>S: ctrlstatus request
S-->>C: reply responderRole STANDBY plus known primary hint
Note over C,S: Standby answer is a non authoritative hint view
C->>P: re-query targeting the reported primary
P-->>C: authoritative reply responderRole PRIMARY
```
The dispatch gate allow-lists `ctrlstatus` in every role (incl. no-quorum), so any node answers with its local view (satisfying any-node visibility). An **authoritative** answer comes only from the node whose `responderRole == PRIMARY`. When the first reply is from a non-primary, the client follows the redirect — it maps the reply's `primaryName` to an address via the reply's `peers` list (or rotates the configured master list) and re-queries — until it obtains the primary's authoritative reply or exhausts its budget. The plain `ctrlstatus` command may display the first (hint) reply with its "answered by" framing; `wait-primary` and any authoritative use require the primary's reply.
Gate decision: `ctrlstatus` is allow-listed in every role branch and does not require quorum or a committed read — it returns the node's local view even when fenced or without quorum (precisely when an operator most wants it).
### wait-primary probe (Req 9)
`wait-primary` keeps its existing monotonic-deadline poll loop and timeout/exit semantics; only the probe changes from `joblist_query` to `xcecli_ctrlstatus`. Success predicate: the status reply reports a **non-`"unknown"`** current primary. On success it prints the served primary's identity. (Rationale + caveat below.)
## Requirements Traceability
| Requirement | Summary | Components | Interfaces |
|-------------|---------|------------|------------|
| 1.1, 1.2, 1.4 | Single-call read-only focused status | `handle_ctrlstatus`, `CtrlStatus` schema | opcode `ctrlstatus`, reply `CtrlStatus` |
| 1.3 | "unknown" for undeterminable fields | `handle_ctrlstatus` | empty Text/sentinel mapping |
| 2.1, 2.2, 2.3 | Current primary identity (unknown when unelected; standby's known primary) | `handle_ctrlstatus` (reads `ha->primary_host` under `ha->lock`) | `CtrlStatus.primaryName` |
| 3.1, 3.2, 3.3 | Answered-by node + role, distinct from primary | `handle_ctrlstatus` | `CtrlStatus.responderName`, `responderRole` |
| 4.1, 4.2 | Configured controllers + count | `handle_ctrlstatus` (reads `peers_cfg`) | `CtrlStatus.peers`, `peerCount` |
| 5.1, 5.2, 5.3 | Config version + content checksum (loaded pair + committed) | `handle_ctrlstatus` (reads loaded `config_version_tag`/`config_sha256` + `committed_config_tag`) | `CtrlStatus.configVersion`, `configChecksum`, `committedConfigVersion` |
| 6.1–6.4 | Term, commit/apply, fenced, cluster name | `handle_ctrlstatus` | `CtrlStatus.term`, `committedLsn`, `appliedLsn`, `fenced`, `clusterName` |
| 7.1, 7.2, 7.3 | Failover-aware any-node delivery | `xcecli_ctrlstatus`, dispatch-gate allow-lists | `XCECLI_FAILOVER_CALL` (is_write=false), gate entries |
| 8.1, 8.2, 8.3 | Test-client `ctrlstatus` command | `cmd_ctrlstatus` | client command + exit code |
| 9.1–9.4 | wait-primary uses an authoritative (primary-only) status probe | `cmd_wait_primary` probe swap, `xcecli_ctrlstatus_primary` | redirect-to-primary, `responderRole == PRIMARY` success |
## Components and Interfaces
| Component | Layer | Intent | Req Coverage | Key Dependencies | Contracts |
|-----------|-------|--------|--------------|------------------|-----------|
| `CtrlStatus`/`CtrlPeer` schema | Schema | Reply payload for the status query | 2–6 | capnp codegen (P0) | API (reply) |
| `handle_ctrlstatus` | Controller | Snapshot local HA/config state into the reply | 1–6 | `ha_state_t`, `myconf_t`, persist accessors (P0) | Service |
| dispatch-gate allow-lists | Controller HA | Permit `ctrlstatus` in every role | 7 | `ha_dispatch_gate` (P0) | Service |
| `xcecli_ctrlstatus` / `xcecli_ctrlstatus_primary` | libxce | Failover-aware read (any-node) + authoritative reach-primary read + free | 1, 7, 9 | failover wrapper, redirect hint (P0) | Service |
| `cmd_ctrlstatus` | Test client | Print the status (responder view) | 8 | `xcecli_ctrlstatus` (P0) | State |
| `cmd_wait_primary` probe | Test client | Authoritative serving-primary probe | 9 | `xcecli_ctrlstatus_primary` (P0) | State |
### Schema — reply payload (Cap'n Proto)
```capnp
struct CtrlPeer $C.mapname("xce_ctrlpeer_t") {
name @0: Text;
host @1: Text;
port @2: UInt32;
}
struct CtrlStatus $C.mapname("xce_ctrlstatus_t") {
clusterName @0: Text;
primaryName @1: Text; # empty ⇒ "unknown" (no elected/known primary)
responderName @2: Text; # the node that answered
responderRole @3: UInt8; # ctrl_role_t: INIT/ELECTING/PRIMARY/STANDBY
term @4: UInt64;
# Loaded config (consistent pair: version is the 8-byte prefix of the checksum)
configVersion @5: UInt64; # loaded config_version_tag (8-byte SHA-256 prefix)
configChecksum @6: Data; # loaded config_sha256 (full 32 bytes)
# Committed config version (may differ from loaded mid config-change)
committedConfigVersion @7: UInt64; # committed_config_tag
committedLsn @8: UInt64;
appliedLsn @9: UInt64;
fenced @10: Bool; # in_catch_up OR recovery_unrecoverable
peers @11: List(CtrlPeer) $C.mapname("peers.list") $C.maplistcount("peers.count");
peerCount @12: UInt32;
}
```
- **Config fields (Req 5):** `configVersion`/`configChecksum` are the **loaded** config's tag and full SHA-256 — a self-consistent pair (the version is the first 8 bytes of the checksum), so two nodes with identical loaded content match on both and divergent content differs on both (5.3). `committedConfigVersion` is the separately-tracked committed tag, exposing "what I run" vs "what is committed"; in steady state all three agree.
- New OPCode ordinal: `ctrlstatus @40` (next free after `preVoteReply @39`). New reply-union member: `ctrlstatus @17` (next free after `walackreplymsg @16`). Field ordinals are append-only — never reuse/reorder.
#### `handle_ctrlstatus` — Service contract
| Field | Detail |
|-------|--------|
| Intent | Build a `CtrlStatus` from the responding node's local HA/config state and reply |
| Requirements | 1.1–1.4, 2.1–2.3, 3.1–3.3, 4.1–4.2, 5.1–5.3, 6.1–6.4 |
**Responsibilities & Constraints**
- Read-only: must not modify any HA, persist, or config state (1.2).
- Field sourcing: `clusterName`/`responderName` from `c->myconf`/own peerinfo; `responderRole`, `term`, `appliedLsn`, `committedLsn`, `committedConfigVersion` (`committed_config_tag`) via `atomic_load_explicit(..., acquire)`; `configVersion` from the loaded `config_version_tag` and `configChecksum` from the loaded `config_sha256[32]` (a self-consistent pair — Req 5.3); `primaryName`/peers as below; `fenced = in_catch_up || persist_recovery_unrecoverable(...)`.
- **Locking:** `primary_host`/`primary_port` are guarded by `ha->lock` — take it for a consistent snapshot, copy out, release. All other fields are atomics or immutable config.
- **"unknown" mapping (1.3, 2.2, 2.3):** if `primary_host[0] == '\0'` (no known primary), emit an empty `primaryName`, which the client renders as `"unknown"`.
- **Peers (4.x):** iterate `c->myconf->peers_cfg.candidates[0..count-1]` → one `CtrlPeer{name,host,port}` each; set `peerCount`.
- Answerable in every role; returns the local view even when fenced/no-quorum.
**Dependencies**
- Inbound: `cluster.c` opcode switch — routes the request (P0).
- Outbound: `ha_state_t`, `myconf_t`, persist accessors — read-only snapshot (P0).
- Outbound: `init_reply_by_req`, `xce_reply` — reply encode/send (P0).
#### Dispatch-gate allow-lists — Service contract
- Add `OPCode_ctrlstatus` to the allow-list in `ha_dispatch_gate` for `CTRL_PRIMARY` (including the no-quorum branch), `CTRL_STANDBY`, `CTRL_ELECTING`, and `CTRL_INIT` (7.1, 7.2).
- Constraint: the change is scoped strictly to `ctrlstatus`; no other opcode's gating may widen. `ctrlstatus` never requires quorum or a committed read.
#### libxce client API — Service contract
```c
/* opcode-only request sender (clone of xce_clusterconf_req) */
int xce_ctrlstatus_req(endpoint_t* ep);
/* send + await reply; on success MOVE_PTR the reply union out */
int xce_ctrlstatus(xce_ctx_t* ctx, xce_ctrlstatus_t** status);
/* failover-aware read (is_write=false): returns whatever node answers (may be a
* standby hint view). Use for the operator command. */
int xcecli_ctrlstatus(xcecli_ctx_t c, xce_ctrlstatus_t** status);
/* AUTHORITATIVE read: returns only a reply whose responderRole == PRIMARY. If the
* first answer is from a non-primary, follow the redirect — target the reported
* primaryName via the reply's peers list, else rotate the configured master list
* — until the primary answers or the failover budget is exhausted. */
int xcecli_ctrlstatus_primary(xcecli_ctx_t c, xce_ctrlstatus_t** status);
#define xcecli_free_ctrlstatus(s) xce_free_ctrlstatus(s)
```
- Preconditions: valid client context.
- Postconditions: on success `*status` owns a decoded `CtrlStatus` (caller frees); on failure returns a typed negative (7.3) and `*status` untouched. For `xcecli_ctrlstatus_primary`, success additionally guarantees `responderRole == PRIMARY` (an authoritative answer).
- Invariants: `is_write=false` — retryable/rotatable across the configured controllers.
#### `cmd_ctrlstatus` (test client) — State contract
- Calls `xcecli_ctrlstatus`; on success prints the fields in human-readable form (8.1) — `Primary`, `answered by <responderName> (<role>)` distinct from `Primary:` (3.3), peers + count, `configVersion` and `committedConfigVersion`, `configChecksum` as hex, term, commit/apply, fenced, cluster name; renders empty `primaryName` as `unknown`. Returns 0 on success, `<0` on failure with the error printed (8.2, 8.3). Frees the reply.
#### `cmd_wait_primary` probe swap — State contract
- Replace the single `joblist_query(...)` probe call in the deadline loop with `xcecli_ctrlstatus_primary` (the authoritative variant).
- **Success predicate (9.1, 9.2):** the probe obtains an authoritative reply with `responderRole == PRIMARY` (the answer came from the serving primary). On success, complete and print that primary's identity. A standby-only/hint answer is NOT success — the probe follows the redirect to the primary (or keeps polling) until the primary answers or the deadline elapses.
- **Decision (resolves review Issue 1):** require the answer to come **from the primary**, not merely "a primary name was reported by some node." This avoids a false success on a standby's stale `primary_host` after a primary death (the false-positive class removed from the durability soak) and faithfully meets Req 9.1's "**serving** primary." The any-node hint view remains available for the operator `ctrlstatus` command via `xcecli_ctrlstatus`.
- **Preserved (9.4):** the timeout-argument parsing, monotonic deadline, poll interval, log-level quieting, and success/failure exit codes are unchanged; only the probe call and the success message change.
## Data Models
The only new data is the reply payload above. `CtrlStatus` is a flat value object (no persistence, no aggregate); `CtrlPeer` is a bounded value list (≤ configured controller count, typically 3). `configChecksum` is the loaded config's 32-byte SHA-256 and `configVersion` is its own 8-byte prefix (a self-consistent pair); `committedConfigVersion` is the separately-tracked committed tag. Two nodes with identical loaded config content produce identical `configVersion` + `configChecksum`; differing content differs in both (5.3). A node mid config-change may show `configVersion != committedConfigVersion`.
## Error Handling
- **No reachable controller (7.3):** `xcecli_ctrlstatus` returns the failover wrapper's typed negative (e.g. no-master/timeout); `cmd_ctrlstatus` prints it and returns non-zero (8.3); `wait-primary` keeps polling until its deadline (9.3).
- **Undeterminable field (1.3):** emit the defined "unknown"/empty value, never uninitialized data.
- **Standby with no known primary (2.3):** `primaryName` empty → "unknown"; the call still succeeds (the status itself is valid).
- **Read-only guarantee (1.2):** the handler takes only read locks/atomic loads; no state mutation on any path.
## Testing Strategy
### Unit / handler
- `handle_ctrlstatus` on a PRIMARY returns `primaryName==self`, `responderRole==PRIMARY`, populated peers/version/checksum (2.1, 3.x, 4.x, 5.x).
- `handle_ctrlstatus` on a STANDBY with a known primary returns that primary and `responderRole==STANDBY`; with no known primary returns empty `primaryName` (2.2, 2.3).
- `fenced` reflects `in_catch_up`/`recovery_unrecoverable` (6.3); the handler mutates no state (1.2).
- `configVersion`/`configChecksum` are a self-consistent loaded pair — equal across two nodes with identical loaded config, differ when content differs; `committedConfigVersion` is exposed separately (5.1, 5.2, 5.3).
### Integration
- Dispatch-gate: a `ctrlstatus` request is answered (not `-XCE_ERR_STANDBY`/`-XCE_ERR_NOT_READY`) on STANDBY, ELECTING, INIT, and PRIMARY-no-quorum nodes; no other opcode's gating changed (7.1, 7.2).
- libxce round-trip: `xcecli_ctrlstatus` decodes all fields; `is_write=false` rotates across configured controllers; no-reachable-controller returns a typed failure (7.3).
- Authoritative read: `xcecli_ctrlstatus_primary`, when the first answer is from a standby, follows the redirect to the primary and returns a reply with `responderRole == PRIMARY`; it never returns a standby/hint reply as the authoritative result (9.1, 9.2).
### E2E (live `/xce-deploy-ha`)
- `ctrlstatus` against a healthy 3-node cluster prints the primary, peers, version+checksum; querying a standby still reports the primary with `answered by <standby>` (2.3, 8.1).
- `wait-primary` (authoritative probe) returns immediately on a healthy cluster and names the primary; after a primary kill, blocks until the **new primary itself** answers, then names it; does NOT false-succeed on a standby's stale primary view; times out when no primary serves (9.1–9.4). Directly useful to the `ha-commit-durability` Phase 6 liveness/>60s-election investigation.
## Security Considerations
`ctrlstatus` exposes cluster topology, role, term, and config version/checksum — the same trust level as the existing `clusterconf`/state-report reads (no secrets, no config content, no job data). It is read-only and goes through the same authenticated client path as other client ops; no new authorization surface is introduced. Broadening the dispatch gate is scoped strictly to this one read opcode.