myxce/.kiro/specs/controller-status/research.md

# Gap Analysis — controller-status (ctrlstatus)

Analyzed the requirements against the existing codebase. The existing `clusterconf` opcode is a clean **opcode-only read** to model the plumbing on, and **all the status data already exists in-process**. The single requirements-critical new piece is the HA **dispatch-gate allow-list** (so the op answers on a standby/electing/no-quorum node).

## Summary

- **Recommended approach: a NEW dedicated lean opcode** (`ctrlstatus`), NOT an extension of `clusterconf`. This decisively resolves the spec's open "extend clusterconf vs new op" question: `clusterconf` is **primary-only when serving** (absent from every standby allow-list) and returns the heavy resource/queue/worker dump — extending it would still be primary-only (fails Req 7 any-node, Req 2.3 standby-reports-primary) and would carry the inventory (fails Req 1.4 focused view).
- **Config CRC: already exists, reuse.** `config_version_tag` = first 8 bytes of SHA-256 over the committed config blob (`myconf.h:31`, `config_repl.c:120-123`); full digest cached in `config_sha256[32]` (`myconf.h:32`) and round-trip-verified on load (`config_repl.c:652`). Requirement 5's version + checksum are both directly readable — no new computation.
- **All other status data exists**: role/term/applied_lsn/committed_config_tag are `_Atomic` (lock-free reads); `primary_host`/`primary_port` need `ha->lock`; peers via `c->myconf->peers_cfg.candidates[]`; fenced via `in_catch_up` + `persist_recovery_unrecoverable()`.
- **The one real gap**: `ha_dispatch_gate` (`ha.c:2059`) is an allow-list that defaults to `-XCE_ERR_STANDBY` (standby), `-XCE_ERR_NOT_READY` (init/electing), and gates the primary no-quorum branch. `OPCode_ctrlstatus` must be explicitly allow-listed in STANDBY + INIT + ELECTING + PRIMARY-no-quorum so a status read always answers — including when fenced/no-quorum, which is exactly when an operator queries it.
- **Generated codegen helps**: capnp structs are generated at build (`src/lib/meson.build`), so adding the schema struct + rebuild auto-produces `xce_ctrlstatus_t`, `RequestReply_data_ctrlstatus`, and `free_CtrlStatus`.

## Requirement → Asset Map (gaps tagged)

| Req | Need | Existing asset | Gap |
|-----|------|----------------|-----|
| 1 single-call read | opcode-only read + lean reply | `clusterconf` opcode-only pattern (`request.c:1734` `nomsg`); no-alloc reply model `ha_nop_reply_state_report` (`ha.c:1784`) | **Missing**: new `CtrlStatus` struct + `handle_ctrlstatus` (clone the no-alloc model, not the heap-xlist clusterconf one) |
| 2 primary identity (`unknown`) | current primary name | `ha->primary_host`/`primary_port` (`ha.h:149`) | **Constraint**: read under `ha->lock`; map empty→`"unknown"` |
| 3 answered-by node + role | own name + role | `c->myconf->name` (`myconf.h:29`); `ha->role` atomic (`ha.h:146`) | Reuse (lock-free) |
| 4 peers + count | configured controllers | `c->myconf->peers_cfg.candidates[]` + `candidates_count` (`cluster.h:32-38`) | Reuse, **no gap** |
| 5 config version + CRC | version + content checksum | `config_version_tag` (8-byte SHA-256 prefix) + `config_sha256[32]` (`myconf.h:31-32`); `committed_config_tag` (`ha.h:173`) | Reuse, **no gap** — expose, don't compute |
| 6 term/commit/apply/fenced/name | HA health | `ha->term`, `ha->applied_lsn` atomics; `persist.committed_lsn` atomic (no accessor — read directly); `in_catch_up` + `persist_recovery_unrecoverable()` (`persist.h:432`); cluster name | Reuse |
| 7 failover-aware any-node | retryable read on any node | `XCECLI_FAILOVER_CALL(is_write=false)` (`client.c:315`) | **Missing (CRITICAL)**: dispatch-gate allow-list entries for STANDBY/INIT/ELECTING/PRIMARY-no-quorum (`ha.c:2059`) |
| 8 test-client command | `ctrlstatus` print command | `cmd_*` table + display patterns in `test.c` | **Missing**: `cmd_ctrlstatus` (small) |
| 9 wait-primary uses it | switch the probe | `cmd_wait_primary` probe at `test.c:1161` | **Missing**: one-line probe swap + the new client fn |

## Implementation Approach Options

### Option A — Extend `clusterconf` with HA-status fields
Add primary/role/term/config-version fields to `ClusterConf` + `handle_clusterconf`.
- ✅ Fewest new symbols.
- ❌ **Fails Req 7 / 2.3**: `clusterconf` is not standby-allow-listed → primary-only when serving; a status op must answer on any node. ❌ **Fails Req 1.4**: still returns the resource/queue/worker inventory. ❌ Bloats a heap-xlist handler.
- **Rejected.**

### Option B — New dedicated lean opcode `ctrlstatus` (RECOMMENDED)
New `OPCode_ctrlstatus @40`, lean `struct CtrlStatus`, `handle_ctrlstatus` on the no-alloc `state_report` model, **dispatch-gate allow-listed on all roles**, libxce `xce_ctrlstatus`/`xcecli_ctrlstatus` (is_write=false), `cmd_ctrlstatus`, wait-primary probe swap.
- ✅ Lean focused view (Req 1.4), any-node answerable (Req 7/2.3), no heap alloc, clean separation from `clusterconf`.
- ✅ Clones the proven opcode-only read plumbing; capnp regen auto-generates the C types.
- ❌ Touches ~9 files across schema/controller/libxce/client + the HA dispatch gate.
- **Recommended.**

### Option C — Hybrid (reuse plumbing pattern, new op)
This is Option B in practice: reuse the `clusterconf` *plumbing pattern* (request/reply/failover/free) and the `state_report` *handler pattern*, but a brand-new opcode + struct + dispatch-gate work. No separate value beyond B.

## Effort & Risk

- **Effort: M (3–7 days).** All data exists and the clone pattern is proven (much is mechanical: schema + regen + req/api/cli/free + handler + command). The non-mechanical parts are the **dispatch-gate allow-listing** (HA-critical path, needs care + tests) and per-layer tests (capnp round-trip, handler-on-standby, client failover, wait-primary swap).
- **Risk: Low–Medium.** Low: no new infrastructure, content CRC already exists, read-only/no HA-behavior change. Medium: the dispatch gate is on the request-handling critical path — a wrong allow-list entry could let a standby serve something it shouldn't or fail to answer; must be scoped strictly to this read-only status op and covered by tests proving it answers on standby/electing/no-quorum **without** widening any other opcode's gating.

## Research Items to Carry into Design

1. **Reply field set & types**: expose `committed_config_tag` (8-byte version) as the version AND the full `config_sha256[32]` (hex) as the checksum? The tag is already a content-hash prefix — design should decide whether the full 32-byte digest is also returned for stronger same-version divergence detection (Req 5.3).
2. **Dispatch-gate semantics (key decision)**: a status read should answer **regardless of quorum/serving/fence** (that's when you most want it). Confirm `ctrlstatus` bypasses the quorum/serving gate in all role branches, and define precisely what each role returns (e.g. an ELECTING node returns role=electing, primary=unknown).
3. **Handler location**: `reply.c` (next to `handle_clusterconf`) vs `ha.c` (next to `ha_nop_reply_state_report`, the no-alloc model it should follow). Lean toward the `state_report` model.
4. **`"unknown"` representation** in the capnp reply (empty Text vs explicit sentinel) and how the client/test command renders it.
5. **wait-primary "serving primary present" definition** from the status reply: primary identity != unknown — and/or the responding node reports role=primary? Define the exact success predicate so the probe is unambiguous, preserving Req 9.4 (timeout/exit semantics unchanged).
6. **`committed_lsn` accessor**: none exists (`persist.committed_lsn` is read directly elsewhere). Design may add a thin `persist_committed_lsn()` for the handler, or read the atomic directly — minor.

## Files in the clone map (for design/tasks)

- `src/schema/controller.capnp` — `OPCode ctrlstatus @40`; `struct CtrlStatus $C.mapname("xce_ctrlstatus_t")`; reply member `ctrlstatus @17`. (regenerates `controller.capnp.h/.c`)
- `src/controller/cluster.c:335` — `case OPCode_ctrlstatus → handle_ctrlstatus`.
- `handle_ctrlstatus` — new (model: `ha_nop_reply_state_report`, `ha.c:1784`; reply send via `xce_reply`, `init_reply_by_req`).
- `src/controller/ha.c:2059` — **add `OPCode_ctrlstatus` to STANDBY + INIT + ELECTING + PRIMARY-no-quorum allow-lists (the critical gap).**
- `src/lib/request.c` — `xce_ctrlstatus_req` (opcode-only, clone `request.c:1734`); `src/include/request.h` decl.
- `src/lib/xceapi.c:400` — `xce_ctrlstatus` (`MOVE_PTR` reply union).
- `src/lib/client.c:315` — `xcecli_ctrlstatus` (`XCECLI_FAILOVER_CALL`, is_write=false); `src/include/client.h:55,90` decl + free macro.
- `src/controller/testsuite/test.c` — `cmd_ctrlstatus` command + registration; swap `cmd_wait_primary` probe at `test.c:1161`.

---

## Design Synthesis (decisions resolved for design.md)

The 6 carried research items are resolved as:
1. **Reply field set**: expose BOTH `configVersion` (the 8-byte `committed_config_tag`) AND `configChecksum` (the full 32-byte `config_sha256`) — version for quick compare, full digest for strong same-version divergence detection (Req 5.3).
2. **Dispatch-gate semantics**: `ctrlstatus` is allow-listed in ALL role branches (PRIMARY incl. no-quorum, STANDBY, ELECTING, INIT) and bypasses the quorum/serving/fence gate — a status read must answer precisely when the cluster is degraded. Each role returns its local view (PRIMARY→self; STANDBY→its known primary or unknown; ELECTING/INIT→unknown).
3. **Handler placement & shape**: `handle_ctrlstatus` in `reply.c` next to `handle_clusterconf`, following the no-alloc `ha_nop_reply_state_report` model; only the small bounded peer list allocates. `primary_host`/`port` read under `ha->lock`; everything else atomic/immutable.
4. **"unknown" representation**: empty `Text primaryName` on the wire; the client renders it as `"unknown"` (Req 1.3/2.2/2.3).
5. **wait-primary predicate**: success = reply reports a non-"unknown" primary (NOT "responder==PRIMARY", which could starve if a standby answers first under failover rotation). Accepted caveat: a brief standby-stale-primary window; still strictly more informative than the old joblist probe. Preserves all existing wait-primary timeout/exit semantics (Req 9.4).
6. **`committed_lsn` access**: read the persist atomic directly (no accessor exists); optionally add a thin `persist_committed_lsn()` — minor, implementer's choice.

**Build-vs-adopt**: ADOPT the `clusterconf` opcode-only plumbing pattern + the `state_report` no-alloc handler model; BUILD only the new schema structs, handler body, dispatch-gate entries, and the thin client/test wiring. No new infrastructure; config checksum reused (not built).

---

## Design-review updates (2026-06-19, /kiro-validate-design)

Two review issues resolved with the user; design.md updated:

1. **wait-primary predicate → AUTHORITATIVE from the primary (user decision).** A standby's reply is a non-authoritative HINT (it may name a just-failed primary from stale `primary_host`). The op still answers on any node (Req 2.3/7), but for an authoritative answer the client follows the redirect — maps the reply's `primaryName` to an address via the reply's `peers` list, else rotates the configured master list — until a reply with `responderRole == PRIMARY`. New `xcecli_ctrlstatus_primary` provides this; `wait-primary` uses it (success = the primary itself answered), eliminating the stale-standby false-success (the same false-positive class removed from the durability soak). The plain `xcecli_ctrlstatus` (any-node hint view) backs the operator `ctrlstatus` command.
2. **Config fields → loaded pair + committed tag (user decision).** Report `configVersion`+`configChecksum` from the LOADED config (self-consistent: version = first 8 bytes of the checksum) AND `committedConfigVersion` (`committed_config_tag`) separately — "what I run" vs "what is committed". Avoids the earlier mismatched committed-tag + loaded-checksum pairing.

Also fixed: mermaid render issues — removed `/`, parentheses, and quotes from diagram labels/messages (the LR architecture graph + the new redirect sequence diagram).

---

## Gap Analysis — Delta for Requirements 10 & 11 (2026-06-20)

Reopened after a live HA-deploy E2E finding: a standby answering `ctrlstatus` showed
`committedLsn: 0` / `committedConfigVersion: 0` while `appliedLsn: 38` (an apparent
applied-greater-than-committed inversion), and there was no way to correlate a captured
status against the per-node logs. Scope of this delta = Req 10 (off-primary committed
fidelity) + Req 11 (responder snapshot timestamp). Reqs 1–9 are implemented and unchanged.

### Requirement-to-Asset Map

| Need (Req) | Existing asset | Gap |
|---|---|---|
| 10.1 standby reports *learned* committed LSN | `ctrlstatus_fill` reads `c->persist.committed_lsn` (`ctrlstatus.c:130`) — a **primary-only** marker, 0 on a node that never led | **Constraint**: wrong source field; the learned value `ha->last_known_committed_lsn` (`ha.h:170`) exists but is unused here |
| 10.1 standby reports *learned* committed config tag | `ctrlstatus_fill` reads `ha->committed_config_tag` (`ctrlstatus.c:109`) — also primary-only | **Constraint**: learned `ha->last_known_committed_config_tag` (`ha.h:172`) exists but is unused here |
| 10.1 freshness of the learned value | `last_known_committed_lsn` written only at `cluster_apply_ctrlhello_reply` (`cluster.c:291`, `ha_convergence.c:375`) and an election safe-commit path (`ha.c:925`) | **Constraint (key)**: refreshed only at **CtrlHello join / RDHUP-failover handshake**, NOT per heartbeat. The `OP_nop` heartbeat (`cluster.c:300`) carries no committed payload; the richer `HeartbeatMsg{committedLsn@2, committedConfigTag@3}` (`controller.capnp:761`) is **defined but never consumed** on receive → the learned value can LAG between joins |
| 10.2 genuinely-unknown → 0/unknown | existing `NULLSTR`/0 conventions (Req 1.3) | none — `last_known_committed_lsn` is 0 before first learn, which is the correct "unknown" |
| 11.1 responder wall-clock at sample time | none on `CtrlStatus` (schema `controller.capnp:496`, fields @0–@12; next free ordinal **@13**) | **Missing**: new field + a `CLOCK_REALTIME` stamp in `ctrlstatus_fill`. Note `ha_on_heartbeat_received` uses `CLOCK_MONOTONIC` — **not** reusable for log correlation |
| 11.2 same clock/format as logs | logger header `src/include/xcelog.h` produces `[YYYY-MM-DD HH:MM:SS,mmm]` | **Research Needed**: confirm the exact formatter/helper in `xcelog.h` to reuse (or match) for byte-comparable output |
| 11.3 test client prints timestamp | `cmd_ctrlstatus` in `src/controller/testsuite/test.c` (C `xce_test` client; print block ~line 1297) — the "client" is xce_test, not Go | **Missing**: one print line; libxce decode is codegen from schema (auto after regen) |

### Implementation Approach Options

**Req 10 — Option A (Extend `ctrlstatus_fill`) — RECOMMENDED.** When `responderRole != CTRL_PRIMARY`,
source `committedLsn` from `ha->last_known_committed_lsn` and `committedConfigVersion` from
`ha->last_known_committed_config_tag`; keep the primary's own `persist.committed_lsn` /
`committed_config_tag` when it IS primary (those are the authoritative live values). Read-only,
satisfies Req 10.4, mirrors the existing `last_known_*` usage in the RDHUP/freshness paths
(`ha.c:2570`, `ha.c:4126`). A simpler variant — report `max(persist.committed_lsn,
last_known_committed_lsn)` uniformly (no role branch) — also kills the 0-inversion; design picks.
*Trade-off*: ✅ tiny, in-boundary, fixes the inversion. ❌ value can lag between joins (see below).

**Req 10 — out-of-boundary (do NOT pull into this spec).** Making the off-primary committed
value *live-fresh* requires consuming `HeartbeatMsg.committedLsn` (or extending `OP_nop`) and
storing it into `last_known_committed_lsn` on **every** heartbeat — a heartbeat/replication
protocol change. That violates Req 10.4 ("shall not change how committed positions are produced,
replicated, or persisted") and belongs to **ha-heartbeat-liveness** / **ha-repl-hardening**.
controller-status should report the honest best-known value and the design must state the lag caveat.

**Req 11 — Option A (Extend) — RECOMMENDED.** Add `responderTime @13: UInt64` (epoch millis) to
`CtrlStatus`; stamp it in `ctrlstatus_fill` from `CLOCK_REALTIME`; render in `cmd_ctrlstatus`
formatted via the `xcelog.h` convention so it lines up with `ctrl-N.log`. *Design decision*:
numeric epoch-millis (client formats — locale-safe, clean) vs pre-formatted `Text` (zero client
work, mirrors the `configChecksum`-as-Text precedent). Recommend numeric epoch-millis.
*Trade-off*: ✅ one field + one stamp + one print line. ❌ adds a schema ordinal (codegen regen).

### Effort & Risk

- **Req 10**: **S** (≤1 day) — one function's field sourcing + TDD (standby-known-primary asserts learned value; pre-learn asserts 0). **Risk Low** — read-only, no HA-state change.
- **Req 11**: **S** (1–2 days) — schema field + `ctrlstatus_fill` stamp + `cmd_ctrlstatus` print + codegen regen + round-trip decode test. **Risk Low**.
- **Combined**: **S**. The one **Medium-risk temptation to refuse**: expanding into heartbeat committed-propagation — keep it out of this spec.

### Research Items to Carry to Design

1. Confirm the exact wall-clock formatter/helper in `src/include/xcelog.h`; reuse it (or document the chosen format) so Req 11's value is byte-comparable to log lines.
2. Decide Req 11 wire type: `UInt64` epoch-millis (recommended) vs `Text`.
3. Decide Req 10 sourcing: role-branch (`persist.*` on primary, `last_known_*` off-primary) vs uniform `max()`; keep LSN and config-tag symmetric.
4. State the **staleness caveat** in design (learned value refreshes at join, not per heartbeat) and explicitly route live-fresh-committed to **ha-heartbeat-liveness / ha-repl-hardening** as out-of-boundary.