myxce/.kiro/specs/controller-status/research.md

Gap Analysis — controller-status (ctrlstatus)

Analyzed the requirements against the existing codebase. The existing clusterconf opcode is a clean opcode-only read to model the plumbing on, and all the status data already exists in-process. The single requirements-critical new piece is the HA dispatch-gate allow-list (so the op answers on a standby/electing/no-quorum node).

Summary

• Recommended approach: a NEW dedicated lean opcode (ctrlstatus), NOT an extension of clusterconf. This decisively resolves the spec's open "extend clusterconf vs new op" question: clusterconf is primary-only when serving (absent from every standby allow-list) and returns the heavy resource/queue/worker dump — extending it would still be primary-only (fails Req 7 any-node, Req 2.3 standby-reports-primary) and would carry the inventory (fails Req 1.4 focused view).
• Config CRC: already exists, reuse. config_version_tag = first 8 bytes of SHA-256 over the committed config blob (myconf.h:31, config_repl.c:120-123); full digest cached in config_sha256[32] (myconf.h:32) and round-trip-verified on load (config_repl.c:652). Requirement 5's version + checksum are both directly readable — no new computation.
• All other status data exists: role/term/applied_lsn/committed_config_tag are _Atomic (lock-free reads); primary_host/primary_port need ha->lock; peers via c->myconf->peers_cfg.candidates[]; fenced via in_catch_up + persist_recovery_unrecoverable().
• The one real gap: ha_dispatch_gate (ha.c:2059) is an allow-list that defaults to -XCE_ERR_STANDBY (standby), -XCE_ERR_NOT_READY (init/electing), and gates the primary no-quorum branch. OPCode_ctrlstatus must be explicitly allow-listed in STANDBY + INIT + ELECTING + PRIMARY-no-quorum so a status read always answers — including when fenced/no-quorum, which is exactly when an operator queries it.
• Generated codegen helps: capnp structs are generated at build (src/lib/meson.build), so adding the schema struct + rebuild auto-produces xce_ctrlstatus_t, RequestReply_data_ctrlstatus, and free_CtrlStatus.

Requirement → Asset Map (gaps tagged)

Req	Need	Existing asset	Gap
1 single-call read	opcode-only read + lean reply	clusterconf opcode-only pattern (request.c:1734 nomsg); no-alloc reply model ha_nop_reply_state_report (ha.c:1784)	Missing: new CtrlStatus struct + handle_ctrlstatus (clone the no-alloc model, not the heap-xlist clusterconf one)
2 primary identity (unknown)	current primary name	ha->primary_host/primary_port (ha.h:149)	Constraint: read under ha->lock; map empty→"unknown"
3 answered-by node + role	own name + role	c->myconf->name (myconf.h:29); ha->role atomic (ha.h:146)	Reuse (lock-free)
4 peers + count	configured controllers	c->myconf->peers_cfg.candidates[] + candidates_count (cluster.h:32-38)	Reuse, no gap
5 config version + CRC	version + content checksum	config_version_tag (8-byte SHA-256 prefix) + config_sha256[32] (myconf.h:31-32); committed_config_tag (ha.h:173)	Reuse, no gap — expose, don't compute
6 term/commit/apply/fenced/name	HA health	ha->term, ha->applied_lsn atomics; persist.committed_lsn atomic (no accessor — read directly); in_catch_up + persist_recovery_unrecoverable() (persist.h:432); cluster name	Reuse
7 failover-aware any-node	retryable read on any node	XCECLI_FAILOVER_CALL(is_write=false) (client.c:315)	Missing (CRITICAL): dispatch-gate allow-list entries for STANDBY/INIT/ELECTING/PRIMARY-no-quorum (ha.c:2059)
8 test-client command	ctrlstatus print command	cmd_* table + display patterns in test.c	Missing: cmd_ctrlstatus (small)
9 wait-primary uses it	switch the probe	cmd_wait_primary probe at test.c:1161	Missing: one-line probe swap + the new client fn

Implementation Approach Options

Option A — Extend clusterconf with HA-status fields

Add primary/role/term/config-version fields to ClusterConf + handle_clusterconf.

• ✅ Fewest new symbols.
• ❌ Fails Req 7 / 2.3: clusterconf is not standby-allow-listed → primary-only when serving; a status op must answer on any node. ❌ Fails Req 1.4: still returns the resource/queue/worker inventory. ❌ Bloats a heap-xlist handler.
• Rejected.

Option B — New dedicated lean opcode ctrlstatus (RECOMMENDED)

New OPCode_ctrlstatus @40, lean struct CtrlStatus, handle_ctrlstatus on the no-alloc state_report model, dispatch-gate allow-listed on all roles, libxce xce_ctrlstatus/xcecli_ctrlstatus (is_write=false), cmd_ctrlstatus, wait-primary probe swap.

• ✅ Lean focused view (Req 1.4), any-node answerable (Req 7/2.3), no heap alloc, clean separation from clusterconf.
• ✅ Clones the proven opcode-only read plumbing; capnp regen auto-generates the C types.
• ❌ Touches ~9 files across schema/controller/libxce/client + the HA dispatch gate.
• Recommended.

Option C — Hybrid (reuse plumbing pattern, new op)

This is Option B in practice: reuse the clusterconf plumbing pattern (request/reply/failover/free) and the state_report handler pattern, but a brand-new opcode + struct + dispatch-gate work. No separate value beyond B.

Effort & Risk

• Effort: M (3–7 days). All data exists and the clone pattern is proven (much is mechanical: schema + regen + req/api/cli/free + handler + command). The non-mechanical parts are the dispatch-gate allow-listing (HA-critical path, needs care + tests) and per-layer tests (capnp round-trip, handler-on-standby, client failover, wait-primary swap).
• Risk: Low–Medium. Low: no new infrastructure, content CRC already exists, read-only/no HA-behavior change. Medium: the dispatch gate is on the request-handling critical path — a wrong allow-list entry could let a standby serve something it shouldn't or fail to answer; must be scoped strictly to this read-only status op and covered by tests proving it answers on standby/electing/no-quorum without widening any other opcode's gating.

Research Items to Carry into Design

1. Reply field set & types: expose committed_config_tag (8-byte version) as the version AND the full config_sha256[32] (hex) as the checksum? The tag is already a content-hash prefix — design should decide whether the full 32-byte digest is also returned for stronger same-version divergence detection (Req 5.3).
2. Dispatch-gate semantics (key decision): a status read should answer regardless of quorum/serving/fence (that's when you most want it). Confirm ctrlstatus bypasses the quorum/serving gate in all role branches, and define precisely what each role returns (e.g. an ELECTING node returns role=electing, primary=unknown).
3. Handler location: reply.c (next to handle_clusterconf) vs ha.c (next to ha_nop_reply_state_report, the no-alloc model it should follow). Lean toward the state_report model.
4. "unknown" representation in the capnp reply (empty Text vs explicit sentinel) and how the client/test command renders it.
5. wait-primary "serving primary present" definition from the status reply: primary identity != unknown — and/or the responding node reports role=primary? Define the exact success predicate so the probe is unambiguous, preserving Req 9.4 (timeout/exit semantics unchanged).
6. committed_lsn accessor: none exists (persist.committed_lsn is read directly elsewhere). Design may add a thin persist_committed_lsn() for the handler, or read the atomic directly — minor.

Files in the clone map (for design/tasks)

• src/schema/controller.capnp — OPCode ctrlstatus @40; struct CtrlStatus $C.mapname("xce_ctrlstatus_t"); reply member ctrlstatus @17. (regenerates controller.capnp.h/.c)
• src/controller/cluster.c:335 — case OPCode_ctrlstatus → handle_ctrlstatus.
• handle_ctrlstatus — new (model: ha_nop_reply_state_report, ha.c:1784; reply send via xce_reply, init_reply_by_req).
• src/controller/ha.c:2059 — add OPCode_ctrlstatus to STANDBY + INIT + ELECTING + PRIMARY-no-quorum allow-lists (the critical gap).
• src/lib/request.c — xce_ctrlstatus_req (opcode-only, clone request.c:1734); src/include/request.h decl.
• src/lib/xceapi.c:400 — xce_ctrlstatus (MOVE_PTR reply union).
• src/lib/client.c:315 — xcecli_ctrlstatus (XCECLI_FAILOVER_CALL, is_write=false); src/include/client.h:55,90 decl + free macro.
• src/controller/testsuite/test.c — cmd_ctrlstatus command + registration; swap cmd_wait_primary probe at test.c:1161.

---

Design Synthesis (decisions resolved for design.md)

The 6 carried research items are resolved as:

1. Reply field set: expose BOTH configVersion (the 8-byte committed_config_tag) AND configChecksum (the full 32-byte config_sha256) — version for quick compare, full digest for strong same-version divergence detection (Req 5.3).
2. Dispatch-gate semantics: ctrlstatus is allow-listed in ALL role branches (PRIMARY incl. no-quorum, STANDBY, ELECTING, INIT) and bypasses the quorum/serving/fence gate — a status read must answer precisely when the cluster is degraded. Each role returns its local view (PRIMARY→self; STANDBY→its known primary or unknown; ELECTING/INIT→unknown).
3. Handler placement & shape: handle_ctrlstatus in reply.c next to handle_clusterconf, following the no-alloc ha_nop_reply_state_report model; only the small bounded peer list allocates. primary_host/port read under ha->lock; everything else atomic/immutable.
4. "unknown" representation: empty Text primaryName on the wire; the client renders it as "unknown" (Req 1.3/2.2/2.3).
5. wait-primary predicate: success = reply reports a non-"unknown" primary (NOT "responder==PRIMARY", which could starve if a standby answers first under failover rotation). Accepted caveat: a brief standby-stale-primary window; still strictly more informative than the old joblist probe. Preserves all existing wait-primary timeout/exit semantics (Req 9.4).
6. committed_lsn access: read the persist atomic directly (no accessor exists); optionally add a thin persist_committed_lsn() — minor, implementer's choice.

Build-vs-adopt: ADOPT the clusterconf opcode-only plumbing pattern + the state_report no-alloc handler model; BUILD only the new schema structs, handler body, dispatch-gate entries, and the thin client/test wiring. No new infrastructure; config checksum reused (not built).

---

Design-review updates (2026-06-19, /kiro-validate-design)

Two review issues resolved with the user; design.md updated:

1. wait-primary predicate → AUTHORITATIVE from the primary (user decision). A standby's reply is a non-authoritative HINT (it may name a just-failed primary from stale primary_host). The op still answers on any node (Req 2.3/7), but for an authoritative answer the client follows the redirect — maps the reply's primaryName to an address via the reply's peers list, else rotates the configured master list — until a reply with responderRole == PRIMARY. New xcecli_ctrlstatus_primary provides this; wait-primary uses it (success = the primary itself answered), eliminating the stale-standby false-success (the same false-positive class removed from the durability soak). The plain xcecli_ctrlstatus (any-node hint view) backs the operator ctrlstatus command.
2. Config fields → loaded pair + committed tag (user decision). Report configVersion+configChecksum from the LOADED config (self-consistent: version = first 8 bytes of the checksum) AND committedConfigVersion (committed_config_tag) separately — "what I run" vs "what is committed". Avoids the earlier mismatched committed-tag + loaded-checksum pairing.

Also fixed: mermaid render issues — removed /, parentheses, and quotes from diagram labels/messages (the LR architecture graph + the new redirect sequence diagram).