myxce/.kiro/specs/controller-status/research.md
Rongsong Shen 40882c6359
docs(controller-status): add Requirement 13 delta (per-controller connection status)
Adds requirements, design, and research for showing each configured
controller's connection status (ok/catching-up/dead/unknown) in
ctrlstatus's peer list, with the same off-primary fidelity pattern
already established (primary reports the true picture; a standby
reports unknown for what it cannot itself observe).

Design review found and fixed one gap before implementation: no
defined fallback if the responder's own name doesn't match any
configured peer entry (a pre-existing misconfiguration class, not new
to this delta) -- now an explicit, logged (WARN) degradation instead
of a silently wrong status.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
2026-09-16 10:40:08 +00:00

32 KiB
Raw Blame History

Gap Analysis — controller-status (ctrlstatus)

Analyzed the requirements against the existing codebase. The existing clusterconf opcode is a clean opcode-only read to model the plumbing on, and all the status data already exists in-process. The single requirements-critical new piece is the HA dispatch-gate allow-list (so the op answers on a standby/electing/no-quorum node).

Summary

  • Recommended approach: a NEW dedicated lean opcode (ctrlstatus), NOT an extension of clusterconf. This decisively resolves the spec's open "extend clusterconf vs new op" question: clusterconf is primary-only when serving (absent from every standby allow-list) and returns the heavy resource/queue/worker dump — extending it would still be primary-only (fails Req 7 any-node, Req 2.3 standby-reports-primary) and would carry the inventory (fails Req 1.4 focused view).
  • Config CRC: already exists, reuse. config_version_tag = first 8 bytes of SHA-256 over the committed config blob (myconf.h:31, config_repl.c:120-123); full digest cached in config_sha256[32] (myconf.h:32) and round-trip-verified on load (config_repl.c:652). Requirement 5's version + checksum are both directly readable — no new computation.
  • All other status data exists: role/term/applied_lsn/committed_config_tag are _Atomic (lock-free reads); primary_host/primary_port need ha->lock; peers via c->myconf->peers_cfg.candidates[]; fenced via in_catch_up + persist_recovery_unrecoverable().
  • The one real gap: ha_dispatch_gate (ha.c:2059) is an allow-list that defaults to -XCE_ERR_STANDBY (standby), -XCE_ERR_NOT_READY (init/electing), and gates the primary no-quorum branch. OPCode_ctrlstatus must be explicitly allow-listed in STANDBY + INIT + ELECTING + PRIMARY-no-quorum so a status read always answers — including when fenced/no-quorum, which is exactly when an operator queries it.
  • Generated codegen helps: capnp structs are generated at build (src/lib/meson.build), so adding the schema struct + rebuild auto-produces xce_ctrlstatus_t, RequestReply_data_ctrlstatus, and free_CtrlStatus.

Requirement → Asset Map (gaps tagged)

Req Need Existing asset Gap
1 single-call read opcode-only read + lean reply clusterconf opcode-only pattern (request.c:1734 nomsg); no-alloc reply model ha_nop_reply_state_report (ha.c:1784) Missing: new CtrlStatus struct + handle_ctrlstatus (clone the no-alloc model, not the heap-xlist clusterconf one)
2 primary identity (unknown) current primary name ha->primary_host/primary_port (ha.h:149) Constraint: read under ha->lock; map empty→"unknown"
3 answered-by node + role own name + role c->myconf->name (myconf.h:29); ha->role atomic (ha.h:146) Reuse (lock-free)
4 peers + count configured controllers c->myconf->peers_cfg.candidates[] + candidates_count (cluster.h:32-38) Reuse, no gap
5 config version + CRC version + content checksum config_version_tag (8-byte SHA-256 prefix) + config_sha256[32] (myconf.h:31-32); committed_config_tag (ha.h:173) Reuse, no gap — expose, don't compute
6 term/commit/apply/fenced/name HA health ha->term, ha->applied_lsn atomics; persist.committed_lsn atomic (no accessor — read directly); in_catch_up + persist_recovery_unrecoverable() (persist.h:432); cluster name Reuse
7 failover-aware any-node retryable read on any node XCECLI_FAILOVER_CALL(is_write=false) (client.c:315) Missing (CRITICAL): dispatch-gate allow-list entries for STANDBY/INIT/ELECTING/PRIMARY-no-quorum (ha.c:2059)
8 test-client command ctrlstatus print command cmd_* table + display patterns in test.c Missing: cmd_ctrlstatus (small)
9 wait-primary uses it switch the probe cmd_wait_primary probe at test.c:1161 Missing: one-line probe swap + the new client fn

Implementation Approach Options

Option A — Extend clusterconf with HA-status fields

Add primary/role/term/config-version fields to ClusterConf + handle_clusterconf.

  • ✅ Fewest new symbols.
  • ❌ Fails Req 7 / 2.3: clusterconf is not standby-allow-listed → primary-only when serving; a status op must answer on any node. ❌ Fails Req 1.4: still returns the resource/queue/worker inventory. ❌ Bloats a heap-xlist handler.
  • Rejected.

New OPCode_ctrlstatus @40, lean struct CtrlStatus, handle_ctrlstatus on the no-alloc state_report model, dispatch-gate allow-listed on all roles, libxce xce_ctrlstatus/xcecli_ctrlstatus (is_write=false), cmd_ctrlstatus, wait-primary probe swap.

  • ✅ Lean focused view (Req 1.4), any-node answerable (Req 7/2.3), no heap alloc, clean separation from clusterconf.
  • ✅ Clones the proven opcode-only read plumbing; capnp regen auto-generates the C types.
  • ❌ Touches ~9 files across schema/controller/libxce/client + the HA dispatch gate.
  • Recommended.

Option C — Hybrid (reuse plumbing pattern, new op)

This is Option B in practice: reuse the clusterconf plumbing pattern (request/reply/failover/free) and the state_report handler pattern, but a brand-new opcode + struct + dispatch-gate work. No separate value beyond B.

Effort & Risk

  • Effort: M (3–7 days). All data exists and the clone pattern is proven (much is mechanical: schema + regen + req/api/cli/free + handler + command). The non-mechanical parts are the dispatch-gate allow-listing (HA-critical path, needs care + tests) and per-layer tests (capnp round-trip, handler-on-standby, client failover, wait-primary swap).
  • Risk: Low–Medium. Low: no new infrastructure, content CRC already exists, read-only/no HA-behavior change. Medium: the dispatch gate is on the request-handling critical path — a wrong allow-list entry could let a standby serve something it shouldn't or fail to answer; must be scoped strictly to this read-only status op and covered by tests proving it answers on standby/electing/no-quorum without widening any other opcode's gating.

Research Items to Carry into Design

  1. Reply field set & types: expose committed_config_tag (8-byte version) as the version AND the full config_sha256[32] (hex) as the checksum? The tag is already a content-hash prefix — design should decide whether the full 32-byte digest is also returned for stronger same-version divergence detection (Req 5.3).
  2. Dispatch-gate semantics (key decision): a status read should answer regardless of quorum/serving/fence (that's when you most want it). Confirm ctrlstatus bypasses the quorum/serving gate in all role branches, and define precisely what each role returns (e.g. an ELECTING node returns role=electing, primary=unknown).
  3. Handler location: reply.c (next to handle_clusterconf) vs ha.c (next to ha_nop_reply_state_report, the no-alloc model it should follow). Lean toward the state_report model.
  4. "unknown" representation in the capnp reply (empty Text vs explicit sentinel) and how the client/test command renders it.
  5. wait-primary "serving primary present" definition from the status reply: primary identity != unknown — and/or the responding node reports role=primary? Define the exact success predicate so the probe is unambiguous, preserving Req 9.4 (timeout/exit semantics unchanged).
  6. committed_lsn accessor: none exists (persist.committed_lsn is read directly elsewhere). Design may add a thin persist_committed_lsn() for the handler, or read the atomic directly — minor.

Files in the clone map (for design/tasks)

  • src/schema/controller.capnp — OPCode ctrlstatus @40; struct CtrlStatus $C.mapname("xce_ctrlstatus_t"); reply member ctrlstatus @17. (regenerates controller.capnp.h/.c)
  • src/controller/cluster.c:335 — case OPCode_ctrlstatus → handle_ctrlstatus.
  • handle_ctrlstatus — new (model: ha_nop_reply_state_report, ha.c:1784; reply send via xce_reply, init_reply_by_req).
  • src/controller/ha.c:2059 — add OPCode_ctrlstatus to STANDBY + INIT + ELECTING + PRIMARY-no-quorum allow-lists (the critical gap).
  • src/lib/request.c — xce_ctrlstatus_req (opcode-only, clone request.c:1734); src/include/request.h decl.
  • src/lib/xceapi.c:400 — xce_ctrlstatus (MOVE_PTR reply union).
  • src/lib/client.c:315 — xcecli_ctrlstatus (XCECLI_FAILOVER_CALL, is_write=false); src/include/client.h:55,90 decl + free macro.
  • src/controller/testsuite/test.c — cmd_ctrlstatus command + registration; swap cmd_wait_primary probe at test.c:1161.

Design Synthesis (decisions resolved for design.md)

The 6 carried research items are resolved as:

  1. Reply field set: expose BOTH configVersion (the 8-byte committed_config_tag) AND configChecksum (the full 32-byte config_sha256) — version for quick compare, full digest for strong same-version divergence detection (Req 5.3).
  2. Dispatch-gate semantics: ctrlstatus is allow-listed in ALL role branches (PRIMARY incl. no-quorum, STANDBY, ELECTING, INIT) and bypasses the quorum/serving/fence gate — a status read must answer precisely when the cluster is degraded. Each role returns its local view (PRIMARY→self; STANDBY→its known primary or unknown; ELECTING/INIT→unknown).
  3. Handler placement & shape: handle_ctrlstatus in reply.c next to handle_clusterconf, following the no-alloc ha_nop_reply_state_report model; only the small bounded peer list allocates. primary_host/port read under ha->lock; everything else atomic/immutable.
  4. "unknown" representation: empty Text primaryName on the wire; the client renders it as "unknown" (Req 1.3/2.2/2.3).
  5. wait-primary predicate: success = reply reports a non-"unknown" primary (NOT "responder==PRIMARY", which could starve if a standby answers first under failover rotation). Accepted caveat: a brief standby-stale-primary window; still strictly more informative than the old joblist probe. Preserves all existing wait-primary timeout/exit semantics (Req 9.4).
  6. committed_lsn access: read the persist atomic directly (no accessor exists); optionally add a thin persist_committed_lsn() — minor, implementer's choice.

Build-vs-adopt: ADOPT the clusterconf opcode-only plumbing pattern + the state_report no-alloc handler model; BUILD only the new schema structs, handler body, dispatch-gate entries, and the thin client/test wiring. No new infrastructure; config checksum reused (not built).


Design-review updates (2026-06-19, /kiro-validate-design)

Two review issues resolved with the user; design.md updated:

  1. wait-primary predicate → AUTHORITATIVE from the primary (user decision). A standby's reply is a non-authoritative HINT (it may name a just-failed primary from stale primary_host). The op still answers on any node (Req 2.3/7), but for an authoritative answer the client follows the redirect — maps the reply's primaryName to an address via the reply's peers list, else rotates the configured master list — until a reply with responderRole == PRIMARY. New xcecli_ctrlstatus_primary provides this; wait-primary uses it (success = the primary itself answered), eliminating the stale-standby false-success (the same false-positive class removed from the durability soak). The plain xcecli_ctrlstatus (any-node hint view) backs the operator ctrlstatus command.
  2. Config fields → loaded pair + committed tag (user decision). Report configVersion+configChecksum from the LOADED config (self-consistent: version = first 8 bytes of the checksum) AND committedConfigVersion (committed_config_tag) separately — "what I run" vs "what is committed". Avoids the earlier mismatched committed-tag + loaded-checksum pairing.

Also fixed: mermaid render issues — removed /, parentheses, and quotes from diagram labels/messages (the LR architecture graph + the new redirect sequence diagram).


Gap Analysis — Delta for Requirements 10 & 11 (2026-06-20)

Reopened after a live HA-deploy E2E finding: a standby answering ctrlstatus showed committedLsn: 0 / committedConfigVersion: 0 while appliedLsn: 38 (an apparent applied-greater-than-committed inversion), and there was no way to correlate a captured status against the per-node logs. Scope of this delta = Req 10 (off-primary committed fidelity) + Req 11 (responder snapshot timestamp). Reqs 1–9 are implemented and unchanged.

Requirement-to-Asset Map

Need (Req) Existing asset Gap
10.1 standby reports learned committed LSN ctrlstatus_fill reads c->persist.committed_lsn (ctrlstatus.c:130) — a primary-only marker, 0 on a node that never led Constraint: wrong source field; the learned value ha->last_known_committed_lsn (ha.h:170) exists but is unused here
10.1 standby reports learned committed config tag ctrlstatus_fill reads ha->committed_config_tag (ctrlstatus.c:109) — also primary-only Constraint: learned ha->last_known_committed_config_tag (ha.h:172) exists but is unused here
10.1 freshness of the learned value last_known_committed_lsn written only at cluster_apply_ctrlhello_reply (cluster.c:291, ha_convergence.c:375) and an election safe-commit path (ha.c:925) Constraint (key): refreshed only at CtrlHello join / RDHUP-failover handshake, NOT per heartbeat. The OP_nop heartbeat (cluster.c:300) carries no committed payload; the richer HeartbeatMsg{committedLsn@2, committedConfigTag@3} (controller.capnp:761) is defined but never consumed on receive → the learned value can LAG between joins
10.2 genuinely-unknown → 0/unknown existing NULLSTR/0 conventions (Req 1.3) none — last_known_committed_lsn is 0 before first learn, which is the correct "unknown"
11.1 responder wall-clock at sample time none on CtrlStatus (schema controller.capnp:496, fields @0–@12; next free ordinal @13) Missing: new field + a CLOCK_REALTIME stamp in ctrlstatus_fill. Note ha_on_heartbeat_received uses CLOCK_MONOTONIC — not reusable for log correlation
11.2 same clock/format as logs logger header src/include/xcelog.h produces [YYYY-MM-DD HH:MM:SS,mmm] Research Needed: confirm the exact formatter/helper in xcelog.h to reuse (or match) for byte-comparable output
11.3 test client prints timestamp cmd_ctrlstatus in src/controller/testsuite/test.c (C xce_test client; print block ~line 1297) — the "client" is xce_test, not Go Missing: one print line; libxce decode is codegen from schema (auto after regen)

Implementation Approach Options

Req 10 — Option A (Extend ctrlstatus_fill) — RECOMMENDED. When responderRole != CTRL_PRIMARY, source committedLsn from ha->last_known_committed_lsn and committedConfigVersion from ha->last_known_committed_config_tag; keep the primary's own persist.committed_lsn / committed_config_tag when it IS primary (those are the authoritative live values). Read-only, satisfies Req 10.4, mirrors the existing last_known_* usage in the RDHUP/freshness paths (ha.c:2570, ha.c:4126). A simpler variant — report max(persist.committed_lsn, last_known_committed_lsn) uniformly (no role branch) — also kills the 0-inversion; design picks. Trade-off: ✅ tiny, in-boundary, fixes the inversion. ❌ value can lag between joins (see below).

Req 10 — out-of-boundary (do NOT pull into this spec). Making the off-primary committed value live-fresh requires consuming HeartbeatMsg.committedLsn (or extending OP_nop) and storing it into last_known_committed_lsn on every heartbeat — a heartbeat/replication protocol change. That violates Req 10.4 ("shall not change how committed positions are produced, replicated, or persisted") and belongs to ha-heartbeat-liveness / ha-repl-hardening. controller-status should report the honest best-known value and the design must state the lag caveat.

Req 11 — Option A (Extend) — RECOMMENDED. Add responderTime @13: UInt64 (epoch millis) to CtrlStatus; stamp it in ctrlstatus_fill from CLOCK_REALTIME; render in cmd_ctrlstatus formatted via the xcelog.h convention so it lines up with ctrl-N.log. Design decision: numeric epoch-millis (client formats — locale-safe, clean) vs pre-formatted Text (zero client work, mirrors the configChecksum-as-Text precedent). Recommend numeric epoch-millis. Trade-off: ✅ one field + one stamp + one print line. ❌ adds a schema ordinal (codegen regen).

Effort & Risk

  • Req 10: S (≤1 day) — one function's field sourcing + TDD (standby-known-primary asserts learned value; pre-learn asserts 0). Risk Low — read-only, no HA-state change.
  • Req 11: S (1–2 days) — schema field + ctrlstatus_fill stamp + cmd_ctrlstatus print + codegen regen + round-trip decode test. Risk Low.
  • Combined: S. The one Medium-risk temptation to refuse: expanding into heartbeat committed-propagation — keep it out of this spec.

Research Items to Carry to Design

  1. Confirm the exact wall-clock formatter/helper in src/include/xcelog.h; reuse it (or document the chosen format) so Req 11's value is byte-comparable to log lines.
  2. Decide Req 11 wire type: UInt64 epoch-millis (recommended) vs Text.
  3. Decide Req 10 sourcing: role-branch (persist.* on primary, last_known_* off-primary) vs uniform max(); keep LSN and config-tag symmetric.
  4. State the staleness caveat in design (learned value refreshes at join, not per heartbeat) and explicitly route live-fresh-committed to ha-heartbeat-liveness / ha-repl-hardening as out-of-boundary.

Gap Analysis — Delta for Requirement 12 (2026-09-15)

Reopened by a user enhancement request: ctrlstatus should report the term in which the current primary was elected and the wall-clock time it was elected, with standby-side fidelity (learned, not "unknown") explicitly requested — mirroring Req 10's off-primary fidelity pattern rather than the simpler primary-only alternative. Scope of this delta = Req 12 only. Reqs 1–11 are implemented and unchanged.

Requirement-to-Asset Map

Need (Req) Existing asset Gap
12.1 term of the current primary's election ha->term (ha.h, already reported as CtrlStatus.term, Req 6.1) No gap — this codebase's Raft term already propagates correctly to every node via the existing heartbeat/vote term-sync path (heartbeat_msg_t.term, src/include/typemap.h:613). Reuse, don't add a field.
12.1 wall-clock time of the current primary's election none — ha_state_t has no election timestamp of any kind (unlike last_known_committed_lsn/last_known_committed_config_tag, which already exist for committed position) Missing: a new ha_state_t field, a new promotion-time stamp (both roads), a new heartbeat-carried field, and a new CtrlStatus reply field
12.2 off-primary (standby) fidelity for election time ha_observe_heartbeat_committed (ha.c:9085) is the existing precedent for learning a primary-sourced value via heartbeat, monotonic-on-LSN Missing: heartbeat_msg_t (typemap.h:611-616) carries term/committed_lsn/committed_config_tag but no election timestamp — must add electedAtMs to both the C struct and its capnp schema (HeartbeatMsg, next free ordinal after committedConfigTag@3)
12.2 primary's own promotion-time self-observation raft_cb_become_leader (ha.c:1615) already self-observes last_known_committed_lsn at its promotion instant (ha.c:1665, inside if (promo_ectx)); ha_do_promote (ha.c:2927) is the non-raft road's promotion function Gap (needs implementation-time confirmation, not just reuse): the committed-LSN self-observe point in raft_cb_become_leader sits BEFORE the function's later ha_promotion_gate_decision eligibility gate, which can still refuse promotion — the NEW election-time stamp must sit AFTER that gate, at the point the function is irrevocably committed to CTRL_PRIMARY, a different line than the existing precedent. This delta's own codegraph exploration did not reach that later line (function body was truncated by explore-tool budget); flagged as a Research Item in design.md rather than guessed.
12.3 unknown when no primary known existing primaryName-unknown derivation (Req 2.2/2.3) No gap — gate the new fields' output on this existing derivation; no new "unknown" logic needed
12.4 same clock/format as Req 11 ctrlstatus_format_time_ms (src/include/xce/ctrlstatus_fmt.h:33) already extracted as a shared, reusable formatter (confirmed via the discovery-phase CodeGraph hint — it is already used by both ctrlstatus.c and joblist_display.c) No gap — reuse verbatim, zero new formatting code
12.6 test client prints both fields cmd_ctrlstatus (src/controller/testsuite/test.c) Missing: two print lines

Implementation Approach Options

Req 12 term half — Option A (reuse ha->term, CLI-side gate) — RECOMMENDED. No new field; cmd_ctrlstatus derives the displayed electedTerm as primaryName known ? term : unknown. Trade-off: ✅ zero new state, zero schema change for this half. ❌ accepts the same narrow standby-hint transient-staleness window already accepted elsewhere in this spec (a standby that just voted for a new candidate, before its primaryName catches up) — judged acceptable, not a regression, and explicitly called out as a trade-off in design.md rather than glossed over.

Req 12 term half — Option B (a dedicated learned last_known_elected_term, symmetric with last_known_committed_lsn) — REJECTED. Would be fully symmetric with Req 10's own pattern, but Req 10 needed a learned channel specifically because the field it replaced (persist.committed_lsn) is genuinely primary-only and wrong (0) off-primary — no such pre-existing wrong-value bug exists for term, which is already correctly synced cluster-wide. Adding a parallel field here would be unjustified duplication (design-synthesis Simplification lens): same value, two storage locations, extra room for the two to drift.

Req 12 time half — Option A (new last_known_elected_at_ms, unconditional heartbeat-carried overwrite) — RECOMMENDED. Mirrors ha_observe_heartbeat_committed's existing shape (new heartbeat_msg_t field, primary-side self-observe at promotion, standby-side store on receive) but does NOT need a monotonic/value compare on write, because every write site is already gated by an existing freshness precondition (see design.md Locked Decision 2). Trade-off: ✅ small, in-boundary, symmetric with the existing learned-state pattern. ❌ two new promotion-road stamp sites to get exactly right (see the gate-ordering gap above) — the one non-trivial risk in this delta, called out explicitly as Research Items in design.md rather than assumed correct.

Req 12 time half — Option B (derive an approximate elected time from the WAL_NOOP own-term gate entry's timestamp, ha_arm_own_term_gate, ha.c:5079) — REJECTED. The own-term no-op is appended AFTER promotion completes and is not guaranteed to exist on every promotion road in every configuration (comment at ha.c:5088-5093 shows it can fail and leave the gate "UNSET, degraded-but-safe" without aborting promotion) — an unreliable proxy for "the moment of election." A direct, dedicated stamp at the actual role-transition instant is simpler and strictly correct.

Effort & Risk

  • Req 12 term half: XS (a few hours) — one cmd_ctrlstatus print-layer change, no schema/handler change. Risk: Low.
  • Req 12 time half: S (1–2 days) — one new ha_state_t field, two promotion-road stamp sites (needs the exact post-eligibility-gate line confirmed per-road, see Research Items), one heartbeat schema field + send/receive wiring, one ctrlstatus_fill read, one print block, TDD coverage across a unit test file, a heartbeat sibling test, and a promotion-road integration test. Risk: Low–Medium — the only real risk is stamping the election time at the WRONG point in raft_cb_become_leader (before its final eligibility gate, like the existing last_known_committed_lsn self-observe does), which would show a bogus election time for a promotion attempt that was actually refused; mitigated by an explicit Research Item + a dedicated "no stamp on a refused promotion" test.
  • Combined: S. No heartbeat/replication protocol semantics change (the heartbeat message only gains one more echoed field, exactly like Req 10/11's own precedent) — stays in-boundary, no need to route anything to ha-heartbeat-liveness/ha-repl-hardening this time (unlike Req 10, which did need that routing for its "live-fresh" out-of-boundary item).

Design Synthesis (decisions resolved for design.md)

  1. Generalization: electedAtMs's heartbeat-carry + promotion-self-observe shape generalizes the SAME pattern committedLsn/committedConfigTag already established — no new abstraction invented, the existing heartbeat_msg_t/HeartbeatMsg extension point is reused directly.
  2. Build vs. Adopt: ADOPT the existing heartbeat-carried learned-state pattern wholesale (no alternative transport considered — piggybacking on the already-flowing heartbeat is strictly simpler than any new channel); BUILD only the one new field + two stamp sites + one receive-side store. electedTerm needs no build at all (pure reuse).
  3. Simplification: rejected a parallel last_known_elected_term field (Option B above) and rejected a role-branch in ctrlstatus_fill for electedAtMs (unlike Req 10, one unconditional read suffices here since there is no pre-existing primary-only marker to disambiguate against) — both concrete instances of preferring the smaller design that still satisfies every acceptance criterion.

Gap Analysis — Delta for Requirement 13 (2026-09-16)

Reopened by a user enhancement request: ctrlstatus's "Controllers (N):" peer list should show each configured controller's connection status (ok/catching-up/dead), not just name and address. User-confirmed at requirements time: three-state granularity, and best-effort per-responder reporting ("unknown" for peers a responder can't observe) rather than a primary-only-authoritative requirement. Scope of this delta = Req 13 only. Reqs 1–12 are implemented and unchanged.

Requirement-to-Asset Map

Need (Req) Existing asset Gap
13.1 well-defined status value set, per peer CtrlPeer struct (controller.capnp, xce_ctrlpeer_t in typemap.h) already has name/host/port, next free ordinal @3 Missing: a status field on CtrlPeer — this is the first Req 12+ delta to extend the nested list-element struct rather than CtrlStatus itself
13.2 self always ok ctrlstatus_fill already establishes st->responderName for its own identity No gap — trivial name-match against the peer loop's own candidates[i].name
13.3 primary reports true status for every peer replicator_t.standbys[] (src/include/repl.h) already tracks each standby's live connection (ep), catch-up state (in_catch_up), host, and port — built and maintained for replication/quorum purposes, unrelated to this spec Missing: a lookup from a configured peer (peers_cfg.candidates[i], identified by name+host+port) to its corresponding replicator_t.standbys[] slot. Constraint discovered: no positional/index correlation exists between the two arrays — standby_conn_t carries host/port but no configured-peer name or stable index, so the lookup must match by host+port value, not by name or array position. repl_active_standby_count (repl.c) already implements the exact ep != NULL && !in_catch_up predicate this delta needs to reuse for the "ok" case.
13.4 non-primary reports unknown for what it can't determine ha->last_activity/liveness_poll_fn (a standby's own liveness signal about its primary) Constraint discovered (closes an open requirements-time question): this liveness signal is a bare elapsed-time scalar with no peer identity attached — there is no mechanism (analogous to ha->primary_host, itself already known to be vestigial per Req 5.1's ade0081 finding) that would let a standby attribute it to one specific configured peer. A standby can determine ONLY its own entry; every other entry, including the primary's, must report unknown. Building peer-identity-carrying liveness data would be materially larger than this delta (a new signal on the heartbeat/connection-accept path) — explicitly out of boundary, not silently absorbed here.
13.5 read-only n/a No gap — the new read is read-only by construction (a lock-scoped array scan), same shape as the existing ctrlstatus_fill reads
13.6 test client prints it cmd_ctrlstatus's existing per-peer print line (src/harness/test.c) Missing: one more %s in the existing printf, fed by a new status-string renderer

Implementation Approach Options

Req 13.3 sourcing — Option A (host+port match against replicator_t.standbys[]) — RECOMMENDED. The only asset that exists; matches the exact predicate repl_active_standby_count already uses. Trade-off: ✅ reuses proven, already-correct logic; no new state. ❌ an O(candidates × MAX_STANDBYS) scan per query — trivially cheap given both are small, fixed-size (peer counts are single-digit in practice, MAX_STANDBYS is a small compile-time bound), not a concern at this request rate.

Req 13.3 sourcing — Option B (add a stable peer-index field to standby_conn_t, matching standby_ep_ctx_t.peer_idx's existing pattern) — REJECTED (for this delta). Would remove the host/port string-compare, but requires touching repl_add_standby's call sites (production connection-establishment code, not a read-only reporting path) to thread the index through — real scope creep for a purely cosmetic optimization over Option A, which is already O(small×small) and correct. Revisit only if a future need requires this correlation elsewhere too.

Req 13.4 non-primary visibility — Option A (self-only, unknown otherwise) — RECOMMENDED (per requirements-time user confirmation, closed further by this gap analysis). The only option that doesn't require new cross-node signaling.

Req 13.4 non-primary visibility — Option B (thread the primary's peer identity through the heartbeat so a standby CAN report its own primary's status) — REJECTED (out of boundary). Genuinely useful, but a real protocol change (a new heartbeat-carried field identifying the sender, analogous in spirit to how Req 12's electedAtMs rides the heartbeat) — bigger than a "show status" delta, deserves its own requirements pass if ever pursued, not a silent scope absorption here.

Effort & Risk

  • Req 13, combined: S (1–2 days). No HA-critical path touched at all — unlike Req 12, this delta adds zero new promotion-road or heartbeat-protocol code; it is a pure additional read inside the already-read-only ctrlstatus_fill, the lowest-risk category of change this spec has made so far. Risk: Low. The one genuine unknown flagged as a Research Item (the exact cluster_t → replicator_t field path) is a one-line confirmation, not a design fork.

Design Synthesis (decisions resolved for design.md)

  1. Generalization: none needed — this is a single, self-contained addition (one new field, one new lookup, one new renderer); no broader capability is being generalized out of it.
  2. Build vs. Adopt: ADOPT replicator_t.standbys[] and repl_active_standby_count's exact ep != NULL && !in_catch_up predicate wholesale — zero new tracking state; BUILD only the lookup-by-host/port loop, the wire field, and the string renderer.
  3. Simplification: rejected Option B for both 13.3 (index-based correlation) and 13.4 (heartbeat-carried primary identity) — both would introduce real new production-path coupling to serve a read-only reporting feature; the smaller, already-available-data option satisfies every acceptance criterion the user confirmed.