myxce/.kiro/specs/controller-status/design.md

Design Document — controller-status (ctrlstatus)

Overview

Purpose: add a single, cheap, read-only client→controller query — ctrlstatus, an XCE analog of LSF lsid — that returns basic cluster/controller identity and health: who is the current primary, the answering node's identity and role, the configured controllers (peers), the committed configuration version and content checksum, and a few HA health/progress fields (term, commit/apply position, fenced state, cluster name).

Users: operators inspecting a running cluster, and test/E2E authors orchestrating failover (including wait-primary, which switches its probe to this query).

Impact: today there is no status query — callers infer "is there a serving primary" indirectly (e.g. by listing jobs). This adds a new opcode by cloning the existing opcode-only read pattern of clusterconf, reads only already-maintained HA/config state (no behavior change), and crucially makes the query answerable on any node by adding it to the HA dispatch-gate allow-lists.

Goals

• A new read-only opcode returning the status fields of Requirements 2–6 in one request/response (1.1, 1.4).
• Answerable on any node — a standby reports a hint view (the primary it knows); an authoritative answer comes only from the primary, reached by following the redirect / rotating the master list (2.3, 7.1–7.3).
• Report the loaded config version + content checksum (and the committed version) so same-version divergence is detectable (5.2, 5.3).
• A ctrlstatus test-client command (8.x) and a wait-primary probe switched to an authoritative (primary-only) status query (9.x).

Non-Goals

• No operator-facing CLI/REST lsid surface (Go tier) — later pass.
• No change to HA election, replication, or commit behavior — the query only reads existing state.
• No new configuration checksum computation — the committed-config SHA-256 already exists; this only exposes it.

Boundary Commitments

This Spec Owns

• A new Cap'n Proto opcode ctrlstatus and its reply payload (CtrlStatus, CtrlPeer).
• The controller handler handle_ctrlstatus that snapshots local HA/config state into the reply.
• The dispatch-gate allow-list entries that let ctrlstatus answer in every controller role (incl. no-quorum).
• The libxce client API (xce_ctrlstatus_req / xce_ctrlstatus / xcecli_ctrlstatus + free helper).
• The ctrlstatus test-client command and the wait-primary probe switch.

Out of Boundary

• The Go CLI/REST lsid surface (later spec).
• Any change to how HA state (role, term, primary identity, committed config tag, commit/apply LSN, fence) is produced or maintained — this spec only reads it.
• The existing clusterconf opcode and its resource/queue/worker payload (left unchanged).

Allowed Dependencies

• The Cap'n Proto schema + build-time codegen (src/schema/controller.capnp → generated controller.capnp.h/.c).
• The existing request/reply plumbing (init_reply_by_req, xce_reply, opcode-only *_req senders) and client failover wrapper (XCECLI_FAILOVER_CALL, is_write=false).
• Read access to existing HA state (ha_state_t), config (myconf_t: committed_config_tag, config_sha256, peers_cfg), and persist accessors (persist_recovery_unrecoverable, persist_durable_lsn).

Revalidation Triggers

• Any change to the CtrlStatus/CtrlPeer reply shape or the ctrlstatus opcode ordinal → re-check libxce decode + the test command + any future consumer.
• Any change to the dispatch-gate role allow-lists → re-check that ctrlstatus still answers in every role and that no other opcode's gating widened.
• A change to the meaning of committed_config_tag / config_sha256 → re-check the version/checksum fields.

Architecture

Existing Architecture Analysis

• Opcode-only read precedent: clusterconf is sent with no request body (Request_body_nomsg), dispatched in cluster.c switch(request.opcode), handled by handle_clusterconf (reply.c:10), and returned via xce_reply. The client side is xce_clusterconf → xcecli_clusterconf (XCECLI_FAILOVER_CALL, is_write=false). ctrlstatus clones this pattern.
• No-alloc state-report precedent: ha_nop_reply_state_report (ha.c:3802) reads HA atomics into a stack struct and points the reply union at it without heap allocation. handle_ctrlstatus follows this model (a small bounded peer list is the only allocation).
• Dispatch gate (the critical seam): ha_dispatch_gate (ha.c:4525) is an allow-list keyed by role. The CTRL_STANDBY branch defaults to -XCE_ERR_STANDBY; CTRL_INIT/CTRL_ELECTING default to -XCE_ERR_NOT_READY; the CTRL_PRIMARY no-quorum branch allow-lists only specific reads. clusterconf is not in any standby allow-list — it is implicitly primary-only-when-serving. A status query must not inherit that; it must be explicitly allow-listed in every role branch.
• Config content checksum already exists: config_version_tag = first 8 bytes of SHA-256 over the committed config blob; full digest in config_sha256[32], round-trip-verified on load (config_repl.c).

Pattern & dependency direction

Extend-in-place along the existing request path. Dependency direction (unchanged): schema → libxce (request/reply/client) → controller handler; the test client depends on libxce. No upward dependencies introduced.

graph LR
    TestClient[xce_test ctrlstatus and wait_primary] --> Libxce[libxce xcecli_ctrlstatus]
    Libxce --> Failover[XCECLI_FAILOVER_CALL is_write false]
    Failover --> Gate[ha_dispatch_gate allow_list all roles]
    Gate --> Handler[handle_ctrlstatus snapshots local state]
    Handler --> HAState[ha_state_t and myconf_t]
    Handler --> Reply[xce_reply CtrlStatus]

File Structure Plan

Modified Files

• src/schema/controller.capnp — add OPCode ctrlstatus @40; new struct CtrlStatus $C.mapname("xce_ctrlstatus_t") and struct CtrlPeer $C.mapname("xce_ctrlpeer_t"); reply-union member ctrlstatus @17: CtrlStatus. (Build regenerates controller.capnp.h/.c → xce_ctrlstatus_t, RequestReply_data_ctrlstatus, free_CtrlStatus.)
• src/controller/reply.c — add handle_ctrlstatus (snapshot local HA/config state into a CtrlStatus reply; small peer list; xce_reply). Mirrors handle_clusterconf/ha_nop_reply_state_report.
• src/controller/cluster.c — add case OPCode_ctrlstatus: rc = handle_ctrlstatus(...) to the opcode switch.
• src/controller/ha.c — add OPCode_ctrlstatus to the ha_dispatch_gate allow-lists for CTRL_PRIMARY (incl. the no-quorum branch), CTRL_STANDBY, CTRL_ELECTING, CTRL_INIT. (the requirements-critical change)
• src/lib/request.c + src/include/request.h — add xce_ctrlstatus_req (opcode-only sender, clone of xce_clusterconf_req).
• src/lib/xceapi.c — add xce_ctrlstatus (send req, cbuf_get status, MOVE_PTR reply union).
• src/lib/client.c + src/include/client.h — add xcecli_ctrlstatus (XCECLI_FAILOVER_CALL, is_write=false), the authoritative xcecli_ctrlstatus_primary (follows redirect / rotates the master list until responderRole == PRIMARY), and the xcecli_free_ctrlstatus macro.
• src/controller/testsuite/test.c — add cmd_ctrlstatus + table registration (uses xcecli_ctrlstatus, shows the responder's view); switch cmd_wait_primary's probe (currently joblist_query(...) at the deadline-loop call site) to xcecli_ctrlstatus_primary (authoritative, primary-only success).

New Files

• None (capnp C types are generated; all handlers/APIs slot into existing TUs).

System Flows

Status query: any-node hint then authoritative-from-primary

sequenceDiagram
    participant C as Client
    participant S as Standby node
    participant P as Primary node
    C->>S: ctrlstatus request
    S-->>C: reply responderRole STANDBY plus known primary hint
    Note over C,S: Standby answer is a non authoritative hint view
    C->>P: re-query targeting the reported primary
    P-->>C: authoritative reply responderRole PRIMARY

The dispatch gate allow-lists ctrlstatus in every role (incl. no-quorum), so any node answers with its local view (satisfying any-node visibility). An authoritative answer comes only from the node whose responderRole == PRIMARY. When the first reply is from a non-primary, the client follows the redirect — it maps the reply's primaryName to an address via the reply's peers list (or rotates the configured master list) and re-queries — until it obtains the primary's authoritative reply or exhausts its budget. The plain ctrlstatus command may display the first (hint) reply with its "answered by" framing; wait-primary and any authoritative use require the primary's reply. Gate decision: ctrlstatus is allow-listed in every role branch and does not require quorum or a committed read — it returns the node's local view even when fenced or without quorum (precisely when an operator most wants it).

wait-primary probe (Req 9)

wait-primary keeps its existing monotonic-deadline poll loop and timeout/exit semantics; only the probe changes from joblist_query to xcecli_ctrlstatus. Success predicate: the status reply reports a non-"unknown" current primary. On success it prints the served primary's identity. (Rationale + caveat below.)

Requirements Traceability

Requirement	Summary	Components	Interfaces
1.1, 1.2, 1.4	Single-call read-only focused status	handle_ctrlstatus, CtrlStatus schema	opcode ctrlstatus, reply CtrlStatus
1.3	"unknown" for undeterminable fields	handle_ctrlstatus	empty Text/sentinel mapping
2.1, 2.2, 2.3	Current primary identity (unknown when unelected; standby's known primary)	handle_ctrlstatus (reads ha->primary_host under ha->lock)	CtrlStatus.primaryName
3.1, 3.2, 3.3	Answered-by node + role, distinct from primary	handle_ctrlstatus	CtrlStatus.responderName, responderRole
4.1, 4.2	Configured controllers + count	handle_ctrlstatus (reads peers_cfg)	CtrlStatus.peers, peerCount
5.1, 5.2, 5.3	Config version + content checksum (loaded pair + committed)	handle_ctrlstatus (reads loaded config_version_tag/config_sha256 + committed_config_tag)	CtrlStatus.configVersion, configChecksum, committedConfigVersion
6.1–6.4	Term, commit/apply, fenced, cluster name	handle_ctrlstatus	CtrlStatus.term, committedLsn, appliedLsn, fenced, clusterName
7.1, 7.2, 7.3	Failover-aware any-node delivery	xcecli_ctrlstatus, dispatch-gate allow-lists	XCECLI_FAILOVER_CALL (is_write=false), gate entries
8.1, 8.2, 8.3	Test-client ctrlstatus command	cmd_ctrlstatus	client command + exit code
9.1–9.4	wait-primary uses an authoritative (primary-only) status probe	cmd_wait_primary probe swap, xcecli_ctrlstatus_primary	redirect-to-primary, responderRole == PRIMARY success

Components and Interfaces

Component	Layer	Intent	Req Coverage	Key Dependencies	Contracts
CtrlStatus/CtrlPeer schema	Schema	Reply payload for the status query	2–6	capnp codegen (P0)	API (reply)
handle_ctrlstatus	Controller	Snapshot local HA/config state into the reply	1–6	ha_state_t, myconf_t, persist accessors (P0)	Service
dispatch-gate allow-lists	Controller HA	Permit ctrlstatus in every role	7	ha_dispatch_gate (P0)	Service
xcecli_ctrlstatus / xcecli_ctrlstatus_primary	libxce	Failover-aware read (any-node) + authoritative reach-primary read + free	1, 7, 9	failover wrapper, redirect hint (P0)	Service
cmd_ctrlstatus	Test client	Print the status (responder view)	8	xcecli_ctrlstatus (P0)	State
cmd_wait_primary probe	Test client	Authoritative serving-primary probe	9	xcecli_ctrlstatus_primary (P0)	State

Schema — reply payload (Cap'n Proto)

struct CtrlPeer $C.mapname("xce_ctrlpeer_t") {
  name @0: Text;
  host @1: Text;
  port @2: UInt32;
}

struct CtrlStatus $C.mapname("xce_ctrlstatus_t") {
  clusterName       @0:  Text;
  primaryName       @1:  Text;    # empty ⇒ "unknown" (no elected/known primary)
  responderName     @2:  Text;    # the node that answered
  responderRole     @3:  UInt8;   # ctrl_role_t: INIT/ELECTING/PRIMARY/STANDBY
  term              @4:  UInt64;
  # Loaded config (consistent pair: version is the 8-byte prefix of the checksum)
  configVersion     @5:  UInt64;  # loaded config_version_tag (8-byte SHA-256 prefix)
  configChecksum    @6:  Data;    # loaded config_sha256 (full 32 bytes)
  # Committed config version (may differ from loaded mid config-change)
  committedConfigVersion @7: UInt64;  # committed_config_tag
  committedLsn      @8:  UInt64;
  appliedLsn        @9:  UInt64;
  fenced            @10: Bool;    # in_catch_up OR recovery_unrecoverable
  peers             @11: List(CtrlPeer) $C.mapname("peers.list") $C.maplistcount("peers.count");
  peerCount         @12: UInt32;
}

• Config fields (Req 5): configVersion/configChecksum are the loaded config's tag and full SHA-256 — a self-consistent pair (the version is the first 8 bytes of the checksum), so two nodes with identical loaded content match on both and divergent content differs on both (5.3). committedConfigVersion is the separately-tracked committed tag, exposing "what I run" vs "what is committed"; in steady state all three agree.
• New OPCode ordinal: ctrlstatus @40 (next free after preVoteReply @39). New reply-union member: ctrlstatus @17 (next free after walackreplymsg @16). Field ordinals are append-only — never reuse/reorder.

handle_ctrlstatus — Service contract

Field	Detail
Intent	Build a CtrlStatus from the responding node's local HA/config state and reply
Requirements	1.1–1.4, 2.1–2.3, 3.1–3.3, 4.1–4.2, 5.1–5.3, 6.1–6.4

Responsibilities & Constraints

• Read-only: must not modify any HA, persist, or config state (1.2).
• Field sourcing: clusterName/responderName from c->myconf/own peerinfo; responderRole, term, appliedLsn, committedLsn, committedConfigVersion (committed_config_tag) via atomic_load_explicit(..., acquire); configVersion from the loaded config_version_tag and configChecksum from the loaded config_sha256[32] (a self-consistent pair — Req 5.3); primaryName/peers as below; fenced = in_catch_up || persist_recovery_unrecoverable(...).
• Locking: primary_host/primary_port are guarded by ha->lock — take it for a consistent snapshot, copy out, release. All other fields are atomics or immutable config.
• "unknown" mapping (1.3, 2.2, 2.3): if primary_host[0] == '\0' (no known primary), emit an empty primaryName, which the client renders as "unknown".
• Peers (4.x): iterate c->myconf->peers_cfg.candidates[0..count-1] → one CtrlPeer{name,host,port} each; set peerCount.
• Answerable in every role; returns the local view even when fenced/no-quorum.

Dependencies

• Inbound: cluster.c opcode switch — routes the request (P0).
• Outbound: ha_state_t, myconf_t, persist accessors — read-only snapshot (P0).
• Outbound: init_reply_by_req, xce_reply — reply encode/send (P0).

Dispatch-gate allow-lists — Service contract

• Add OPCode_ctrlstatus to the allow-list in ha_dispatch_gate for CTRL_PRIMARY (including the no-quorum branch), CTRL_STANDBY, CTRL_ELECTING, and CTRL_INIT (7.1, 7.2).
• Constraint: the change is scoped strictly to ctrlstatus; no other opcode's gating may widen. ctrlstatus never requires quorum or a committed read.

libxce client API — Service contract

/* opcode-only request sender (clone of xce_clusterconf_req) */
int  xce_ctrlstatus_req(endpoint_t* ep);
/* send + await reply; on success MOVE_PTR the reply union out */
int  xce_ctrlstatus(xce_ctx_t* ctx, xce_ctrlstatus_t** status);
/* failover-aware read (is_write=false): returns whatever node answers (may be a
 * standby hint view). Use for the operator command. */
int  xcecli_ctrlstatus(xcecli_ctx_t c, xce_ctrlstatus_t** status);
/* AUTHORITATIVE read: returns only a reply whose responderRole == PRIMARY. If the
 * first answer is from a non-primary, follow the redirect — target the reported
 * primaryName via the reply's peers list, else rotate the configured master list
 * — until the primary answers or the failover budget is exhausted. */
int  xcecli_ctrlstatus_primary(xcecli_ctx_t c, xce_ctrlstatus_t** status);
#define xcecli_free_ctrlstatus(s)  xce_free_ctrlstatus(s)

• Preconditions: valid client context.
• Postconditions: on success *status owns a decoded CtrlStatus (caller frees); on failure returns a typed negative (7.3) and *status untouched. For xcecli_ctrlstatus_primary, success additionally guarantees responderRole == PRIMARY (an authoritative answer).
• Invariants: is_write=false — retryable/rotatable across the configured controllers.

cmd_ctrlstatus (test client) — State contract

• Calls xcecli_ctrlstatus; on success prints the fields in human-readable form (8.1) — Primary, answered by <responderName> (<role>) distinct from Primary: (3.3), peers + count, configVersion and committedConfigVersion, configChecksum as hex, term, commit/apply, fenced, cluster name; renders empty primaryName as unknown. Returns 0 on success, <0 on failure with the error printed (8.2, 8.3). Frees the reply.

cmd_wait_primary probe swap — State contract

• Replace the single joblist_query(...) probe call in the deadline loop with xcecli_ctrlstatus_primary (the authoritative variant).
• Success predicate (9.1, 9.2): the probe obtains an authoritative reply with responderRole == PRIMARY (the answer came from the serving primary). On success, complete and print that primary's identity. A standby-only/hint answer is NOT success — the probe follows the redirect to the primary (or keeps polling) until the primary answers or the deadline elapses.
• Decision (resolves review Issue 1): require the answer to come from the primary, not merely "a primary name was reported by some node." This avoids a false success on a standby's stale primary_host after a primary death (the false-positive class removed from the durability soak) and faithfully meets Req 9.1's "serving primary." The any-node hint view remains available for the operator ctrlstatus command via xcecli_ctrlstatus.
• Preserved (9.4): the timeout-argument parsing, monotonic deadline, poll interval, log-level quieting, and success/failure exit codes are unchanged; only the probe call and the success message change.

Data Models

The only new data is the reply payload above. CtrlStatus is a flat value object (no persistence, no aggregate); CtrlPeer is a bounded value list (≤ configured controller count, typically 3). configChecksum is the loaded config's 32-byte SHA-256 and configVersion is its own 8-byte prefix (a self-consistent pair); committedConfigVersion is the separately-tracked committed tag. Two nodes with identical loaded config content produce identical configVersion + configChecksum; differing content differs in both (5.3). A node mid config-change may show configVersion != committedConfigVersion.

Error Handling

• No reachable controller (7.3): xcecli_ctrlstatus returns the failover wrapper's typed negative (e.g. no-master/timeout); cmd_ctrlstatus prints it and returns non-zero (8.3); wait-primary keeps polling until its deadline (9.3).
• Undeterminable field (1.3): emit the defined "unknown"/empty value, never uninitialized data.
• Standby with no known primary (2.3): primaryName empty → "unknown"; the call still succeeds (the status itself is valid).
• Read-only guarantee (1.2): the handler takes only read locks/atomic loads; no state mutation on any path.

Testing Strategy

Unit / handler

• handle_ctrlstatus on a PRIMARY returns primaryName==self, responderRole==PRIMARY, populated peers/version/checksum (2.1, 3.x, 4.x, 5.x).
• handle_ctrlstatus on a STANDBY with a known primary returns that primary and responderRole==STANDBY; with no known primary returns empty primaryName (2.2, 2.3).
• fenced reflects in_catch_up/recovery_unrecoverable (6.3); the handler mutates no state (1.2).
• configVersion/configChecksum are a self-consistent loaded pair — equal across two nodes with identical loaded config, differ when content differs; committedConfigVersion is exposed separately (5.1, 5.2, 5.3).

Integration

• Dispatch-gate: a ctrlstatus request is answered (not -XCE_ERR_STANDBY/-XCE_ERR_NOT_READY) on STANDBY, ELECTING, INIT, and PRIMARY-no-quorum nodes; no other opcode's gating changed (7.1, 7.2).
• libxce round-trip: xcecli_ctrlstatus decodes all fields; is_write=false rotates across configured controllers; no-reachable-controller returns a typed failure (7.3).
• Authoritative read: xcecli_ctrlstatus_primary, when the first answer is from a standby, follows the redirect to the primary and returns a reply with responderRole == PRIMARY; it never returns a standby/hint reply as the authoritative result (9.1, 9.2).

E2E (live /xce-deploy-ha)

• ctrlstatus against a healthy 3-node cluster prints the primary, peers, version+checksum; querying a standby still reports the primary with answered by <standby> (2.3, 8.1).
• wait-primary (authoritative probe) returns immediately on a healthy cluster and names the primary; after a primary kill, blocks until the new primary itself answers, then names it; does NOT false-succeed on a standby's stale primary view; times out when no primary serves (9.1–9.4). Directly useful to the ha-commit-durability Phase 6 liveness/>60s-election investigation.

Security Considerations

ctrlstatus exposes cluster topology, role, term, and config version/checksum — the same trust level as the existing clusterconf/state-report reads (no secrets, no config content, no job data). It is read-only and goes through the same authenticated client path as other client ops; no new authorization surface is introduced. Broadening the dispatch gate is scoped strictly to this one read opcode.

---

Design Delta — Requirements 10 & 11 (2026-06-20)

Merge note (as-built correction). The handler shipped as ctrlstatus_fill + handle_ctrlstatus in src/controller/ctrlstatus.c (not reply.c); configChecksum is carried as Text lowercase-hex (the Data capnp type maps to the project's structured xce_data_t); primaryName is role-derived (primary → self, else empty) after E2E fix ade0081 — ha->primary_host is vestigial. The Req 10/11 changes are written against this as-built reality; the Reqs 1–9 design above stands as the original record.

What changed and why

Live E2E (2026-06-20): a standby's ctrlstatus reported committedLsn: 0 / committedConfigVersion: 0 while appliedLsn: 38 — an apparent commit/apply inversion — because ctrlstatus_fill reads the primary-only markers persist.committed_lsn (ctrlstatus.c:130) and ha->committed_config_tag (:109), which stay 0 on a node that never led. And there was no responder timestamp to line a captured status up against ctrl-N.log. Req 10 fixes the off-primary committed sourcing; Req 11 adds a responder snapshot timestamp.

Boundary Commitments (delta)

• This spec owns (added): the off-primary field-sourcing fix inside ctrlstatus_fill (read the learned committed values when not primary); a new read-only CtrlStatus.responderTimeMs field stamped at snapshot time; its render in cmd_ctrlstatus.
• Out of Boundary (firm): making the off-primary committed value live-fresh — i.e. consuming HeartbeatMsg.committedLsn / extending OP_nop to refresh last_known_committed_lsn on every heartbeat — is a heartbeat/replication-protocol change. It violates Req 10.4 and is routed to ha-heartbeat-liveness / ha-repl-hardening. controller-status reports the honest best-known value only. (Note: superseded by ha-heartbeat-committed-propagation — that spec implemented exactly this, adding ha_consume_heartbeat_committed/ha_observe_heartbeat_committed, invoked from the primary's heartbeat in cluster.c, which now refreshes last_known_committed_lsn/last_known_committed_config_tag on every heartbeat, term-guarded and monotonic. See that spec's design.md.)
• Allowed Dependencies (added): read access to ha->last_known_committed_lsn (_Atomic uint64_t, ha.h:183) and ha->last_known_committed_config_tag (ha.h:185); clock_gettime(CLOCK_REALTIME); the existing xcelog.h wall-clock format for client rendering. (Note: last_known_committed_config_tag is now also _Atomic uint64_t — migrated by ha-heartbeat-committed-propagation from the plain uint64_t this design was written against.)
• Revalidation Triggers (added): any change to where/when last_known_committed_lsn/last_known_committed_config_tag are written (today: cluster_apply_ctrlhello_reply at join/RDHUP, ha_observe_heartbeat_committed on every heartbeat, and the election safe-commit path) → re-check the off-primary committed reporting and the staleness caveat. (Note: the heartbeat writer was added by ha-heartbeat-committed-propagation, superseding the "join/RDHUP only" premise below.)

Locked design decisions (from research items)

1. Req 10 sourcing → role-branch (not uniform max()). When responderRole == CTRL_PRIMARY, keep the node's own authoritative live markers (persist.committed_lsn, ha->committed_config_tag). Otherwise report the learned pair (ha->last_known_committed_lsn, ha->last_known_committed_config_tag). Rationale: a standby's learned value is its only honest "committed" knowledge; a primary's own marker is authoritative and must not be shadowed by a possibly-stale learned value. (Mirrors the last_known_* read precedent at ha.c:5319 (ha_vote_self_freshness), ha.c:6732.)
2. Req 10 staleness caveat (documented behavior, not a bug). last_known_committed_lsn is refreshed at CtrlHello join / RDHUP-failover, not per heartbeat, so an off-primary committed value MAY lag the live primary between joins. This is the in-boundary contract; live-fresh is the out-of-boundary item above. The 0-inversion is eliminated regardless (a standby that has joined has a non-zero learned value; a never-joined node correctly reports 0 = unknown, per Req 1.3 / 10.2). (Note: superseded by ha-heartbeat-committed-propagation — last_known_committed_lsn/_config_tag are now also refreshed on every heartbeat via ha_observe_heartbeat_committed, so the join/RDHUP-only staleness window described here no longer applies; see that spec's design.md.)
3. Req 11 wire type → UInt64 epoch-milliseconds. Numeric, locale-safe; the client formats it. (Pre-formatted Text was the alternative; rejected to keep the wire value machine-comparable and avoid encoding a locale into the reply.)
4. Req 11 clock → CLOCK_REALTIME, stamped in ctrlstatus_fill at snapshot time. NOT CLOCK_MONOTONIC (which ha_on_heartbeat_received uses for liveness) — monotonic is not wall-clock and cannot be correlated to logs. The client renders the millis to local YYYY-MM-DD HH:MM:SS,mmm using the xcelog.h convention so it is directly comparable to a log line.

Research item carried to implementation

• Confirm the exact wall-clock formatter/helper in src/include/xcelog.h and reuse it (or match its YYYY-MM-DD HH:MM:SS,mmm form) in cmd_ctrlstatus so the printed timestamp is byte-comparable to ctrl-N.log. last_known_committed_config_tag was non-atomic at design time (written under lock in ha_convergence.c); the status read was planned as a best-effort aligned 8-byte load (acceptable for a reporting snapshot) — do not add locking on the read path. (Note: superseded by ha-heartbeat-committed-propagation, which migrated the field to _Atomic uint64_t at ha.h:185, written via atomic_store_explicit at ha_convergence.c:368 and ha.c:9011; the as-built ctrlstatus_fill correctly reads it with atomic_load_explicit, not a bare aligned load.)

Schema delta (Cap'n Proto)

Append one field to CtrlStatus (next free ordinal @13, after peerCount @12; append-only, never reorder):

  responderTimeMs   @13: UInt64 $C.mapname("responder_time_ms");  # CLOCK_REALTIME epoch millis at snapshot (Req 11.1)

No change to committedLsn @8 / committedConfigVersion @7 wire shape — Req 10 changes only the value source, not the field.

ctrlstatus_fill delta — Service contract (src/controller/ctrlstatus.c)

• Req 10 (committed sourcing): replace the unconditional reads at ctrlstatus.c:108–110,130 with the role-branch above:
  • responderRole == CTRL_PRIMARY → committedLsn = persist.committed_lsn, committedConfigVersion = ha->committed_config_tag (atomic acquire loads, unchanged).
  • else → committedLsn = ha->last_known_committed_lsn (atomic acquire), committedConfigVersion = ha->last_known_committed_config_tag (plain aligned load).
  • appliedLsn (ha->applied_lsn) is unchanged — it is already correct in every role.
• Req 11 (timestamp): stamp st->responder_time_ms from clock_gettime(CLOCK_REALTIME) (tv_sec*1000 + tv_nsec/1e6) at the top of the snapshot, before/with the other field loads, so it marks the responder's sample moment.
• Read-only invariant (10.4, 1.2) preserved: all additions are atomic/aligned loads + a clock read; no HA/persist/config mutation.

cmd_ctrlstatus delta — State contract (src/controller/testsuite/test.c)

• Print one added line rendering responder_time_ms as local YYYY-MM-DD HH:MM:SS,mmm (xcelog.h form) alongside the existing fields (Req 11.3). The committed lines need no rendering change — Req 10 only changes the values they already print.

File Structure Plan (delta)

Modified files:

• src/schema/controller.capnp — add responderTimeMs @13: UInt64 to CtrlStatus (regen controller.capnp.h/.c → xce_ctrlstatus_t.responder_time_ms).
• src/controller/ctrlstatus.c — ctrlstatus_fill: role-branch committed sourcing (Req 10) + CLOCK_REALTIME timestamp stamp (Req 11).
• src/controller/testsuite/test.c — cmd_ctrlstatus: render the responder timestamp line (Req 11.3).
• src/controller/testsuite/test_ctrlstatus.c — add/extend unit tests (below).

No new files (the capnp field is generated; all logic slots into existing TUs).

Requirements Traceability (delta)

Requirement	Summary	Components	Interfaces
10.1	Off-primary reports learned committed LSN + config version	ctrlstatus_fill role-branch	CtrlStatus.committedLsn, committedConfigVersion (value source = ha->last_known_committed_*)
10.2	No learned value yet → 0/unknown	ctrlstatus_fill	last_known_committed_lsn == 0 (correct unknown, per 1.3)
10.3	No spurious applied>committed inversion	ctrlstatus_fill	appliedLsn vs role-correct committedLsn
10.4	Read-only reporting; no production/replication change	ctrlstatus_fill; out-of-boundary routing	atomic/aligned loads only
11.1	Responder wall-clock at sample time	ctrlstatus_fill (CLOCK_REALTIME)	CtrlStatus.responderTimeMs
11.2	Same clock/format as logs	cmd_ctrlstatus render (xcelog.h form)	local YYYY-MM-DD HH:MM:SS,mmm
11.3	Test client prints the timestamp	cmd_ctrlstatus	added print line

Testing Strategy (delta)

Unit (test_ctrlstatus.c, TDD red→green):

• STANDBY with last_known_committed_lsn = 36, persist.committed_lsn = 0, applied_lsn = 38 → reply reports committedLsn == 36 (not 0) and appliedLsn == 38 (no inversion below the learned value) (10.1, 10.3).
• STANDBY pre-join (last_known_committed_lsn == 0) → reply reports committedLsn == 0 / committedConfigVersion == 0 (correct unknown) (10.2).
• PRIMARY → reply reports committedLsn == persist.committed_lsn and committedConfigVersion == ha->committed_config_tag (authoritative own markers unchanged) (10.1 primary branch).
• committedConfigVersion off-primary equals last_known_committed_config_tag (symmetry with the LSN) (10.1).
• responder_time_ms is non-zero and within a tolerance of a test-captured CLOCK_REALTIME sample; the handler still mutates no state (11.1, 1.2/10.4).

Integration / round-trip (libxce): xcecli_ctrlstatus decodes the new responderTimeMs field end-to-end (no manual decode — codegen), proving wire compatibility.

E2E (live /xce-deploy-ha): reproduce the original finding — submit jobs so the primary commits, query a standby; assert the standby's committedLsn is non-zero and tracks (within the join-refresh caveat) rather than 0, and that the printed responder timestamp matches the standby's own ctrl-N.log line for that query. (Note: the "join-refresh caveat" is narrower than described above — ha-heartbeat-committed-propagation added per-heartbeat refresh, so tracking is now near-live rather than join/RDHUP-only; see that spec's design.md.)

---

Design Delta — Requirement 12 (2026-09-15)

What changed and why

User enhancement request: ctrlstatus should additionally report the term in which the current primary was elected and the wall-clock time it was elected, so operators can tell how long the current primary has served and correlate its election with the logs — with the same off-primary (standby) fidelity Req 10 already established for committed position, not a bare "unknown" off-primary. Scope of this delta = Req 12 only. Reqs 1–11 are implemented and unchanged.

Boundary Commitments (delta)

• This spec owns (added): a new ha_state_t.last_known_elected_at_ms field (_Atomic uint64_t, declared alongside last_known_committed_lsn/last_known_committed_config_tag in ha.h), self-stamped by BOTH promotion roads at the exact point each commits to CTRL_PRIMARY; a new HeartbeatMsg.electedAtMs wire field the primary echoes on every outgoing heartbeat; the receiving standby's unconditional store of that value into its own last_known_elected_at_ms; a new CtrlStatus.electedAtMs reply field; ctrlstatus_fill populating it (one unconditional read, no role-branch — see Locked Decision 2); cmd_ctrlstatus's two new printed lines (electedTerm, electedAt), gated on elected_at_ms == 0 (corrected 2026-09-16 from an original primaryName-based gate — see Locked Decision 3).
• Out of Boundary (firm): electedTerm gets no new field — it reuses the existing CtrlStatus.term (Req 6.1) with CLI-side "unknown" gating (Locked Decision 1); this spec does not change how elections are conducted, how ha->term advances, or how/when ha->role transitions — every stamp site added here is purely observational at an already-decided promotion instant (Req 12.5).
• Allowed Dependencies (added): ha->term and the existing primaryName derivation (Req 2/3, unchanged); clock_gettime(CLOCK_REALTIME) (existing precedent, Req 11); ctrlstatus_format_time_ms (src/include/xce/ctrlstatus_fmt.h) reused verbatim for rendering — no new formatting code.
• Revalidation Triggers (added): any change to where ha->role finally becomes CTRL_PRIMARY on either promotion road (ha_do_promote or raft_cb_become_leader) → re-check the last_known_elected_at_ms stamp site is still past every refusal/eligibility gate; any change to the HeartbeatMsg/heartbeat_msg_t shape → re-check the electedAtMs ordinal and decode; any change to the primaryName-unknown derivation (Req 2) → re-check the electedTerm/electedAtMs "unknown" gating that reuses it.

Locked design decisions

1. electedTerm reuses ha->term (Req 6.1's existing wire field) — no new state. This codebase's Raft term already propagates correctly to every node via the existing heartbeat/vote term-sync path; there is no "stale/zero" class of bug here the way Req 10 had for committedLsn (which read an unrelated primary-only marker, persist.committed_lsn, that stays 0 off-primary by construction). cmd_ctrlstatus derives the displayed electedTerm as elected_at_ms != 0 ? term : unknown (corrected 2026-09-16 — see Locked Decision 3; originally primaryName known ? term : unknown) — purely a print-layer decision, no handler change for this half of the requirement. Accepted trade-off: in the narrow window where a standby has just voted for a new candidate (bumping its own ha->term) before its voted_for_name-derived primaryName reflects that same candidate, the displayed electedTerm could transiently be a half-step out of step with the displayed primary — the same class of transient standby-hint staleness already accepted for the original primaryName/term fields (see Architecture's "any-node hint then authoritative-from-primary" flow) and for Req 10's own documented staleness caveat. Not a regression, not new risk surface.
2. electedAtMs is genuinely new state, written unconditionally (no monotonic/term compare on write). Both promotion roads write last_known_elected_at_ms exactly once, at the instant each road COMMITS to CTRL_PRIMARY — i.e., past every refusal/eligibility gate, never at function entry (see Research Items below for the exact confirm-at-implementation lines). The heartbeat-receive site writes it only when already processing a heartbeat from the node's currently-recognized leader — the same precondition ha_observe_heartbeat_committed's existing call site already relies on for committed_lsn/committed_config_tag. Because every write site is already gated by an existing freshness precondition, no second "which term does my stored value correspond to" comparison is needed. Simplification vs. Req 10's pattern: since there is no pre-existing primary-only marker analogous to persist.committed_lsn for election time, ctrlstatus_fill needs no role-branch for this field — one unconditional read of last_known_elected_at_ms correctly serves both the primary (self-stamped) and a standby (heartbeat-learned) case.
3. Unknown gating happens at output time, not by proactively zeroing state. Both electedTerm and electedAtMs are gated in cmd_ctrlstatus on elected_at_ms == 0 (see live-E2E correction below), not on the primaryName-known/unknown derivation Req 2.2/2.3 computes. A stale last_known_elected_at_ms left over from a since-lost primary is never surfaced because a fresh promotion or heartbeat-learn always overwrites it before it could be observed as stale — the reader never sees an in-between state, so no separate staleness check is needed beyond the zero-check itself.

   Live-E2E correction (2026-09-16, task 7.6 finding): this decision originally gated on the SAME primaryName-known/unknown derivation Req 2.2/2.3 already computes, reasoning it would mirror how the original design treats a stale voted_for_name/primary_host. A live 3-node cluster run proved this wrong: primaryName is unconditionally empty on every non-primary responder (Req 2's own ctrlstatus_fill only self-reports the primary's name when the responder itself is primary — ha->primary_host is vestigial, per the ade0081 design note above), so Req 2.3's "standby knows the primary" branch is dead code in the current build via the plain ctrlstatus command. Gating on primaryName therefore made electedTerm/electedAt print "unknown" on every standby, permanently — even though elected_at_ms was correctly, non-zero, and convergently populated the whole time (independently proven live: term/committedLsn/appliedLsn, sourced the identical unconditional way, DID correctly display and converge on the same standby query). Gating on status->elected_at_ms == 0 instead is both necessary and sufficient: it stays 0 until a real promotion stamps it (either road) or a heartbeat carries a learned value, and — unlike term, which can be legitimately nonzero from mere vote/candidacy activity before any primary is actually elected — elected_at_ms only ever becomes nonzero via an actual election. This is a cmd_ctrlstatus-only correction; ctrlstatus_fill's own unconditional, no-role-branch read (Locked Decision 2) is unaffected and was already correct.

4. Wire type: electedAtMs @14: UInt64 epoch-milliseconds — mirrors Req 11's responderTimeMs numeric-not-Text precedent exactly (machine-comparable; the client formats it via the same existing ctrlstatus_format_time_ms).

Research items carried to implementation

1. Confirm the exact statement in raft_cb_become_leader (ha.c:1615) where ha->role is finally stored as CTRL_PRIMARY — past the ha_promotion_gate_decision eligibility gate (later in the function than this delta's own discovery reached) — and stamp last_known_elected_at_ms there. Do not stamp at the earlier last_known_committed_lsn self-observe point (ha.c:1665, inside the if (promo_ectx) block): that point precedes the eligibility gate and can still be followed by a promotion refusal, which must not produce a bogus election time.
2. Confirm which of ha_observe_heartbeat_committed's 9 existing call sites corresponds to processing a genuinely RECEIVED heartbeat (as opposed to a local/test-only invocation) and add the unconditional last_known_elected_at_ms store immediately alongside that call, reading the new electedAtMs field off the same decoded heartbeat_msg_t. Also confirm the outgoing side — where primary_heartbeat_fn (ha.c:9791) populates the heartbeat_msg_t it sends via xce_ha_heartbeat_req — and set electedAtMs = last_known_elected_at_ms there.
3. Confirm ha_do_promote (ha.c:2927, the function ha_promote_to_primary's own comment describes as owning "the term increment + sets role=PRIMARY" for the non-raft road) is the correct single stamp site for that road; its body was outside this delta's discovery window.

Constraints checked

• Async I/O invariant (ha-async-io.md): every write added here is an atomic store or a clock_gettime(CLOCK_REALTIME) call on the existing promotion/heartbeat control-flow paths — no new connect(), file I/O, or other blocking call is introduced; identical shape to Req 11's already-accepted CLOCK_REALTIME stamp.
• Persistence / save-restore completeness: not applicable — last_known_elected_at_ms is ephemeral in-memory HA state, never written to ha_term_file_t/WAL/any durable format, and is fully re-derived on the next promotion or heartbeat exactly like its sibling last_known_committed_lsn/last_known_committed_config_tag (also not persisted). No save/restore path is added or modified by this delta.

Schema delta (Cap'n Proto)

# CtrlStatus — next free ordinal @14 (after responderTimeMs @13; append-only, never reorder)
  electedAtMs   @14: UInt64 $C.mapname("elected_at_ms");
  # epoch millis of the current primary's election, learned off-primary via
  # heartbeat; 0/unset when no primary is known (gated together with primaryName)

# HeartbeatMsg — next free ordinal @4 (after committedConfigTag @3)
  electedAtMs   @4: UInt64 $C.mapname("elected_at_ms");
  # primary's own last_known_elected_at_ms, echoed on every heartbeat (Req 12.2)

No change to term @4 on CtrlStatus — Req 12's term half changes only the CLI's derived label/gating, not the wire field or its value.

File Structure Plan (delta)

Modified files:

• ha.h — add _Atomic uint64_t last_known_elected_at_ms; to ha_state_t, next to last_known_committed_lsn/last_known_committed_config_tag.
• src/schema/controller.capnp — add electedAtMs to CtrlStatus (@14) and to HeartbeatMsg (@4); regen controller.capnp.h/.c.
• src/controller/ha.c — ha_do_promote and raft_cb_become_leader: stamp last_known_elected_at_ms from CLOCK_REALTIME at each road's final promotion-commit point (Research Items 1, 3); the heartbeat-receive call site (ha_consume_heartbeat_committed): unconditional store alongside the existing ha_observe_heartbeat_committed call (Research Item 2); primary_heartbeat_fn: read last_known_elected_at_ms into a local, passed through to the outgoing heartbeat (Research Item 2).
• src/controller/ctrlstatus.c — ctrlstatus_fill: read last_known_elected_at_ms unconditionally into st->elected_at_ms (no role-branch, Locked Decision 2).
• src/harness/test.c — cmd_ctrlstatus: two new printed lines, electedTerm (derived from term, gated on elected_at_ms) and electedAt (rendered via ctrlstatus_format_time_ms), both "unknown" when elected_at_ms == 0 (corrected 2026-09-16 — see Locked Decision 3; originally gated on primaryName). Path correction (as-built, confirmed 2026-09-15): cmd_ctrlstatus/cmd_wait_primary now live in src/harness/test.c, not src/controller/testsuite/test.c as Reqs 1-11's original design text says — the test-client harness was consolidated into src/harness/ by a later refactor (see the xce-harness-component spec) after those sections were written. Historical Reqs 1-11 text is left as-is (accurate as of when implemented); this delta targets the current, correct location.
• src/include/request.h, src/lib/request.c — as-built addition (2026-09-16), not anticipated at design time: xce_ha_heartbeat_req, not primary_heartbeat_fn directly, owns building the outgoing heartbeat_msg_t (it already threaded term/committed_lsn/committed_config_tag as scalar args this same way) — task 7.3 found this only once it read the real send path, and added elected_at_ms as a 4th scalar parameter, mechanically mirroring the 3 existing ones. No election/heartbeat protocol semantics changed, only one added field in an already-established parameter-threading pattern.
• src/controller/testsuite/test_ctrlstatus.c — extend/add unit tests (below).
• src/controller/testsuite/test_ha_heartbeat_committed.c — coverage for the heartbeat-carried electedAtMs store (the reconcile-free negative-space test, mirroring test_observe_no_reconcile_no_other_mutation).
• src/controller/testsuite/test_ha_role.c — extend test_promote_increments_term-style coverage to also assert last_known_elected_at_ms is stamped on promotion (and NOT stamped on a refused promotion attempt).
• src/controller/testsuite/test_ha_liveness.c, src/controller/testsuite/test_ha_partition.c — as-built addition (2026-09-16): signature-conformance updates only (these files mirror ha_heartbeat_req_hook's signature for their own test doubles; the new 4th parameter had to be threaded through, no test behavior changed).
• src/lib/testsuite/test_request.c — as-built addition (2026-09-16): extends the existing xce_ha_heartbeat_req round-trip test to also cover elected_at_ms, proving the new field survives a real encode/decode cycle.

No new files (capnp fields are generated; all logic slots into existing TUs) — consistent with this spec's established pattern. Known gap (2026-09-16 feature-level validation): cmd_ctrlstatus's print-gate logic (the exact code bug-1818 regressed in) has no checked-in unit/regression test — this print layer has never had one, for any of its lines, going back to Req 8's original implementation; coverage is live-E2E (task 7.6) plus an ephemeral scratch-program check at fix time, not a permanent regression guard. Accepted as a pre-existing, deliberate scope boundary of this spec (see Requirement 8's own design text), not a new gap introduced by this delta — flagged here for whoever next touches this function.

Requirements Traceability (delta)

Requirement	Summary	Components	Interfaces
12.1	Report the primary's election term + time alongside its identity	ctrlstatus_fill (electedAtMs read), cmd_ctrlstatus (electedTerm derived from term, gated on elected_at_ms)	CtrlStatus.term (reused), CtrlStatus.electedAtMs (new)
12.2	Standby reports the LEARNED election term/time, not stale/unset	heartbeat-carried electedAtMs; existing term-sync for electedTerm	HeartbeatMsg.electedAtMs, unconditional store into last_known_elected_at_ms
12.3	Unknown when no primary is known	cmd_ctrlstatus gating on elected_at_ms == 0 (corrected 2026-09-16, see Locked Decision 3)	independent of Req 2.2/2.3's primaryName derivation, which is unconditionally empty on a standby in the current build
12.4	Election time on the same clock/format as logs	ctrlstatus_fill (CLOCK_REALTIME stamp), cmd_ctrlstatus render	reuses ctrlstatus_format_time_ms (Req 11.2 precedent)
12.5	Read-only; no election/leadership behavior change	stamp sites are observational only; no gating logic altered	n/a
12.6	Test client prints both fields	cmd_ctrlstatus	two new printed lines

ctrlstatus_fill / promotion / heartbeat delta — Service contract

• Req 12.1/12.2 (sourcing): st->elected_at_ms = ha->last_known_elected_at_ms (atomic acquire load), unconditionally — no role-branch (Locked Decision 2).
• Req 12.5 (read-only invariant) preserved: the handler itself performs only atomic loads, exactly as every other field; all writes to last_known_elected_at_ms happen upstream, at promotion or heartbeat-processing time, never inside ctrlstatus_fill.
• cmd_ctrlstatus (Req 12.1, 12.3, 12.6): print

  electedTerm: <term>          (or "unknown" when elected_at_ms == 0)
  electedAt: <ctrlstatus_format_time_ms(elected_at_ms)>   (or "unknown" when elected_at_ms == 0)
  

  placed after the existing Primary: line, grouping the primary's identity with its election term/time. Gated on elected_at_ms == 0, not on primaryName (corrected 2026-09-16 — see Locked Decision 3).

Testing Strategy (delta)

Unit (test_ctrlstatus.c):

• PRIMARY: ctrlstatus_fill reports elected_at_ms == ha->last_known_elected_at_ms (self-stamped); CLI-derived electedTerm == st.term.
• STANDBY with a learned last_known_elected_at_ms (simulating an already-processed heartbeat) and a known primaryName → reports the learned value, not 0/stale (12.2).
• STANDBY with primaryName unknown (no vote / self-vote / electing) → CLI renders electedTerm/electedAt as "unknown" regardless of any stale stored last_known_elected_at_ms (12.3).
• Extend the existing test_ctrlstatus_no_mutation snapshot/compare to also cover last_known_elected_at_ms (12.5).

New/extended heartbeat unit test (sibling to test_ha_heartbeat_committed.c):

• Processing a received heartbeat with a given electedAtMs unconditionally overwrites last_known_elected_at_ms and leaves last_known_committed_lsn/last_known_committed_config_tag/role/term/applied untouched — reconcile-free, mirrors test_observe_no_reconcile_no_other_mutation (12.2, 12.5).

Integration:

• Extend test_ha_role.c's test_promote_increments_term-style coverage (non-raft road, ha_do_promote) to also assert last_known_elected_at_ms is stamped on promotion.
• Add/extend a raft_cb_become_leader promotion test to assert the same for the raft road, specifically confirming NO stamp occurs on a refused/gated promotion attempt (Research Item 1).
• Wire round-trip: xcecli_ctrlstatus decodes the new CtrlStatus.electedAtMs field end-to-end (codegen, no manual decode).

E2E (live /xce-deploy-ha):

• Query ctrlstatus on the primary right after a fresh election: electedTerm equals the printed term; electedAt is close to "now."
• Query a STANDBY: electedTerm/electedAt match the primary's own values (within one heartbeat interval), not "unknown" and not stale/zero — proving standby learned fidelity end-to-end (12.2).
• Kill the primary, wait for a new election, query the surviving standby immediately vs. after one heartbeat interval — demonstrates the field converges promptly via heartbeat, unlike Req 10's original join/RDHUP-only caveat (heartbeat-carried delivery here is in-boundary from day one of this delta, not a follow-up spec).

---

Design Delta — Requirement 13 (2026-09-16)

What changed and why

User enhancement request: ctrlstatus's "Controllers (N):" list should additionally show each configured controller's connection status (ok / catching-up / dead), not just name and address, so operators can see at a glance which controllers are healthy, rejoining, or unreachable. Confirmed with the user: three-state granularity (reusing the already-tracked connected-vs-catching-up distinction) and best-effort per-responder reporting — "unknown" for peers a responder cannot itself observe, consistent with this spec's established off-primary fidelity pattern, not a primary-only-authoritative requirement. Scope of this delta = Req 13 only. Reqs 1–12 are implemented and unchanged.

Boundary Commitments (delta)

• This spec owns (added): a new CtrlPeer.status reply field (per-peer, not per-response — this is the first Req 12+ delta to extend the nested CtrlPeer struct rather than CtrlStatus itself); ctrlstatus_fill populating it per peer, sourced differently by responder role (primary: matched against live connection state; non-primary: self only, "unknown" otherwise); a status-code-to-string renderer (mirroring the existing responderRole/ctrlstatus_role_str pattern); cmd_ctrlstatus's peer-list print line extended to show it.
• Out of Boundary (firm): this spec does not change how standby connections are established, monitored, evicted, or torn down — it only reads the connection-tracking state that already exists for replication/quorum purposes (Req 13.5). It does not give a standby visibility into peers it genuinely cannot observe (confirmed at design time: this codebase's connection model is asymmetric — only the primary holds live connections to every standby; standbys do not connect to each other, and no existing mechanism lets a standby identify even its OWN primary by peer identity, only "elapsed time since last primary message") — a standby reporting anything but "unknown" for a peer other than itself would require a genuinely new cross-node identity-carrying mechanism, which is out of scope for this delta (see Locked Decision 3).
• Allowed Dependencies (added): the existing replication connection registry (replicator_t.standbys[], src/include/repl.h) that already tracks each standby's live connection (ep), catch-up state (in_catch_up), and host/port — read-only, no new writes to it.
• Revalidation Triggers (added): any change to standby_conn_t's fields or to how/when ep/in_catch_up are set (repl_add_standby/repl_remove_standby/repl_start_catchup/repl_complete_catchup) → re-check the peer-status sourcing still reads the correct signal; any change to peers_cfg.candidates[]'s host/port fields → re-check the host+port matching still correlates correctly to replicator_t.standbys[].

Locked design decisions

1. Wire type: CtrlPeer.status @3: UInt8, a small fixed-value code — not Text. Mirrors the existing CtrlStatus.responderRole @3: UInt8 precedent exactly (backed by a C enum + a ctrlstatus_..._str() string renderer at the client layer), rather than introducing a new stringly-typed convention for this one field.
2. Status values, in ordinal order: 0 = unknown, 1 = ok, 2 = catching-up, 3 = dead. unknown is 0 (the default/zero value) so a peer entry that nothing ever sets defaults to the safe, honest "can't tell" state rather than an implicitly-true ok.
3. Primary-side sourcing: match by host+port, not by name or position. standby_conn_t (the primary's live connection-tracking slot) stores host/port but not a configured peer name or a stable index into peers_cfg.candidates[] — so ctrlstatus_fill, when the responder is primary, must scan replicator_t.standbys[] for a slot whose host/port match each configured peer's hostname/port. No match (or a matched slot with ep == NULL) → dead; a match with ep != NULL && in_catch_up → catching-up; a match with ep != NULL && !in_catch_up → ok. This mirrors repl_active_standby_count's own existing ep != NULL && !in_catch_up predicate (repl.c) — reuse that exact condition rather than inventing a new one. Correction (task 8.6, 2026-09-16, live-E2E finding): the scan must NOT stop at the first host/port match. A standby that is killed and later reconnects can end up occupying a NEW replicator_t.standbys[] slot while its OLD slot (host/port still populated, ep == NULL) also still matches the same configured peer — repl_add_standby's ep-identity idempotency check cannot reuse the old slot once its ep is gone. A first-match-wins scan then reports that peer dead PERMANENTLY, even after it has genuinely reconnected and fully caught up (live-reproduced: ctrlstatus stayed dead for 5+ minutes after a rejoin the primary's own log showed as fully caught up), violating Req 13.3's "true, currently-known" guarantee. The corrected algorithm scans ALL MAX_STANDBYS slots for a given peer and prefers ANY slot with ep != NULL (deriving ok/catching-up from that live slot) over a slot with ep == NULL; only falls back to dead when no matching slot has ep != NULL. See .wolf/buglog.json bug-1819.
4. The responder always reports its own entry as ok (Req 13.2) — matched by comparing each configured peer's name against st->responderName, the same identity ctrlstatus_fill already establishes for itself.
5. Non-primary responder: only self is determinable; every other peer — including the primary — reports unknown. This was an open question at requirements time (whether a standby could additionally report its own primary's status via the liveness signal it already tracks for failover); design-time investigation closes it: that liveness signal (ha->last_activity, liveness_poll_fn) is a bare elapsed-time scalar with no peer identity attached, and this codebase has no other mechanism (analogous to the vestigial ha->primary_host) that would let a standby reliably attribute it to one specific configured peer. Building one would be materially larger than this delta (a genuinely new identity-carrying signal on the heartbeat/connection-accept path) and is explicitly out of boundary here — a future spec's concern, not silently absorbed into this one.
6. Read-only, no role-branch beyond what Decision 3/5 already requires: ctrlstatus_fill performs only reads (of peers_cfg, and — only when the responder is primary — of replicator_t.standbys[] under its existing lock); no new state is written, no existing connection-management behavior changes.

Research item carried to implementation

1. Confirm the exact field path from cluster_t to its replicator_t (repl.h's own comment says only "embedded in persist_t or cluster_t (P2-4)", ambiguous at design time — likely c->persist.repl given replicator_t.ce couples it to the commit engine which lives off persist_t, but confirm against the current struct definition, not this note, before wiring the read).

Constraints checked

• Async I/O invariant (ha-async-io.md): the new read takes replicator_t.lock (a mylock_t) briefly to scan a small, fixed-size (MAX_STANDBYS) in-memory array — no network I/O, no unbounded wait, identical in shape to the already-established repl_active_standby_count's own lock-and-scan. Correction (second design-review pass, 2026-09-16): the previous version of this note claimed "the primary-identity read already takes ha->lock" — that was true only of the ORIGINAL Req 1-9 design text, not the as-built code: after the ade0081 fix (Req 5.1, primaryName made role-derived/self-only), ctrlstatus_fill's current source takes NO lock anywhere — every existing field is an atomic load or a plain config read. This delta's replicator_t.lock acquisition is therefore the FIRST lock this function has taken since that fix, not a repeat of an existing pattern within ctrlstatus_fill itself. It is still safe: ctrlstatus_fill runs on the request-handling path (not the trigger-only delaymgr thread), and the exact same lock is already taken from repl_active_standby_count on the write-admission quorum path — a comparably latency-sensitive context — with no documented contention issue.
• Persistence / save-restore completeness: not applicable — CtrlPeer.status is derived entirely from existing in-memory connection state (replicator_t.standbys[], already not persisted) and existing config (peers_cfg); nothing new is written to disk, WAL, or any snapshot format, and there is no save/restore path for this delta to audit.

Schema delta (Cap'n Proto)

# CtrlPeer — next free ordinal @3 (after port @2; append-only, never reorder)
struct CtrlPeer $C.mapname("xce_ctrlpeer_t") {
  name   @0: Text;
  host   @1: Text;
  port   @2: UInt32;
  status @3: UInt8 $C.mapname("status");
  # 0=unknown 1=ok 2=catching-up 3=dead (Req 13.1). Always 1 (ok) for the
  # responder's own entry (Req 13.2). On a non-primary responder, every
  # entry but the responder's own is 0 (unknown) — Req 13.4/Locked Decision 5.
}

No change to CtrlStatus's own top-level fields — this delta extends the nested CtrlPeer list element, not the reply envelope.

File Structure Plan (delta)

Modified files:

• src/schema/controller.capnp — add status @3: UInt8 to CtrlPeer; regen controller.capnp.h/.c.
• src/include/typemap.h — add uint8_t status; to the hand-authored xce_ctrlpeer_t struct, matching the schema field.
• src/controller/ctrlstatus.c — ctrlstatus_fill's peer-building loop: set each xce_ctrlpeer_t.status per Locked Decisions 3–5 (self → ok; primary responder → host/port-matched against replicator_t.standbys[]; non-primary responder, non-self → unknown). Add a small ctrlstatus_peer_status_str() (or similarly-named) helper alongside the existing ctrlstatus_role_str, in whichever header/TU that already lives in.
• src/harness/test.c — cmd_ctrlstatus: extend the existing per-peer print line (printf("\t%s\t%s:%u\n", ...)) to also render the status string.
• src/controller/testsuite/test_ctrlstatus.c — extend/add unit tests (below).

No new files (the capnp field is generated; the small status-code enum and its string renderer slot into an existing header/TU) — consistent with this spec's established pattern.

Requirements Traceability (delta)

Requirement	Summary	Components	Interfaces
13.1	Report each peer's status as one of a well-defined value set, alongside name/address	ctrlstatus_fill	CtrlPeer.status (new)
13.2	Responder always reports its own status as ok	ctrlstatus_fill (self-match by name)	CtrlPeer.status == 1 for the self entry
13.3	Primary reports the true status for every other controller	ctrlstatus_fill (host/port match against replicator_t.standbys[])	CtrlPeer.status sourced from ep/in_catch_up
13.4	Non-primary reports unknown for what it cannot determine	ctrlstatus_fill (non-primary branch)	CtrlPeer.status == 0 for every non-self entry
13.5	Read-only; no change to connection lifecycle	ctrlstatus_fill reads replicator_t only, under its existing lock; writes nothing	n/a
13.6	Test client prints each peer's status	cmd_ctrlstatus	extended per-peer print line

ctrlstatus_fill / peer-status delta — Service contract

• Req 13.1–13.4 (sourcing): for each peers_cfg.candidates[i], set status:
  • candidates[i].name == st->responderName → 1 (ok) — unconditional, regardless of role (Req 13.2).
  • Else, responderRole == CTRL_PRIMARY → scan replicator_t.standbys[] (under its lock) for a slot whose host/port match candidates[i].hostname/.port; no match or ep == NULL → 3 (dead); ep != NULL && in_catch_up → 2 (catching-up); ep != NULL && !in_catch_up → 1 (ok).
  • Else (non-primary responder, non-self peer) → 0 (unknown).
• Self-match fallback (design-review finding, 2026-09-16): the self-match is a straightforward name comparison against st->responderName, the same identity ctrlstatus_fill already establishes for itself and already expects to be self-consistent with peers_cfg (this is a pre-existing assumption of the ORIGINAL Req 1-9 design, not new to this delta — a controller whose own name is absent from its own configured peer list is already a misconfiguration visible today, since it would already be missing an entry from the pre-existing Controllers (N): list's identity, unrelated to status). This delta does not need to invent new self-identity infrastructure to handle that pre-existing failure mode, but it MUST NOT let it produce a silently wrong status: if no candidates[i] matches responderName, ctrlstatus_fill shall WARN once per call (visible in the responder's own log, consistent with this codebase's existing fail-loud-not-silent convention) and leave every peer's status to fall through to the normal non-self branches above (i.e., no entry is force-corrected to ok under this condition) — an explicit, logged degradation rather than an undefined or fabricated result. This keeps Req 13.2's "always ok for self" guarantee intact for every correctly-configured cluster while making a misconfigured one loudly debuggable instead of silently misleading.
• Req 13.5 (read-only) preserved: only reads peers_cfg and, conditionally, replicator_t.standbys[] under its existing lock — no new writes anywhere. The WARN log above is observability-only, not a state mutation.
• cmd_ctrlstatus (Req 13.1, 13.6): extend the existing Controllers (N): per-peer line to also print the rendered status string, e.g. \t%s\t%s:%u\t%s\n with the fourth %s from the new status-string helper.

Testing Strategy (delta)

Unit (test_ctrlstatus.c):

• PRIMARY with a full replicator_t.standbys[] (one slot ep != NULL && !in_catch_up, one ep != NULL && in_catch_up, one peer with no matching slot at all) → ctrlstatus_fill reports ok/catching-up/dead respectively for those three peers, and ok for its own entry (13.1–13.3).
• STANDBY responder → reports ok for its own entry and unknown (0) for every other configured peer, including the primary (13.2, 13.4).
• Handler mutates no state — extend the existing test_ctrlstatus_no_mutation to also snapshot/compare whatever replicator_t fields the primary-branch read touches (13.5).
• Status-string renderer: each of the 4 codes renders its expected string; an out-of-range code renders a safe fallback (not garbage), mirroring ctrlstatus_role_str's own convention if it has one.
• Self-match fallback (design-review finding): a fixture whose responderName matches no candidates[i].name (simulating the misconfiguration case) still produces a defined, non-crashing result via the normal non-self branches, and a WARN is logged — proving the degradation is loud and deterministic, not silent or undefined (13.2).

Integration:

• Wire round-trip: xcecli_ctrlstatus decodes the new CtrlPeer.status field for every peer in the list (codegen, no manual decode).

E2E (live /xce-deploy-ha):

• Query ctrlstatus on the primary of a healthy 3-node cluster: all three peers (including itself) show ok.
• Kill one standby (not the primary): re-query the primary — that standby now shows dead, the other two still ok.
• Query a surviving standby directly: its own entry shows ok, every other entry (including the primary) shows unknown — proving the asymmetric-visibility design decision holds live, not just in a unit test.
• (Optional, if a clean way to hold a node in catch-up long enough to observe exists in this project's E2E tooling) restart the killed standby and query the primary again shortly after: that peer transiently shows catching-up before settling to ok.