myxce/.kiro/specs/controller-status/design.md

Design Document — controller-status (ctrlstatus)

Overview

Purpose: add a single, cheap, read-only client→controller query — ctrlstatus, an XCE analog of LSF lsid — that returns basic cluster/controller identity and health: who is the current primary, the answering node's identity and role, the configured controllers (peers), the committed configuration version and content checksum, and a few HA health/progress fields (term, commit/apply position, fenced state, cluster name).

Users: operators inspecting a running cluster, and test/E2E authors orchestrating failover (including wait-primary, which switches its probe to this query).

Impact: today there is no status query — callers infer "is there a serving primary" indirectly (e.g. by listing jobs). This adds a new opcode by cloning the existing opcode-only read pattern of clusterconf, reads only already-maintained HA/config state (no behavior change), and crucially makes the query answerable on any node by adding it to the HA dispatch-gate allow-lists.

Goals

• A new read-only opcode returning the status fields of Requirements 2–6 in one request/response (1.1, 1.4).
• Answerable on any node — a standby reports a hint view (the primary it knows); an authoritative answer comes only from the primary, reached by following the redirect / rotating the master list (2.3, 7.1–7.3).
• Report the loaded config version + content checksum (and the committed version) so same-version divergence is detectable (5.2, 5.3).
• A ctrlstatus test-client command (8.x) and a wait-primary probe switched to an authoritative (primary-only) status query (9.x).

Non-Goals

• No operator-facing CLI/REST lsid surface (Go tier) — later pass.
• No change to HA election, replication, or commit behavior — the query only reads existing state.
• No new configuration checksum computation — the committed-config SHA-256 already exists; this only exposes it.

Boundary Commitments

This Spec Owns

• A new Cap'n Proto opcode ctrlstatus and its reply payload (CtrlStatus, CtrlPeer).
• The controller handler handle_ctrlstatus that snapshots local HA/config state into the reply.
• The dispatch-gate allow-list entries that let ctrlstatus answer in every controller role (incl. no-quorum).
• The libxce client API (xce_ctrlstatus_req / xce_ctrlstatus / xcecli_ctrlstatus + free helper).
• The ctrlstatus test-client command and the wait-primary probe switch.

Out of Boundary

• The Go CLI/REST lsid surface (later spec).
• Any change to how HA state (role, term, primary identity, committed config tag, commit/apply LSN, fence) is produced or maintained — this spec only reads it.
• The existing clusterconf opcode and its resource/queue/worker payload (left unchanged).

Allowed Dependencies

• The Cap'n Proto schema + build-time codegen (src/schema/controller.capnp → generated controller.capnp.h/.c).
• The existing request/reply plumbing (init_reply_by_req, xce_reply, opcode-only *_req senders) and client failover wrapper (XCECLI_FAILOVER_CALL, is_write=false).
• Read access to existing HA state (ha_state_t), config (myconf_t: committed_config_tag, config_sha256, peers_cfg), and persist accessors (persist_recovery_unrecoverable, persist_durable_lsn).

Revalidation Triggers

• Any change to the CtrlStatus/CtrlPeer reply shape or the ctrlstatus opcode ordinal → re-check libxce decode + the test command + any future consumer.
• Any change to the dispatch-gate role allow-lists → re-check that ctrlstatus still answers in every role and that no other opcode's gating widened.
• A change to the meaning of committed_config_tag / config_sha256 → re-check the version/checksum fields.

Architecture

Existing Architecture Analysis

• Opcode-only read precedent: clusterconf is sent with no request body (Request_body_nomsg), dispatched in cluster.c switch(request.opcode), handled by handle_clusterconf (reply.c:10), and returned via xce_reply. The client side is xce_clusterconf → xcecli_clusterconf (XCECLI_FAILOVER_CALL, is_write=false). ctrlstatus clones this pattern.
• No-alloc state-report precedent: ha_nop_reply_state_report (ha.c:3802) reads HA atomics into a stack struct and points the reply union at it without heap allocation. handle_ctrlstatus follows this model (a small bounded peer list is the only allocation).
• Dispatch gate (the critical seam): ha_dispatch_gate (ha.c:4525) is an allow-list keyed by role. The CTRL_STANDBY branch defaults to -XCE_ERR_STANDBY; CTRL_INIT/CTRL_ELECTING default to -XCE_ERR_NOT_READY; the CTRL_PRIMARY no-quorum branch allow-lists only specific reads. clusterconf is not in any standby allow-list — it is implicitly primary-only-when-serving. A status query must not inherit that; it must be explicitly allow-listed in every role branch.
• Config content checksum already exists: config_version_tag = first 8 bytes of SHA-256 over the committed config blob; full digest in config_sha256[32], round-trip-verified on load (config_repl.c).

Pattern & dependency direction

Extend-in-place along the existing request path. Dependency direction (unchanged): schema → libxce (request/reply/client) → controller handler; the test client depends on libxce. No upward dependencies introduced.

graph LR
    TestClient[xce_test ctrlstatus and wait_primary] --> Libxce[libxce xcecli_ctrlstatus]
    Libxce --> Failover[XCECLI_FAILOVER_CALL is_write false]
    Failover --> Gate[ha_dispatch_gate allow_list all roles]
    Gate --> Handler[handle_ctrlstatus snapshots local state]
    Handler --> HAState[ha_state_t and myconf_t]
    Handler --> Reply[xce_reply CtrlStatus]

File Structure Plan

Modified Files

• src/schema/controller.capnp — add OPCode ctrlstatus @40; new struct CtrlStatus $C.mapname("xce_ctrlstatus_t") and struct CtrlPeer $C.mapname("xce_ctrlpeer_t"); reply-union member ctrlstatus @17: CtrlStatus. (Build regenerates controller.capnp.h/.c → xce_ctrlstatus_t, RequestReply_data_ctrlstatus, free_CtrlStatus.)
• src/controller/reply.c — add handle_ctrlstatus (snapshot local HA/config state into a CtrlStatus reply; small peer list; xce_reply). Mirrors handle_clusterconf/ha_nop_reply_state_report.
• src/controller/cluster.c — add case OPCode_ctrlstatus: rc = handle_ctrlstatus(...) to the opcode switch.
• src/controller/ha.c — add OPCode_ctrlstatus to the ha_dispatch_gate allow-lists for CTRL_PRIMARY (incl. the no-quorum branch), CTRL_STANDBY, CTRL_ELECTING, CTRL_INIT. (the requirements-critical change)
• src/lib/request.c + src/include/request.h — add xce_ctrlstatus_req (opcode-only sender, clone of xce_clusterconf_req).
• src/lib/xceapi.c — add xce_ctrlstatus (send req, cbuf_get status, MOVE_PTR reply union).
• src/lib/client.c + src/include/client.h — add xcecli_ctrlstatus (XCECLI_FAILOVER_CALL, is_write=false), the authoritative xcecli_ctrlstatus_primary (follows redirect / rotates the master list until responderRole == PRIMARY), and the xcecli_free_ctrlstatus macro.
• src/controller/testsuite/test.c — add cmd_ctrlstatus + table registration (uses xcecli_ctrlstatus, shows the responder's view); switch cmd_wait_primary's probe (currently joblist_query(...) at the deadline-loop call site) to xcecli_ctrlstatus_primary (authoritative, primary-only success).

New Files

• None (capnp C types are generated; all handlers/APIs slot into existing TUs).

System Flows

Status query: any-node hint then authoritative-from-primary

sequenceDiagram
    participant C as Client
    participant S as Standby node
    participant P as Primary node
    C->>S: ctrlstatus request
    S-->>C: reply responderRole STANDBY plus known primary hint
    Note over C,S: Standby answer is a non authoritative hint view
    C->>P: re-query targeting the reported primary
    P-->>C: authoritative reply responderRole PRIMARY

The dispatch gate allow-lists ctrlstatus in every role (incl. no-quorum), so any node answers with its local view (satisfying any-node visibility). An authoritative answer comes only from the node whose responderRole == PRIMARY. When the first reply is from a non-primary, the client follows the redirect — it maps the reply's primaryName to an address via the reply's peers list (or rotates the configured master list) and re-queries — until it obtains the primary's authoritative reply or exhausts its budget. The plain ctrlstatus command may display the first (hint) reply with its "answered by" framing; wait-primary and any authoritative use require the primary's reply. Gate decision: ctrlstatus is allow-listed in every role branch and does not require quorum or a committed read — it returns the node's local view even when fenced or without quorum (precisely when an operator most wants it).

wait-primary probe (Req 9)

wait-primary keeps its existing monotonic-deadline poll loop and timeout/exit semantics; only the probe changes from joblist_query to xcecli_ctrlstatus. Success predicate: the status reply reports a non-"unknown" current primary. On success it prints the served primary's identity. (Rationale + caveat below.)

Requirements Traceability

Requirement	Summary	Components	Interfaces
1.1, 1.2, 1.4	Single-call read-only focused status	handle_ctrlstatus, CtrlStatus schema	opcode ctrlstatus, reply CtrlStatus
1.3	"unknown" for undeterminable fields	handle_ctrlstatus	empty Text/sentinel mapping
2.1, 2.2, 2.3	Current primary identity (unknown when unelected; standby's known primary)	handle_ctrlstatus (reads ha->primary_host under ha->lock)	CtrlStatus.primaryName
3.1, 3.2, 3.3	Answered-by node + role, distinct from primary	handle_ctrlstatus	CtrlStatus.responderName, responderRole
4.1, 4.2	Configured controllers + count	handle_ctrlstatus (reads peers_cfg)	CtrlStatus.peers, peerCount
5.1, 5.2, 5.3	Config version + content checksum (loaded pair + committed)	handle_ctrlstatus (reads loaded config_version_tag/config_sha256 + committed_config_tag)	CtrlStatus.configVersion, configChecksum, committedConfigVersion
6.1–6.4	Term, commit/apply, fenced, cluster name	handle_ctrlstatus	CtrlStatus.term, committedLsn, appliedLsn, fenced, clusterName
7.1, 7.2, 7.3	Failover-aware any-node delivery	xcecli_ctrlstatus, dispatch-gate allow-lists	XCECLI_FAILOVER_CALL (is_write=false), gate entries
8.1, 8.2, 8.3	Test-client ctrlstatus command	cmd_ctrlstatus	client command + exit code
9.1–9.4	wait-primary uses an authoritative (primary-only) status probe	cmd_wait_primary probe swap, xcecli_ctrlstatus_primary	redirect-to-primary, responderRole == PRIMARY success

Components and Interfaces

Component	Layer	Intent	Req Coverage	Key Dependencies	Contracts
CtrlStatus/CtrlPeer schema	Schema	Reply payload for the status query	2–6	capnp codegen (P0)	API (reply)
handle_ctrlstatus	Controller	Snapshot local HA/config state into the reply	1–6	ha_state_t, myconf_t, persist accessors (P0)	Service
dispatch-gate allow-lists	Controller HA	Permit ctrlstatus in every role	7	ha_dispatch_gate (P0)	Service
xcecli_ctrlstatus / xcecli_ctrlstatus_primary	libxce	Failover-aware read (any-node) + authoritative reach-primary read + free	1, 7, 9	failover wrapper, redirect hint (P0)	Service
cmd_ctrlstatus	Test client	Print the status (responder view)	8	xcecli_ctrlstatus (P0)	State
cmd_wait_primary probe	Test client	Authoritative serving-primary probe	9	xcecli_ctrlstatus_primary (P0)	State

Schema — reply payload (Cap'n Proto)

struct CtrlPeer $C.mapname("xce_ctrlpeer_t") {
  name @0: Text;
  host @1: Text;
  port @2: UInt32;
}

struct CtrlStatus $C.mapname("xce_ctrlstatus_t") {
  clusterName       @0:  Text;
  primaryName       @1:  Text;    # empty ⇒ "unknown" (no elected/known primary)
  responderName     @2:  Text;    # the node that answered
  responderRole     @3:  UInt8;   # ctrl_role_t: INIT/ELECTING/PRIMARY/STANDBY
  term              @4:  UInt64;
  # Loaded config (consistent pair: version is the 8-byte prefix of the checksum)
  configVersion     @5:  UInt64;  # loaded config_version_tag (8-byte SHA-256 prefix)
  configChecksum    @6:  Data;    # loaded config_sha256 (full 32 bytes)
  # Committed config version (may differ from loaded mid config-change)
  committedConfigVersion @7: UInt64;  # committed_config_tag
  committedLsn      @8:  UInt64;
  appliedLsn        @9:  UInt64;
  fenced            @10: Bool;    # in_catch_up OR recovery_unrecoverable
  peers             @11: List(CtrlPeer) $C.mapname("peers.list") $C.maplistcount("peers.count");
  peerCount         @12: UInt32;
}

• Config fields (Req 5): configVersion/configChecksum are the loaded config's tag and full SHA-256 — a self-consistent pair (the version is the first 8 bytes of the checksum), so two nodes with identical loaded content match on both and divergent content differs on both (5.3). committedConfigVersion is the separately-tracked committed tag, exposing "what I run" vs "what is committed"; in steady state all three agree.
• New OPCode ordinal: ctrlstatus @40 (next free after preVoteReply @39). New reply-union member: ctrlstatus @17 (next free after walackreplymsg @16). Field ordinals are append-only — never reuse/reorder.

handle_ctrlstatus — Service contract

Field	Detail
Intent	Build a CtrlStatus from the responding node's local HA/config state and reply
Requirements	1.1–1.4, 2.1–2.3, 3.1–3.3, 4.1–4.2, 5.1–5.3, 6.1–6.4

Responsibilities & Constraints

• Read-only: must not modify any HA, persist, or config state (1.2).
• Field sourcing: clusterName/responderName from c->myconf/own peerinfo; responderRole, term, appliedLsn, committedLsn, committedConfigVersion (committed_config_tag) via atomic_load_explicit(..., acquire); configVersion from the loaded config_version_tag and configChecksum from the loaded config_sha256[32] (a self-consistent pair — Req 5.3); primaryName/peers as below; fenced = in_catch_up || persist_recovery_unrecoverable(...).
• Locking: primary_host/primary_port are guarded by ha->lock — take it for a consistent snapshot, copy out, release. All other fields are atomics or immutable config.
• "unknown" mapping (1.3, 2.2, 2.3): if primary_host[0] == '\0' (no known primary), emit an empty primaryName, which the client renders as "unknown".
• Peers (4.x): iterate c->myconf->peers_cfg.candidates[0..count-1] → one CtrlPeer{name,host,port} each; set peerCount.
• Answerable in every role; returns the local view even when fenced/no-quorum.

Dependencies

• Inbound: cluster.c opcode switch — routes the request (P0).
• Outbound: ha_state_t, myconf_t, persist accessors — read-only snapshot (P0).
• Outbound: init_reply_by_req, xce_reply — reply encode/send (P0).

Dispatch-gate allow-lists — Service contract

• Add OPCode_ctrlstatus to the allow-list in ha_dispatch_gate for CTRL_PRIMARY (including the no-quorum branch), CTRL_STANDBY, CTRL_ELECTING, and CTRL_INIT (7.1, 7.2).
• Constraint: the change is scoped strictly to ctrlstatus; no other opcode's gating may widen. ctrlstatus never requires quorum or a committed read.

libxce client API — Service contract

/* opcode-only request sender (clone of xce_clusterconf_req) */
int  xce_ctrlstatus_req(endpoint_t* ep);
/* send + await reply; on success MOVE_PTR the reply union out */
int  xce_ctrlstatus(xce_ctx_t* ctx, xce_ctrlstatus_t** status);
/* failover-aware read (is_write=false): returns whatever node answers (may be a
 * standby hint view). Use for the operator command. */
int  xcecli_ctrlstatus(xcecli_ctx_t c, xce_ctrlstatus_t** status);
/* AUTHORITATIVE read: returns only a reply whose responderRole == PRIMARY. If the
 * first answer is from a non-primary, follow the redirect — target the reported
 * primaryName via the reply's peers list, else rotate the configured master list
 * — until the primary answers or the failover budget is exhausted. */
int  xcecli_ctrlstatus_primary(xcecli_ctx_t c, xce_ctrlstatus_t** status);
#define xcecli_free_ctrlstatus(s)  xce_free_ctrlstatus(s)

• Preconditions: valid client context.
• Postconditions: on success *status owns a decoded CtrlStatus (caller frees); on failure returns a typed negative (7.3) and *status untouched. For xcecli_ctrlstatus_primary, success additionally guarantees responderRole == PRIMARY (an authoritative answer).
• Invariants: is_write=false — retryable/rotatable across the configured controllers.

cmd_ctrlstatus (test client) — State contract

• Calls xcecli_ctrlstatus; on success prints the fields in human-readable form (8.1) — Primary, answered by <responderName> (<role>) distinct from Primary: (3.3), peers + count, configVersion and committedConfigVersion, configChecksum as hex, term, commit/apply, fenced, cluster name; renders empty primaryName as unknown. Returns 0 on success, <0 on failure with the error printed (8.2, 8.3). Frees the reply.

cmd_wait_primary probe swap — State contract

• Replace the single joblist_query(...) probe call in the deadline loop with xcecli_ctrlstatus_primary (the authoritative variant).
• Success predicate (9.1, 9.2): the probe obtains an authoritative reply with responderRole == PRIMARY (the answer came from the serving primary). On success, complete and print that primary's identity. A standby-only/hint answer is NOT success — the probe follows the redirect to the primary (or keeps polling) until the primary answers or the deadline elapses.
• Decision (resolves review Issue 1): require the answer to come from the primary, not merely "a primary name was reported by some node." This avoids a false success on a standby's stale primary_host after a primary death (the false-positive class removed from the durability soak) and faithfully meets Req 9.1's "serving primary." The any-node hint view remains available for the operator ctrlstatus command via xcecli_ctrlstatus.
• Preserved (9.4): the timeout-argument parsing, monotonic deadline, poll interval, log-level quieting, and success/failure exit codes are unchanged; only the probe call and the success message change.

Data Models

The only new data is the reply payload above. CtrlStatus is a flat value object (no persistence, no aggregate); CtrlPeer is a bounded value list (≤ configured controller count, typically 3). configChecksum is the loaded config's 32-byte SHA-256 and configVersion is its own 8-byte prefix (a self-consistent pair); committedConfigVersion is the separately-tracked committed tag. Two nodes with identical loaded config content produce identical configVersion + configChecksum; differing content differs in both (5.3). A node mid config-change may show configVersion != committedConfigVersion.

Error Handling

• No reachable controller (7.3): xcecli_ctrlstatus returns the failover wrapper's typed negative (e.g. no-master/timeout); cmd_ctrlstatus prints it and returns non-zero (8.3); wait-primary keeps polling until its deadline (9.3).
• Undeterminable field (1.3): emit the defined "unknown"/empty value, never uninitialized data.
• Standby with no known primary (2.3): primaryName empty → "unknown"; the call still succeeds (the status itself is valid).
• Read-only guarantee (1.2): the handler takes only read locks/atomic loads; no state mutation on any path.

Testing Strategy

Unit / handler

• handle_ctrlstatus on a PRIMARY returns primaryName==self, responderRole==PRIMARY, populated peers/version/checksum (2.1, 3.x, 4.x, 5.x).
• handle_ctrlstatus on a STANDBY with a known primary returns that primary and responderRole==STANDBY; with no known primary returns empty primaryName (2.2, 2.3).
• fenced reflects in_catch_up/recovery_unrecoverable (6.3); the handler mutates no state (1.2).
• configVersion/configChecksum are a self-consistent loaded pair — equal across two nodes with identical loaded config, differ when content differs; committedConfigVersion is exposed separately (5.1, 5.2, 5.3).

Integration

• Dispatch-gate: a ctrlstatus request is answered (not -XCE_ERR_STANDBY/-XCE_ERR_NOT_READY) on STANDBY, ELECTING, INIT, and PRIMARY-no-quorum nodes; no other opcode's gating changed (7.1, 7.2).
• libxce round-trip: xcecli_ctrlstatus decodes all fields; is_write=false rotates across configured controllers; no-reachable-controller returns a typed failure (7.3).
• Authoritative read: xcecli_ctrlstatus_primary, when the first answer is from a standby, follows the redirect to the primary and returns a reply with responderRole == PRIMARY; it never returns a standby/hint reply as the authoritative result (9.1, 9.2).

E2E (live /xce-deploy-ha)

• ctrlstatus against a healthy 3-node cluster prints the primary, peers, version+checksum; querying a standby still reports the primary with answered by <standby> (2.3, 8.1).
• wait-primary (authoritative probe) returns immediately on a healthy cluster and names the primary; after a primary kill, blocks until the new primary itself answers, then names it; does NOT false-succeed on a standby's stale primary view; times out when no primary serves (9.1–9.4). Directly useful to the ha-commit-durability Phase 6 liveness/>60s-election investigation.

Security Considerations

ctrlstatus exposes cluster topology, role, term, and config version/checksum — the same trust level as the existing clusterconf/state-report reads (no secrets, no config content, no job data). It is read-only and goes through the same authenticated client path as other client ops; no new authorization surface is introduced. Broadening the dispatch gate is scoped strictly to this one read opcode.

---

Design Delta — Requirements 10 & 11 (2026-06-20)

Merge note (as-built correction). The handler shipped as ctrlstatus_fill + handle_ctrlstatus in src/controller/ctrlstatus.c (not reply.c); configChecksum is carried as Text lowercase-hex (the Data capnp type maps to the project's structured xce_data_t); primaryName is role-derived (primary → self, else empty) after E2E fix ade0081 — ha->primary_host is vestigial. The Req 10/11 changes are written against this as-built reality; the Reqs 1–9 design above stands as the original record.

What changed and why

Live E2E (2026-06-20): a standby's ctrlstatus reported committedLsn: 0 / committedConfigVersion: 0 while appliedLsn: 38 — an apparent commit/apply inversion — because ctrlstatus_fill reads the primary-only markers persist.committed_lsn (ctrlstatus.c:130) and ha->committed_config_tag (:109), which stay 0 on a node that never led. And there was no responder timestamp to line a captured status up against ctrl-N.log. Req 10 fixes the off-primary committed sourcing; Req 11 adds a responder snapshot timestamp.

Boundary Commitments (delta)

• This spec owns (added): the off-primary field-sourcing fix inside ctrlstatus_fill (read the learned committed values when not primary); a new read-only CtrlStatus.responderTimeMs field stamped at snapshot time; its render in cmd_ctrlstatus.
• Out of Boundary (firm): making the off-primary committed value live-fresh — i.e. consuming HeartbeatMsg.committedLsn / extending OP_nop to refresh last_known_committed_lsn on every heartbeat — is a heartbeat/replication-protocol change. It violates Req 10.4 and is routed to ha-heartbeat-liveness / ha-repl-hardening. controller-status reports the honest best-known value only. (Note: superseded by ha-heartbeat-committed-propagation — that spec implemented exactly this, adding ha_consume_heartbeat_committed/ha_observe_heartbeat_committed, invoked from the primary's heartbeat in cluster.c, which now refreshes last_known_committed_lsn/last_known_committed_config_tag on every heartbeat, term-guarded and monotonic. See that spec's design.md.)
• Allowed Dependencies (added): read access to ha->last_known_committed_lsn (_Atomic uint64_t, ha.h:183) and ha->last_known_committed_config_tag (ha.h:185); clock_gettime(CLOCK_REALTIME); the existing xcelog.h wall-clock format for client rendering. (Note: last_known_committed_config_tag is now also _Atomic uint64_t — migrated by ha-heartbeat-committed-propagation from the plain uint64_t this design was written against.)
• Revalidation Triggers (added): any change to where/when last_known_committed_lsn/last_known_committed_config_tag are written (today: cluster_apply_ctrlhello_reply at join/RDHUP, ha_observe_heartbeat_committed on every heartbeat, and the election safe-commit path) → re-check the off-primary committed reporting and the staleness caveat. (Note: the heartbeat writer was added by ha-heartbeat-committed-propagation, superseding the "join/RDHUP only" premise below.)

Locked design decisions (from research items)

1. Req 10 sourcing → role-branch (not uniform max()). When responderRole == CTRL_PRIMARY, keep the node's own authoritative live markers (persist.committed_lsn, ha->committed_config_tag). Otherwise report the learned pair (ha->last_known_committed_lsn, ha->last_known_committed_config_tag). Rationale: a standby's learned value is its only honest "committed" knowledge; a primary's own marker is authoritative and must not be shadowed by a possibly-stale learned value. (Mirrors the last_known_* read precedent at ha.c:5319 (ha_vote_self_freshness), ha.c:6732.)
2. Req 10 staleness caveat (documented behavior, not a bug). last_known_committed_lsn is refreshed at CtrlHello join / RDHUP-failover, not per heartbeat, so an off-primary committed value MAY lag the live primary between joins. This is the in-boundary contract; live-fresh is the out-of-boundary item above. The 0-inversion is eliminated regardless (a standby that has joined has a non-zero learned value; a never-joined node correctly reports 0 = unknown, per Req 1.3 / 10.2). (Note: superseded by ha-heartbeat-committed-propagation — last_known_committed_lsn/_config_tag are now also refreshed on every heartbeat via ha_observe_heartbeat_committed, so the join/RDHUP-only staleness window described here no longer applies; see that spec's design.md.)
3. Req 11 wire type → UInt64 epoch-milliseconds. Numeric, locale-safe; the client formats it. (Pre-formatted Text was the alternative; rejected to keep the wire value machine-comparable and avoid encoding a locale into the reply.)
4. Req 11 clock → CLOCK_REALTIME, stamped in ctrlstatus_fill at snapshot time. NOT CLOCK_MONOTONIC (which ha_on_heartbeat_received uses for liveness) — monotonic is not wall-clock and cannot be correlated to logs. The client renders the millis to local YYYY-MM-DD HH:MM:SS,mmm using the xcelog.h convention so it is directly comparable to a log line.

Research item carried to implementation

• Confirm the exact wall-clock formatter/helper in src/include/xcelog.h and reuse it (or match its YYYY-MM-DD HH:MM:SS,mmm form) in cmd_ctrlstatus so the printed timestamp is byte-comparable to ctrl-N.log. last_known_committed_config_tag was non-atomic at design time (written under lock in ha_convergence.c); the status read was planned as a best-effort aligned 8-byte load (acceptable for a reporting snapshot) — do not add locking on the read path. (Note: superseded by ha-heartbeat-committed-propagation, which migrated the field to _Atomic uint64_t at ha.h:185, written via atomic_store_explicit at ha_convergence.c:368 and ha.c:9011; the as-built ctrlstatus_fill correctly reads it with atomic_load_explicit, not a bare aligned load.)

Schema delta (Cap'n Proto)

Append one field to CtrlStatus (next free ordinal @13, after peerCount @12; append-only, never reorder):

  responderTimeMs   @13: UInt64 $C.mapname("responder_time_ms");  # CLOCK_REALTIME epoch millis at snapshot (Req 11.1)

No change to committedLsn @8 / committedConfigVersion @7 wire shape — Req 10 changes only the value source, not the field.

ctrlstatus_fill delta — Service contract (src/controller/ctrlstatus.c)

• Req 10 (committed sourcing): replace the unconditional reads at ctrlstatus.c:108–110,130 with the role-branch above:
  • responderRole == CTRL_PRIMARY → committedLsn = persist.committed_lsn, committedConfigVersion = ha->committed_config_tag (atomic acquire loads, unchanged).
  • else → committedLsn = ha->last_known_committed_lsn (atomic acquire), committedConfigVersion = ha->last_known_committed_config_tag (plain aligned load).
  • appliedLsn (ha->applied_lsn) is unchanged — it is already correct in every role.
• Req 11 (timestamp): stamp st->responder_time_ms from clock_gettime(CLOCK_REALTIME) (tv_sec*1000 + tv_nsec/1e6) at the top of the snapshot, before/with the other field loads, so it marks the responder's sample moment.
• Read-only invariant (10.4, 1.2) preserved: all additions are atomic/aligned loads + a clock read; no HA/persist/config mutation.

cmd_ctrlstatus delta — State contract (src/controller/testsuite/test.c)

• Print one added line rendering responder_time_ms as local YYYY-MM-DD HH:MM:SS,mmm (xcelog.h form) alongside the existing fields (Req 11.3). The committed lines need no rendering change — Req 10 only changes the values they already print.

File Structure Plan (delta)

Modified files:

• src/schema/controller.capnp — add responderTimeMs @13: UInt64 to CtrlStatus (regen controller.capnp.h/.c → xce_ctrlstatus_t.responder_time_ms).
• src/controller/ctrlstatus.c — ctrlstatus_fill: role-branch committed sourcing (Req 10) + CLOCK_REALTIME timestamp stamp (Req 11).
• src/controller/testsuite/test.c — cmd_ctrlstatus: render the responder timestamp line (Req 11.3).
• src/controller/testsuite/test_ctrlstatus.c — add/extend unit tests (below).

No new files (the capnp field is generated; all logic slots into existing TUs).

Requirements Traceability (delta)

Requirement	Summary	Components	Interfaces
10.1	Off-primary reports learned committed LSN + config version	ctrlstatus_fill role-branch	CtrlStatus.committedLsn, committedConfigVersion (value source = ha->last_known_committed_*)
10.2	No learned value yet → 0/unknown	ctrlstatus_fill	last_known_committed_lsn == 0 (correct unknown, per 1.3)
10.3	No spurious applied>committed inversion	ctrlstatus_fill	appliedLsn vs role-correct committedLsn
10.4	Read-only reporting; no production/replication change	ctrlstatus_fill; out-of-boundary routing	atomic/aligned loads only
11.1	Responder wall-clock at sample time	ctrlstatus_fill (CLOCK_REALTIME)	CtrlStatus.responderTimeMs
11.2	Same clock/format as logs	cmd_ctrlstatus render (xcelog.h form)	local YYYY-MM-DD HH:MM:SS,mmm
11.3	Test client prints the timestamp	cmd_ctrlstatus	added print line

Testing Strategy (delta)

Unit (test_ctrlstatus.c, TDD red→green):

• STANDBY with last_known_committed_lsn = 36, persist.committed_lsn = 0, applied_lsn = 38 → reply reports committedLsn == 36 (not 0) and appliedLsn == 38 (no inversion below the learned value) (10.1, 10.3).
• STANDBY pre-join (last_known_committed_lsn == 0) → reply reports committedLsn == 0 / committedConfigVersion == 0 (correct unknown) (10.2).
• PRIMARY → reply reports committedLsn == persist.committed_lsn and committedConfigVersion == ha->committed_config_tag (authoritative own markers unchanged) (10.1 primary branch).
• committedConfigVersion off-primary equals last_known_committed_config_tag (symmetry with the LSN) (10.1).
• responder_time_ms is non-zero and within a tolerance of a test-captured CLOCK_REALTIME sample; the handler still mutates no state (11.1, 1.2/10.4).

Integration / round-trip (libxce): xcecli_ctrlstatus decodes the new responderTimeMs field end-to-end (no manual decode — codegen), proving wire compatibility.

E2E (live /xce-deploy-ha): reproduce the original finding — submit jobs so the primary commits, query a standby; assert the standby's committedLsn is non-zero and tracks (within the join-refresh caveat) rather than 0, and that the printed responder timestamp matches the standby's own ctrl-N.log line for that query. (Note: the "join-refresh caveat" is narrower than described above — ha-heartbeat-committed-propagation added per-heartbeat refresh, so tracking is now near-live rather than join/RDHUP-only; see that spec's design.md.)

---

Design Delta — Requirement 12 (2026-09-15)

What changed and why

User enhancement request: ctrlstatus should additionally report the term in which the current primary was elected and the wall-clock time it was elected, so operators can tell how long the current primary has served and correlate its election with the logs — with the same off-primary (standby) fidelity Req 10 already established for committed position, not a bare "unknown" off-primary. Scope of this delta = Req 12 only. Reqs 1–11 are implemented and unchanged.

Boundary Commitments (delta)

• This spec owns (added): a new ha_state_t.last_known_elected_at_ms field (_Atomic uint64_t, declared alongside last_known_committed_lsn/last_known_committed_config_tag in ha.h), self-stamped by BOTH promotion roads at the exact point each commits to CTRL_PRIMARY; a new HeartbeatMsg.electedAtMs wire field the primary echoes on every outgoing heartbeat; the receiving standby's unconditional store of that value into its own last_known_elected_at_ms; a new CtrlStatus.electedAtMs reply field; ctrlstatus_fill populating it (one unconditional read, no role-branch — see Locked Decision 2); cmd_ctrlstatus's two new printed lines (electedTerm, electedAt), gated on the same primaryName-known/unknown derivation Req 2 already computes.
• Out of Boundary (firm): electedTerm gets no new field — it reuses the existing CtrlStatus.term (Req 6.1) with CLI-side "unknown" gating (Locked Decision 1); this spec does not change how elections are conducted, how ha->term advances, or how/when ha->role transitions — every stamp site added here is purely observational at an already-decided promotion instant (Req 12.5).
• Allowed Dependencies (added): ha->term and the existing primaryName derivation (Req 2/3, unchanged); clock_gettime(CLOCK_REALTIME) (existing precedent, Req 11); ctrlstatus_format_time_ms (src/include/xce/ctrlstatus_fmt.h) reused verbatim for rendering — no new formatting code.
• Revalidation Triggers (added): any change to where ha->role finally becomes CTRL_PRIMARY on either promotion road (ha_do_promote or raft_cb_become_leader) → re-check the last_known_elected_at_ms stamp site is still past every refusal/eligibility gate; any change to the HeartbeatMsg/heartbeat_msg_t shape → re-check the electedAtMs ordinal and decode; any change to the primaryName-unknown derivation (Req 2) → re-check the electedTerm/electedAtMs "unknown" gating that reuses it.

Locked design decisions

1. electedTerm reuses ha->term (Req 6.1's existing wire field) — no new state. This codebase's Raft term already propagates correctly to every node via the existing heartbeat/vote term-sync path; there is no "stale/zero" class of bug here the way Req 10 had for committedLsn (which read an unrelated primary-only marker, persist.committed_lsn, that stays 0 off-primary by construction). cmd_ctrlstatus derives the displayed electedTerm as primaryName known ? term : unknown — purely a print-layer decision, no handler change for this half of the requirement. Accepted trade-off: in the narrow window where a standby has just voted for a new candidate (bumping its own ha->term) before its voted_for_name-derived primaryName reflects that same candidate, the displayed electedTerm could transiently be a half-step out of step with the displayed primary — the same class of transient standby-hint staleness already accepted for the original primaryName/term fields (see Architecture's "any-node hint then authoritative-from-primary" flow) and for Req 10's own documented staleness caveat. Not a regression, not new risk surface.
2. electedAtMs is genuinely new state, written unconditionally (no monotonic/term compare on write). Both promotion roads write last_known_elected_at_ms exactly once, at the instant each road COMMITS to CTRL_PRIMARY — i.e., past every refusal/eligibility gate, never at function entry (see Research Items below for the exact confirm-at-implementation lines). The heartbeat-receive site writes it only when already processing a heartbeat from the node's currently-recognized leader — the same precondition ha_observe_heartbeat_committed's existing call site already relies on for committed_lsn/committed_config_tag. Because every write site is already gated by an existing freshness precondition, no second "which term does my stored value correspond to" comparison is needed. Simplification vs. Req 10's pattern: since there is no pre-existing primary-only marker analogous to persist.committed_lsn for election time, ctrlstatus_fill needs no role-branch for this field — one unconditional read of last_known_elected_at_ms correctly serves both the primary (self-stamped) and a standby (heartbeat-learned) case.
3. Unknown gating happens at output time, not by proactively zeroing state. Both electedTerm and electedAtMs are gated in ctrlstatus_fill/cmd_ctrlstatus on the SAME primaryName-known/unknown derivation Req 2.2/2.3 already computes. A stale last_known_elected_at_ms left over from a since-lost primary is simply never surfaced once primaryName itself reports "unknown" — mirrors how the original design already treats a stale voted_for_name/primary_host.
4. Wire type: electedAtMs @14: UInt64 epoch-milliseconds — mirrors Req 11's responderTimeMs numeric-not-Text precedent exactly (machine-comparable; the client formats it via the same existing ctrlstatus_format_time_ms).

Research items carried to implementation

1. Confirm the exact statement in raft_cb_become_leader (ha.c:1615) where ha->role is finally stored as CTRL_PRIMARY — past the ha_promotion_gate_decision eligibility gate (later in the function than this delta's own discovery reached) — and stamp last_known_elected_at_ms there. Do not stamp at the earlier last_known_committed_lsn self-observe point (ha.c:1665, inside the if (promo_ectx) block): that point precedes the eligibility gate and can still be followed by a promotion refusal, which must not produce a bogus election time.
2. Confirm which of ha_observe_heartbeat_committed's 9 existing call sites corresponds to processing a genuinely RECEIVED heartbeat (as opposed to a local/test-only invocation) and add the unconditional last_known_elected_at_ms store immediately alongside that call, reading the new electedAtMs field off the same decoded heartbeat_msg_t. Also confirm the outgoing side — where primary_heartbeat_fn (ha.c:9791) populates the heartbeat_msg_t it sends via xce_ha_heartbeat_req — and set electedAtMs = last_known_elected_at_ms there.
3. Confirm ha_do_promote (ha.c:2927, the function ha_promote_to_primary's own comment describes as owning "the term increment + sets role=PRIMARY" for the non-raft road) is the correct single stamp site for that road; its body was outside this delta's discovery window.

Constraints checked

• Async I/O invariant (ha-async-io.md): every write added here is an atomic store or a clock_gettime(CLOCK_REALTIME) call on the existing promotion/heartbeat control-flow paths — no new connect(), file I/O, or other blocking call is introduced; identical shape to Req 11's already-accepted CLOCK_REALTIME stamp.
• Persistence / save-restore completeness: not applicable — last_known_elected_at_ms is ephemeral in-memory HA state, never written to ha_term_file_t/WAL/any durable format, and is fully re-derived on the next promotion or heartbeat exactly like its sibling last_known_committed_lsn/last_known_committed_config_tag (also not persisted). No save/restore path is added or modified by this delta.

Schema delta (Cap'n Proto)

# CtrlStatus — next free ordinal @14 (after responderTimeMs @13; append-only, never reorder)
  electedAtMs   @14: UInt64 $C.mapname("elected_at_ms");
  # epoch millis of the current primary's election, learned off-primary via
  # heartbeat; 0/unset when no primary is known (gated together with primaryName)

# HeartbeatMsg — next free ordinal @4 (after committedConfigTag @3)
  electedAtMs   @4: UInt64 $C.mapname("elected_at_ms");
  # primary's own last_known_elected_at_ms, echoed on every heartbeat (Req 12.2)

No change to term @4 on CtrlStatus — Req 12's term half changes only the CLI's derived label/gating, not the wire field or its value.

File Structure Plan (delta)

Modified files:

• ha.h — add _Atomic uint64_t last_known_elected_at_ms; to ha_state_t, next to last_known_committed_lsn/last_known_committed_config_tag.
• src/schema/controller.capnp — add electedAtMs to CtrlStatus (@14) and to HeartbeatMsg (@4); regen controller.capnp.h/.c.
• src/controller/ha.c — ha_do_promote and raft_cb_become_leader: stamp last_known_elected_at_ms from CLOCK_REALTIME at each road's final promotion-commit point (Research Items 1, 3); the heartbeat-receive call site: unconditional store alongside the existing ha_observe_heartbeat_committed call (Research Item 2); primary_heartbeat_fn: populate the outgoing heartbeat_msg_t.elected_at_ms from last_known_elected_at_ms (Research Item 2).
• src/controller/ctrlstatus.c — ctrlstatus_fill: read last_known_elected_at_ms unconditionally into st->elected_at_ms (no role-branch, Locked Decision 2).
• src/harness/test.c — cmd_ctrlstatus: two new printed lines, electedTerm (derived from term+primaryName) and electedAt (rendered via ctrlstatus_format_time_ms), both "unknown" when primaryName is unknown. Path correction (as-built, confirmed 2026-09-15): cmd_ctrlstatus/cmd_wait_primary now live in src/harness/test.c, not src/controller/testsuite/test.c as Reqs 1-11's original design text says — the test-client harness was consolidated into src/harness/ by a later refactor (see the xce-harness-component spec) after those sections were written. Historical Reqs 1-11 text is left as-is (accurate as of when implemented); this delta targets the current, correct location.
• src/controller/testsuite/test_ctrlstatus.c — extend/add unit tests (below).
• src/controller/testsuite/test_ha_heartbeat_committed.c (or a new sibling test file) — coverage for the heartbeat-carried electedAtMs store.
• src/controller/testsuite/test_ha_role.c — extend test_promote_increments_term-style coverage to also assert last_known_elected_at_ms is stamped on promotion.

No new files (capnp fields are generated; all logic slots into existing TUs) — consistent with this spec's established pattern.

Requirements Traceability (delta)

Requirement	Summary	Components	Interfaces
12.1	Report the primary's election term + time alongside its identity	ctrlstatus_fill (electedAtMs read), cmd_ctrlstatus (electedTerm derived from term+primaryName)	CtrlStatus.term (reused), CtrlStatus.electedAtMs (new)
12.2	Standby reports the LEARNED election term/time, not stale/unset	heartbeat-carried electedAtMs; existing term-sync for electedTerm	HeartbeatMsg.electedAtMs, unconditional store into last_known_elected_at_ms
12.3	Unknown when no primary is known	ctrlstatus_fill/cmd_ctrlstatus gating on primaryName	shares Req 2.2/2.3's existing "unknown" derivation
12.4	Election time on the same clock/format as logs	ctrlstatus_fill (CLOCK_REALTIME stamp), cmd_ctrlstatus render	reuses ctrlstatus_format_time_ms (Req 11.2 precedent)
12.5	Read-only; no election/leadership behavior change	stamp sites are observational only; no gating logic altered	n/a
12.6	Test client prints both fields	cmd_ctrlstatus	two new printed lines

ctrlstatus_fill / promotion / heartbeat delta — Service contract

• Req 12.1/12.2 (sourcing): st->elected_at_ms = ha->last_known_elected_at_ms (atomic acquire load), unconditionally — no role-branch (Locked Decision 2).
• Req 12.5 (read-only invariant) preserved: the handler itself performs only atomic loads, exactly as every other field; all writes to last_known_elected_at_ms happen upstream, at promotion or heartbeat-processing time, never inside ctrlstatus_fill.
• cmd_ctrlstatus (Req 12.1, 12.3, 12.6): print

  electedTerm: <term>          (or "unknown" when primaryName is unknown)
  electedAt: <ctrlstatus_format_time_ms(elected_at_ms)>   (or "unknown")
  

  placed after the existing Primary: line, grouping the primary's identity with its election term/time.

Testing Strategy (delta)

Unit (test_ctrlstatus.c):

• PRIMARY: ctrlstatus_fill reports elected_at_ms == ha->last_known_elected_at_ms (self-stamped); CLI-derived electedTerm == st.term.
• STANDBY with a learned last_known_elected_at_ms (simulating an already-processed heartbeat) and a known primaryName → reports the learned value, not 0/stale (12.2).
• STANDBY with primaryName unknown (no vote / self-vote / electing) → CLI renders electedTerm/electedAt as "unknown" regardless of any stale stored last_known_elected_at_ms (12.3).
• Extend the existing test_ctrlstatus_no_mutation snapshot/compare to also cover last_known_elected_at_ms (12.5).

New/extended heartbeat unit test (sibling to test_ha_heartbeat_committed.c):

• Processing a received heartbeat with a given electedAtMs unconditionally overwrites last_known_elected_at_ms and leaves last_known_committed_lsn/last_known_committed_config_tag/role/term/applied untouched — reconcile-free, mirrors test_observe_no_reconcile_no_other_mutation (12.2, 12.5).

Integration:

• Extend test_ha_role.c's test_promote_increments_term-style coverage (non-raft road, ha_do_promote) to also assert last_known_elected_at_ms is stamped on promotion.
• Add/extend a raft_cb_become_leader promotion test to assert the same for the raft road, specifically confirming NO stamp occurs on a refused/gated promotion attempt (Research Item 1).
• Wire round-trip: xcecli_ctrlstatus decodes the new CtrlStatus.electedAtMs field end-to-end (codegen, no manual decode).

E2E (live /xce-deploy-ha):

• Query ctrlstatus on the primary right after a fresh election: electedTerm equals the printed term; electedAt is close to "now."
• Query a STANDBY: electedTerm/electedAt match the primary's own values (within one heartbeat interval), not "unknown" and not stale/zero — proving standby learned fidelity end-to-end (12.2).
• Kill the primary, wait for a new election, query the surviving standby immediately vs. after one heartbeat interval — demonstrates the field converges promptly via heartbeat, unlike Req 10's original join/RDHUP-only caveat (heartbeat-carried delivery here is in-boundary from day one of this delta, not a follow-up spec).