myxce/.kiro/specs/controller-status/design.md

# Design Document — controller-status (`ctrlstatus`)

## Overview

**Purpose:** add a single, cheap, read-only client→controller query — `ctrlstatus`, an XCE analog of LSF `lsid` — that returns basic cluster/controller identity and health: who is the current primary, the answering node's identity and role, the configured controllers (peers), the committed configuration version and content checksum, and a few HA health/progress fields (term, commit/apply position, fenced state, cluster name).

**Users:** operators inspecting a running cluster, and test/E2E authors orchestrating failover (including `wait-primary`, which switches its probe to this query).

**Impact:** today there is no status query — callers infer "is there a serving primary" indirectly (e.g. by listing jobs). This adds a new opcode by cloning the existing **opcode-only read** pattern of `clusterconf`, reads only already-maintained HA/config state (no behavior change), and crucially makes the query answerable on **any** node by adding it to the HA dispatch-gate allow-lists.

### Goals
- A new read-only opcode returning the status fields of Requirements 2–6 in one request/response (1.1, 1.4).
- Answerable on **any** node — a standby reports a hint view (the primary it knows); an **authoritative** answer comes only from the primary, reached by following the redirect / rotating the master list (2.3, 7.1–7.3).
- Report the loaded config version + content checksum (and the committed version) so same-version divergence is detectable (5.2, 5.3).
- A `ctrlstatus` test-client command (8.x) and a `wait-primary` probe switched to an authoritative (primary-only) status query (9.x).

### Non-Goals
- No operator-facing CLI/REST `lsid` surface (Go tier) — later pass.
- No change to HA election, replication, or commit behavior — the query only **reads** existing state.
- No new configuration checksum computation — the committed-config SHA-256 already exists; this only exposes it.

## Boundary Commitments

### This Spec Owns
- A new Cap'n Proto opcode `ctrlstatus` and its reply payload (`CtrlStatus`, `CtrlPeer`).
- The controller handler `handle_ctrlstatus` that snapshots local HA/config state into the reply.
- The **dispatch-gate allow-list entries** that let `ctrlstatus` answer in every controller role (incl. no-quorum).
- The libxce client API (`xce_ctrlstatus_req` / `xce_ctrlstatus` / `xcecli_ctrlstatus` + free helper).
- The `ctrlstatus` test-client command and the `wait-primary` probe switch.

### Out of Boundary
- The Go CLI/REST `lsid` surface (later spec).
- Any change to how HA state (role, term, primary identity, committed config tag, commit/apply LSN, fence) is **produced or maintained** — this spec only reads it.
- The existing `clusterconf` opcode and its resource/queue/worker payload (left unchanged).

### Allowed Dependencies
- The Cap'n Proto schema + build-time codegen (`src/schema/controller.capnp` → generated `controller.capnp.h/.c`).
- The existing request/reply plumbing (`init_reply_by_req`, `xce_reply`, opcode-only `*_req` senders) and client failover wrapper (`XCECLI_FAILOVER_CALL`, `is_write=false`).
- Read access to existing HA state (`ha_state_t`), config (`myconf_t`: `committed_config_tag`, `config_sha256`, `peers_cfg`), and persist accessors (`persist_recovery_unrecoverable`, `persist_durable_lsn`).

### Revalidation Triggers
- Any change to the `CtrlStatus`/`CtrlPeer` reply shape or the `ctrlstatus` opcode ordinal → re-check libxce decode + the test command + any future consumer.
- Any change to the dispatch-gate role allow-lists → re-check that `ctrlstatus` still answers in every role and that no other opcode's gating widened.
- A change to the meaning of `committed_config_tag` / `config_sha256` → re-check the version/checksum fields.

## Architecture

### Existing Architecture Analysis
- **Opcode-only read precedent:** `clusterconf` is sent with no request body (`Request_body_nomsg`), dispatched in `cluster.c` `switch(request.opcode)`, handled by `handle_clusterconf` (`reply.c:10`), and returned via `xce_reply`. The client side is `xce_clusterconf` → `xcecli_clusterconf` (`XCECLI_FAILOVER_CALL`, `is_write=false`). **`ctrlstatus` clones this pattern.**
- **No-alloc state-report precedent:** `ha_nop_reply_state_report` (`ha.c:3802`) reads HA atomics into a stack struct and points the reply union at it without heap allocation. `handle_ctrlstatus` follows this model (a small bounded peer list is the only allocation).
- **Dispatch gate (the critical seam):** `ha_dispatch_gate` (`ha.c:4525`) is an allow-list keyed by role. The `CTRL_STANDBY` branch defaults to `-XCE_ERR_STANDBY`; `CTRL_INIT`/`CTRL_ELECTING` default to `-XCE_ERR_NOT_READY`; the `CTRL_PRIMARY` no-quorum branch allow-lists only specific reads. `clusterconf` is **not** in any standby allow-list — it is implicitly primary-only-when-serving. A status query must **not** inherit that; it must be explicitly allow-listed in every role branch.
- **Config content checksum already exists:** `config_version_tag` = first 8 bytes of SHA-256 over the committed config blob; full digest in `config_sha256[32]`, round-trip-verified on load (`config_repl.c`).

### Pattern & dependency direction
Extend-in-place along the existing request path. Dependency direction (unchanged): `schema → libxce (request/reply/client) → controller handler`; the test client depends on libxce. No upward dependencies introduced.

```mermaid
graph LR
    TestClient[xce_test ctrlstatus and wait_primary] --> Libxce[libxce xcecli_ctrlstatus]
    Libxce --> Failover[XCECLI_FAILOVER_CALL is_write false]
    Failover --> Gate[ha_dispatch_gate allow_list all roles]
    Gate --> Handler[handle_ctrlstatus snapshots local state]
    Handler --> HAState[ha_state_t and myconf_t]
    Handler --> Reply[xce_reply CtrlStatus]
```

## File Structure Plan

### Modified Files
- `src/schema/controller.capnp` — add `OPCode ctrlstatus @40`; new `struct CtrlStatus $C.mapname("xce_ctrlstatus_t")` and `struct CtrlPeer $C.mapname("xce_ctrlpeer_t")`; reply-union member `ctrlstatus @17: CtrlStatus`. (Build regenerates `controller.capnp.h/.c` → `xce_ctrlstatus_t`, `RequestReply_data_ctrlstatus`, `free_CtrlStatus`.)
- `src/controller/reply.c` — add `handle_ctrlstatus` (snapshot local HA/config state into a `CtrlStatus` reply; small peer list; `xce_reply`). Mirrors `handle_clusterconf`/`ha_nop_reply_state_report`.
- `src/controller/cluster.c` — add `case OPCode_ctrlstatus: rc = handle_ctrlstatus(...)` to the opcode switch.
- `src/controller/ha.c` — add `OPCode_ctrlstatus` to the `ha_dispatch_gate` allow-lists for `CTRL_PRIMARY` (incl. the no-quorum branch), `CTRL_STANDBY`, `CTRL_ELECTING`, `CTRL_INIT`. **(the requirements-critical change)**
- `src/lib/request.c` + `src/include/request.h` — add `xce_ctrlstatus_req` (opcode-only sender, clone of `xce_clusterconf_req`).
- `src/lib/xceapi.c` — add `xce_ctrlstatus` (send req, `cbuf_get` status, `MOVE_PTR` reply union).
- `src/lib/client.c` + `src/include/client.h` — add `xcecli_ctrlstatus` (`XCECLI_FAILOVER_CALL`, `is_write=false`), the authoritative `xcecli_ctrlstatus_primary` (follows redirect / rotates the master list until `responderRole == PRIMARY`), and the `xcecli_free_ctrlstatus` macro.
- `src/controller/testsuite/test.c` — add `cmd_ctrlstatus` + table registration (uses `xcecli_ctrlstatus`, shows the responder's view); switch `cmd_wait_primary`'s probe (currently `joblist_query(...)` at the deadline-loop call site) to `xcecli_ctrlstatus_primary` (authoritative, primary-only success).

### New Files
- None (capnp C types are generated; all handlers/APIs slot into existing TUs).

## System Flows

### Status query: any-node hint then authoritative-from-primary
```mermaid
sequenceDiagram
    participant C as Client
    participant S as Standby node
    participant P as Primary node
    C->>S: ctrlstatus request
    S-->>C: reply responderRole STANDBY plus known primary hint
    Note over C,S: Standby answer is a non authoritative hint view
    C->>P: re-query targeting the reported primary
    P-->>C: authoritative reply responderRole PRIMARY
```
The dispatch gate allow-lists `ctrlstatus` in every role (incl. no-quorum), so any node answers with its local view (satisfying any-node visibility). An **authoritative** answer comes only from the node whose `responderRole == PRIMARY`. When the first reply is from a non-primary, the client follows the redirect — it maps the reply's `primaryName` to an address via the reply's `peers` list (or rotates the configured master list) and re-queries — until it obtains the primary's authoritative reply or exhausts its budget. The plain `ctrlstatus` command may display the first (hint) reply with its "answered by" framing; `wait-primary` and any authoritative use require the primary's reply.
Gate decision: `ctrlstatus` is allow-listed in every role branch and does not require quorum or a committed read — it returns the node's local view even when fenced or without quorum (precisely when an operator most wants it).

### wait-primary probe (Req 9)
`wait-primary` keeps its existing monotonic-deadline poll loop and timeout/exit semantics; only the probe changes from `joblist_query` to `xcecli_ctrlstatus`. Success predicate: the status reply reports a **non-`"unknown"`** current primary. On success it prints the served primary's identity. (Rationale + caveat below.)

## Requirements Traceability

| Requirement | Summary | Components | Interfaces |
|-------------|---------|------------|------------|
| 1.1, 1.2, 1.4 | Single-call read-only focused status | `handle_ctrlstatus`, `CtrlStatus` schema | opcode `ctrlstatus`, reply `CtrlStatus` |
| 1.3 | "unknown" for undeterminable fields | `handle_ctrlstatus` | empty Text/sentinel mapping |
| 2.1, 2.2, 2.3 | Current primary identity (unknown when unelected; standby's known primary) | `handle_ctrlstatus` (reads `ha->primary_host` under `ha->lock`) | `CtrlStatus.primaryName` |
| 3.1, 3.2, 3.3 | Answered-by node + role, distinct from primary | `handle_ctrlstatus` | `CtrlStatus.responderName`, `responderRole` |
| 4.1, 4.2 | Configured controllers + count | `handle_ctrlstatus` (reads `peers_cfg`) | `CtrlStatus.peers`, `peerCount` |
| 5.1, 5.2, 5.3 | Config version + content checksum (loaded pair + committed) | `handle_ctrlstatus` (reads loaded `config_version_tag`/`config_sha256` + `committed_config_tag`) | `CtrlStatus.configVersion`, `configChecksum`, `committedConfigVersion` |
| 6.1–6.4 | Term, commit/apply, fenced, cluster name | `handle_ctrlstatus` | `CtrlStatus.term`, `committedLsn`, `appliedLsn`, `fenced`, `clusterName` |
| 7.1, 7.2, 7.3 | Failover-aware any-node delivery | `xcecli_ctrlstatus`, dispatch-gate allow-lists | `XCECLI_FAILOVER_CALL` (is_write=false), gate entries |
| 8.1, 8.2, 8.3 | Test-client `ctrlstatus` command | `cmd_ctrlstatus` | client command + exit code |
| 9.1–9.4 | wait-primary uses an authoritative (primary-only) status probe | `cmd_wait_primary` probe swap, `xcecli_ctrlstatus_primary` | redirect-to-primary, `responderRole == PRIMARY` success |

## Components and Interfaces

| Component | Layer | Intent | Req Coverage | Key Dependencies | Contracts |
|-----------|-------|--------|--------------|------------------|-----------|
| `CtrlStatus`/`CtrlPeer` schema | Schema | Reply payload for the status query | 2–6 | capnp codegen (P0) | API (reply) |
| `handle_ctrlstatus` | Controller | Snapshot local HA/config state into the reply | 1–6 | `ha_state_t`, `myconf_t`, persist accessors (P0) | Service |
| dispatch-gate allow-lists | Controller HA | Permit `ctrlstatus` in every role | 7 | `ha_dispatch_gate` (P0) | Service |
| `xcecli_ctrlstatus` / `xcecli_ctrlstatus_primary` | libxce | Failover-aware read (any-node) + authoritative reach-primary read + free | 1, 7, 9 | failover wrapper, redirect hint (P0) | Service |
| `cmd_ctrlstatus` | Test client | Print the status (responder view) | 8 | `xcecli_ctrlstatus` (P0) | State |
| `cmd_wait_primary` probe | Test client | Authoritative serving-primary probe | 9 | `xcecli_ctrlstatus_primary` (P0) | State |

### Schema — reply payload (Cap'n Proto)
```capnp
struct CtrlPeer $C.mapname("xce_ctrlpeer_t") {
  name @0: Text;
  host @1: Text;
  port @2: UInt32;
}

struct CtrlStatus $C.mapname("xce_ctrlstatus_t") {
  clusterName       @0:  Text;
  primaryName       @1:  Text;    # empty ⇒ "unknown" (no elected/known primary)
  responderName     @2:  Text;    # the node that answered
  responderRole     @3:  UInt8;   # ctrl_role_t: INIT/ELECTING/PRIMARY/STANDBY
  term              @4:  UInt64;
  # Loaded config (consistent pair: version is the 8-byte prefix of the checksum)
  configVersion     @5:  UInt64;  # loaded config_version_tag (8-byte SHA-256 prefix)
  configChecksum    @6:  Data;    # loaded config_sha256 (full 32 bytes)
  # Committed config version (may differ from loaded mid config-change)
  committedConfigVersion @7: UInt64;  # committed_config_tag
  committedLsn      @8:  UInt64;
  appliedLsn        @9:  UInt64;
  fenced            @10: Bool;    # in_catch_up OR recovery_unrecoverable
  peers             @11: List(CtrlPeer) $C.mapname("peers.list") $C.maplistcount("peers.count");
  peerCount         @12: UInt32;
}
```
- **Config fields (Req 5):** `configVersion`/`configChecksum` are the **loaded** config's tag and full SHA-256 — a self-consistent pair (the version is the first 8 bytes of the checksum), so two nodes with identical loaded content match on both and divergent content differs on both (5.3). `committedConfigVersion` is the separately-tracked committed tag, exposing "what I run" vs "what is committed"; in steady state all three agree.
- New OPCode ordinal: `ctrlstatus @40` (next free after `preVoteReply @39`). New reply-union member: `ctrlstatus @17` (next free after `walackreplymsg @16`). Field ordinals are append-only — never reuse/reorder.

#### `handle_ctrlstatus` — Service contract
| Field | Detail |
|-------|--------|
| Intent | Build a `CtrlStatus` from the responding node's local HA/config state and reply |
| Requirements | 1.1–1.4, 2.1–2.3, 3.1–3.3, 4.1–4.2, 5.1–5.3, 6.1–6.4 |

**Responsibilities & Constraints**
- Read-only: must not modify any HA, persist, or config state (1.2).
- Field sourcing: `clusterName`/`responderName` from `c->myconf`/own peerinfo; `responderRole`, `term`, `appliedLsn`, `committedLsn`, `committedConfigVersion` (`committed_config_tag`) via `atomic_load_explicit(..., acquire)`; `configVersion` from the loaded `config_version_tag` and `configChecksum` from the loaded `config_sha256[32]` (a self-consistent pair — Req 5.3); `primaryName`/peers as below; `fenced = in_catch_up || persist_recovery_unrecoverable(...)`.
- **Locking:** `primary_host`/`primary_port` are guarded by `ha->lock` — take it for a consistent snapshot, copy out, release. All other fields are atomics or immutable config.
- **"unknown" mapping (1.3, 2.2, 2.3):** if `primary_host[0] == '\0'` (no known primary), emit an empty `primaryName`, which the client renders as `"unknown"`.
- **Peers (4.x):** iterate `c->myconf->peers_cfg.candidates[0..count-1]` → one `CtrlPeer{name,host,port}` each; set `peerCount`.
- Answerable in every role; returns the local view even when fenced/no-quorum.

**Dependencies**
- Inbound: `cluster.c` opcode switch — routes the request (P0).
- Outbound: `ha_state_t`, `myconf_t`, persist accessors — read-only snapshot (P0).
- Outbound: `init_reply_by_req`, `xce_reply` — reply encode/send (P0).

#### Dispatch-gate allow-lists — Service contract
- Add `OPCode_ctrlstatus` to the allow-list in `ha_dispatch_gate` for `CTRL_PRIMARY` (including the no-quorum branch), `CTRL_STANDBY`, `CTRL_ELECTING`, and `CTRL_INIT` (7.1, 7.2).
- Constraint: the change is scoped strictly to `ctrlstatus`; no other opcode's gating may widen. `ctrlstatus` never requires quorum or a committed read.

#### libxce client API — Service contract
```c
/* opcode-only request sender (clone of xce_clusterconf_req) */
int  xce_ctrlstatus_req(endpoint_t* ep);
/* send + await reply; on success MOVE_PTR the reply union out */
int  xce_ctrlstatus(xce_ctx_t* ctx, xce_ctrlstatus_t** status);
/* failover-aware read (is_write=false): returns whatever node answers (may be a
 * standby hint view). Use for the operator command. */
int  xcecli_ctrlstatus(xcecli_ctx_t c, xce_ctrlstatus_t** status);
/* AUTHORITATIVE read: returns only a reply whose responderRole == PRIMARY. If the
 * first answer is from a non-primary, follow the redirect — target the reported
 * primaryName via the reply's peers list, else rotate the configured master list
 * — until the primary answers or the failover budget is exhausted. */
int  xcecli_ctrlstatus_primary(xcecli_ctx_t c, xce_ctrlstatus_t** status);
#define xcecli_free_ctrlstatus(s)  xce_free_ctrlstatus(s)
```
- Preconditions: valid client context.
- Postconditions: on success `*status` owns a decoded `CtrlStatus` (caller frees); on failure returns a typed negative (7.3) and `*status` untouched. For `xcecli_ctrlstatus_primary`, success additionally guarantees `responderRole == PRIMARY` (an authoritative answer).
- Invariants: `is_write=false` — retryable/rotatable across the configured controllers.

#### `cmd_ctrlstatus` (test client) — State contract
- Calls `xcecli_ctrlstatus`; on success prints the fields in human-readable form (8.1) — `Primary`, `answered by <responderName> (<role>)` distinct from `Primary:` (3.3), peers + count, `configVersion` and `committedConfigVersion`, `configChecksum` as hex, term, commit/apply, fenced, cluster name; renders empty `primaryName` as `unknown`. Returns 0 on success, `<0` on failure with the error printed (8.2, 8.3). Frees the reply.

#### `cmd_wait_primary` probe swap — State contract
- Replace the single `joblist_query(...)` probe call in the deadline loop with `xcecli_ctrlstatus_primary` (the authoritative variant).
- **Success predicate (9.1, 9.2):** the probe obtains an authoritative reply with `responderRole == PRIMARY` (the answer came from the serving primary). On success, complete and print that primary's identity. A standby-only/hint answer is NOT success — the probe follows the redirect to the primary (or keeps polling) until the primary answers or the deadline elapses.
- **Decision (resolves review Issue 1):** require the answer to come **from the primary**, not merely "a primary name was reported by some node." This avoids a false success on a standby's stale `primary_host` after a primary death (the false-positive class removed from the durability soak) and faithfully meets Req 9.1's "**serving** primary." The any-node hint view remains available for the operator `ctrlstatus` command via `xcecli_ctrlstatus`.
- **Preserved (9.4):** the timeout-argument parsing, monotonic deadline, poll interval, log-level quieting, and success/failure exit codes are unchanged; only the probe call and the success message change.

## Data Models

The only new data is the reply payload above. `CtrlStatus` is a flat value object (no persistence, no aggregate); `CtrlPeer` is a bounded value list (≤ configured controller count, typically 3). `configChecksum` is the loaded config's 32-byte SHA-256 and `configVersion` is its own 8-byte prefix (a self-consistent pair); `committedConfigVersion` is the separately-tracked committed tag. Two nodes with identical loaded config content produce identical `configVersion` + `configChecksum`; differing content differs in both (5.3). A node mid config-change may show `configVersion != committedConfigVersion`.

## Error Handling

- **No reachable controller (7.3):** `xcecli_ctrlstatus` returns the failover wrapper's typed negative (e.g. no-master/timeout); `cmd_ctrlstatus` prints it and returns non-zero (8.3); `wait-primary` keeps polling until its deadline (9.3).
- **Undeterminable field (1.3):** emit the defined "unknown"/empty value, never uninitialized data.
- **Standby with no known primary (2.3):** `primaryName` empty → "unknown"; the call still succeeds (the status itself is valid).
- **Read-only guarantee (1.2):** the handler takes only read locks/atomic loads; no state mutation on any path.

## Testing Strategy

### Unit / handler
- `handle_ctrlstatus` on a PRIMARY returns `primaryName==self`, `responderRole==PRIMARY`, populated peers/version/checksum (2.1, 3.x, 4.x, 5.x).
- `handle_ctrlstatus` on a STANDBY with a known primary returns that primary and `responderRole==STANDBY`; with no known primary returns empty `primaryName` (2.2, 2.3).
- `fenced` reflects `in_catch_up`/`recovery_unrecoverable` (6.3); the handler mutates no state (1.2).
- `configVersion`/`configChecksum` are a self-consistent loaded pair — equal across two nodes with identical loaded config, differ when content differs; `committedConfigVersion` is exposed separately (5.1, 5.2, 5.3).

### Integration
- Dispatch-gate: a `ctrlstatus` request is answered (not `-XCE_ERR_STANDBY`/`-XCE_ERR_NOT_READY`) on STANDBY, ELECTING, INIT, and PRIMARY-no-quorum nodes; no other opcode's gating changed (7.1, 7.2).
- libxce round-trip: `xcecli_ctrlstatus` decodes all fields; `is_write=false` rotates across configured controllers; no-reachable-controller returns a typed failure (7.3).
- Authoritative read: `xcecli_ctrlstatus_primary`, when the first answer is from a standby, follows the redirect to the primary and returns a reply with `responderRole == PRIMARY`; it never returns a standby/hint reply as the authoritative result (9.1, 9.2).

### E2E (live `/xce-deploy-ha`)
- `ctrlstatus` against a healthy 3-node cluster prints the primary, peers, version+checksum; querying a standby still reports the primary with `answered by <standby>` (2.3, 8.1).
- `wait-primary` (authoritative probe) returns immediately on a healthy cluster and names the primary; after a primary kill, blocks until the **new primary itself** answers, then names it; does NOT false-succeed on a standby's stale primary view; times out when no primary serves (9.1–9.4). Directly useful to the `ha-commit-durability` Phase 6 liveness/>60s-election investigation.

## Security Considerations

`ctrlstatus` exposes cluster topology, role, term, and config version/checksum — the same trust level as the existing `clusterconf`/state-report reads (no secrets, no config content, no job data). It is read-only and goes through the same authenticated client path as other client ops; no new authorization surface is introduced. Broadening the dispatch gate is scoped strictly to this one read opcode.

---

## Design Delta — Requirements 10 & 11 (2026-06-20)

> **Merge note (as-built correction).** The handler shipped as `ctrlstatus_fill` + `handle_ctrlstatus` in **`src/controller/ctrlstatus.c`** (not `reply.c`); `configChecksum` is carried as **Text** lowercase-hex (the `Data` capnp type maps to the project's structured `xce_data_t`); `primaryName` is **role-derived** (primary → self, else empty) after E2E fix `ade0081` — `ha->primary_host` is vestigial. The Req 10/11 changes are written against this as-built reality; the Reqs 1–9 design above stands as the original record.

### What changed and why
Live E2E (2026-06-20): a standby's `ctrlstatus` reported `committedLsn: 0` / `committedConfigVersion: 0` while `appliedLsn: 38` — an apparent commit/apply inversion — because `ctrlstatus_fill` reads the **primary-only** markers `persist.committed_lsn` (`ctrlstatus.c:130`) and `ha->committed_config_tag` (`:109`), which stay `0` on a node that never led. And there was no responder timestamp to line a captured status up against `ctrl-N.log`. Req 10 fixes the off-primary committed sourcing; Req 11 adds a responder snapshot timestamp.

### Boundary Commitments (delta)
- **This spec owns (added):** the off-primary field-sourcing fix inside `ctrlstatus_fill` (read the *learned* committed values when not primary); a new read-only `CtrlStatus.responderTimeMs` field stamped at snapshot time; its render in `cmd_ctrlstatus`.
- **Out of Boundary (firm):** making the off-primary committed value **live-fresh** — i.e. consuming `HeartbeatMsg.committedLsn` / extending `OP_nop` to refresh `last_known_committed_lsn` on every heartbeat — is a heartbeat/replication-protocol change. It violates Req 10.4 and is routed to **ha-heartbeat-liveness** / **ha-repl-hardening**. controller-status reports the honest best-known value only. (Note: superseded by **ha-heartbeat-committed-propagation** — that spec implemented exactly this, adding `ha_consume_heartbeat_committed`/`ha_observe_heartbeat_committed`, invoked from the primary's heartbeat in `cluster.c`, which now refreshes `last_known_committed_lsn`/`last_known_committed_config_tag` on every heartbeat, term-guarded and monotonic. See that spec's design.md.)
- **Allowed Dependencies (added):** read access to `ha->last_known_committed_lsn` (`_Atomic uint64_t`, `ha.h:183`) and `ha->last_known_committed_config_tag` (`ha.h:185`); `clock_gettime(CLOCK_REALTIME)`; the existing `xcelog.h` wall-clock format for client rendering. (Note: `last_known_committed_config_tag` is now also `_Atomic uint64_t` — migrated by ha-heartbeat-committed-propagation from the plain `uint64_t` this design was written against.)
- **Revalidation Triggers (added):** any change to where/when `last_known_committed_lsn`/`last_known_committed_config_tag` are written (today: `cluster_apply_ctrlhello_reply` at join/RDHUP, `ha_observe_heartbeat_committed` on every heartbeat, and the election safe-commit path) → re-check the off-primary committed reporting and the staleness caveat. (Note: the heartbeat writer was added by ha-heartbeat-committed-propagation, superseding the "join/RDHUP only" premise below.)

### Locked design decisions (from research items)
1. **Req 10 sourcing → role-branch (not uniform `max()`).** When `responderRole == CTRL_PRIMARY`, keep the node's own authoritative live markers (`persist.committed_lsn`, `ha->committed_config_tag`). Otherwise report the learned pair (`ha->last_known_committed_lsn`, `ha->last_known_committed_config_tag`). Rationale: a standby's learned value is its only honest "committed" knowledge; a primary's own marker is authoritative and must not be shadowed by a possibly-stale learned value. (Mirrors the `last_known_*` read precedent at `ha.c:5319` (`ha_vote_self_freshness`), `ha.c:6732`.)
2. **Req 10 staleness caveat (documented behavior, not a bug).** `last_known_committed_lsn` is refreshed at **CtrlHello join / RDHUP-failover**, not per heartbeat, so an off-primary committed value MAY lag the live primary between joins. This is the in-boundary contract; live-fresh is the out-of-boundary item above. The `0`-inversion is eliminated regardless (a standby that has joined has a non-zero learned value; a never-joined node correctly reports `0` = unknown, per Req 1.3 / 10.2). (Note: superseded by **ha-heartbeat-committed-propagation** — `last_known_committed_lsn`/`_config_tag` are now also refreshed on every heartbeat via `ha_observe_heartbeat_committed`, so the join/RDHUP-only staleness window described here no longer applies; see that spec's design.md.)
3. **Req 11 wire type → `UInt64` epoch-milliseconds.** Numeric, locale-safe; the client formats it. (Pre-formatted `Text` was the alternative; rejected to keep the wire value machine-comparable and avoid encoding a locale into the reply.)
4. **Req 11 clock → `CLOCK_REALTIME`, stamped in `ctrlstatus_fill` at snapshot time.** NOT `CLOCK_MONOTONIC` (which `ha_on_heartbeat_received` uses for liveness) — monotonic is not wall-clock and cannot be correlated to logs. The client renders the millis to local `YYYY-MM-DD HH:MM:SS,mmm` using the `xcelog.h` convention so it is directly comparable to a log line.

### Research item carried to implementation
- Confirm the exact wall-clock formatter/helper in `src/include/xcelog.h` and reuse it (or match its `YYYY-MM-DD HH:MM:SS,mmm` form) in `cmd_ctrlstatus` so the printed timestamp is byte-comparable to `ctrl-N.log`. `last_known_committed_config_tag` was non-atomic at design time (written under lock in `ha_convergence.c`); the status read was planned as a best-effort aligned 8-byte load (acceptable for a reporting snapshot) — do not add locking on the read path. (Note: superseded by **ha-heartbeat-committed-propagation**, which migrated the field to `_Atomic uint64_t` at `ha.h:185`, written via `atomic_store_explicit` at `ha_convergence.c:368` and `ha.c:9011`; the as-built `ctrlstatus_fill` correctly reads it with `atomic_load_explicit`, not a bare aligned load.)

### Schema delta (Cap'n Proto)
Append one field to `CtrlStatus` (next free ordinal `@13`, after `peerCount @12`; append-only, never reorder):
```capnp
  responderTimeMs   @13: UInt64 $C.mapname("responder_time_ms");  # CLOCK_REALTIME epoch millis at snapshot (Req 11.1)
```
No change to `committedLsn @8` / `committedConfigVersion @7` wire shape — Req 10 changes only the **value source**, not the field.

### `ctrlstatus_fill` delta — Service contract (`src/controller/ctrlstatus.c`)
- **Req 10 (committed sourcing):** replace the unconditional reads at `ctrlstatus.c:108–110,130` with the role-branch above:
  - `responderRole == CTRL_PRIMARY` → `committedLsn = persist.committed_lsn`, `committedConfigVersion = ha->committed_config_tag` (atomic acquire loads, unchanged).
  - else → `committedLsn = ha->last_known_committed_lsn` (atomic acquire), `committedConfigVersion = ha->last_known_committed_config_tag` (plain aligned load).
  - `appliedLsn` (`ha->applied_lsn`) is unchanged here. **Correction (bug-1878, 2026-09-28):** it is *not* correct in every role — `ha->applied_lsn` is a standby-only counter (replicated WAL received/persisted) that a primary never advances, so every primary reported `appliedLsn: 0`. `ctrlstatus_fill` now reports a primary's apply cursor, `persist->last_applied_lsn` (the last entry applied, never above `committedLsn`); a standby still reports `ha->applied_lsn`.
- **Req 11 (timestamp):** stamp `st->responder_time_ms` from `clock_gettime(CLOCK_REALTIME)` (`tv_sec*1000 + tv_nsec/1e6`) at the top of the snapshot, before/with the other field loads, so it marks the responder's sample moment.
- **Read-only invariant (10.4, 1.2) preserved:** all additions are atomic/aligned loads + a clock read; no HA/persist/config mutation.

### `cmd_ctrlstatus` delta — State contract (`src/controller/testsuite/test.c`)
- Print one added line rendering `responder_time_ms` as local `YYYY-MM-DD HH:MM:SS,mmm` (xcelog.h form) alongside the existing fields (Req 11.3). The committed lines need no rendering change — Req 10 only changes the values they already print.

### File Structure Plan (delta)
**Modified files:**
- `src/schema/controller.capnp` — add `responderTimeMs @13: UInt64` to `CtrlStatus` (regen `controller.capnp.h/.c` → `xce_ctrlstatus_t.responder_time_ms`).
- `src/controller/ctrlstatus.c` — `ctrlstatus_fill`: role-branch committed sourcing (Req 10) + `CLOCK_REALTIME` timestamp stamp (Req 11).
- `src/controller/testsuite/test.c` — `cmd_ctrlstatus`: render the responder timestamp line (Req 11.3).
- `src/controller/testsuite/test_ctrlstatus.c` — add/extend unit tests (below).

No new files (the capnp field is generated; all logic slots into existing TUs).

### Requirements Traceability (delta)
| Requirement | Summary | Components | Interfaces |
|-------------|---------|------------|------------|
| 10.1 | Off-primary reports *learned* committed LSN + config version | `ctrlstatus_fill` role-branch | `CtrlStatus.committedLsn`, `committedConfigVersion` (value source = `ha->last_known_committed_*`) |
| 10.2 | No learned value yet → 0/unknown | `ctrlstatus_fill` | `last_known_committed_lsn == 0` (correct unknown, per 1.3) |
| 10.3 | No spurious applied>committed inversion | `ctrlstatus_fill` | `appliedLsn` vs role-correct `committedLsn` |
| 10.4 | Read-only reporting; no production/replication change | `ctrlstatus_fill`; out-of-boundary routing | atomic/aligned loads only |
| 11.1 | Responder wall-clock at sample time | `ctrlstatus_fill` (`CLOCK_REALTIME`) | `CtrlStatus.responderTimeMs` |
| 11.2 | Same clock/format as logs | `cmd_ctrlstatus` render (xcelog.h form) | local `YYYY-MM-DD HH:MM:SS,mmm` |
| 11.3 | Test client prints the timestamp | `cmd_ctrlstatus` | added print line |

### Testing Strategy (delta)
**Unit (`test_ctrlstatus.c`, TDD red→green):**
- STANDBY with `last_known_committed_lsn = 36`, `persist.committed_lsn = 0`, `applied_lsn = 38` → reply reports `committedLsn == 36` (not 0) and `appliedLsn == 38` (no inversion below the learned value) (10.1, 10.3).
- STANDBY pre-join (`last_known_committed_lsn == 0`) → reply reports `committedLsn == 0` / `committedConfigVersion == 0` (correct unknown) (10.2).
- PRIMARY → reply reports `committedLsn == persist.committed_lsn` and `committedConfigVersion == ha->committed_config_tag` (authoritative own markers unchanged) (10.1 primary branch).
- `committedConfigVersion` off-primary equals `last_known_committed_config_tag` (symmetry with the LSN) (10.1).
- `responder_time_ms` is non-zero and within a tolerance of a test-captured `CLOCK_REALTIME` sample; the handler still mutates no state (11.1, 1.2/10.4).

**Integration / round-trip (libxce):** `xcecli_ctrlstatus` decodes the new `responderTimeMs` field end-to-end (no manual decode — codegen), proving wire compatibility.

**E2E (live `/xce-deploy-ha`):** reproduce the original finding — submit jobs so the primary commits, query a **standby**; assert the standby's `committedLsn` is non-zero and tracks (within the join-refresh caveat) rather than `0`, and that the printed `responder timestamp` matches the standby's own `ctrl-N.log` line for that query. (Note: the "join-refresh caveat" is narrower than described above — ha-heartbeat-committed-propagation added per-heartbeat refresh, so tracking is now near-live rather than join/RDHUP-only; see that spec's design.md.)

---

## Design Delta — Requirement 12 (2026-09-15)

### What changed and why
User enhancement request: `ctrlstatus` should additionally report the term in which the current primary was elected and the wall-clock time it was elected, so operators can tell how long the current primary has served and correlate its election with the logs — with the same off-primary (standby) fidelity Req 10 already established for committed position, not a bare "unknown" off-primary. Scope of this delta = Req 12 only. Reqs 1–11 are implemented and unchanged.

### Boundary Commitments (delta)
- **This spec owns (added):** a new `ha_state_t.last_known_elected_at_ms` field (`_Atomic uint64_t`, declared alongside `last_known_committed_lsn`/`last_known_committed_config_tag` in `ha.h`), self-stamped by BOTH promotion roads at the exact point each commits to `CTRL_PRIMARY`; a new `HeartbeatMsg.electedAtMs` wire field the primary echoes on every outgoing heartbeat; the receiving standby's unconditional store of that value into its own `last_known_elected_at_ms`; a new `CtrlStatus.electedAtMs` reply field; `ctrlstatus_fill` populating it (one unconditional read, no role-branch — see Locked Decision 2); `cmd_ctrlstatus`'s two new printed lines (`electedTerm`, `electedAt`), gated on `elected_at_ms == 0` (corrected 2026-09-16 from an original primaryName-based gate — see Locked Decision 3).
- **Out of Boundary (firm):** `electedTerm` gets **no new field** — it reuses the existing `CtrlStatus.term` (Req 6.1) with CLI-side "unknown" gating (Locked Decision 1); this spec does not change how elections are conducted, how `ha->term` advances, or how/when `ha->role` transitions — every stamp site added here is purely observational at an already-decided promotion instant (Req 12.5).
- **Allowed Dependencies (added):** `ha->term` and the existing primaryName derivation (Req 2/3, unchanged); `clock_gettime(CLOCK_REALTIME)` (existing precedent, Req 11); `ctrlstatus_format_time_ms` (`src/include/xce/ctrlstatus_fmt.h`) reused verbatim for rendering — no new formatting code.
- **Revalidation Triggers (added):** any change to where `ha->role` finally becomes `CTRL_PRIMARY` on either promotion road (`ha_do_promote` or `raft_cb_become_leader`) → re-check the `last_known_elected_at_ms` stamp site is still past every refusal/eligibility gate; any change to the `HeartbeatMsg`/`heartbeat_msg_t` shape → re-check the `electedAtMs` ordinal and decode; any change to the primaryName-unknown derivation (Req 2) → re-check the electedTerm/electedAtMs "unknown" gating that reuses it.

### Locked design decisions
1. **`electedTerm` reuses `ha->term` (Req 6.1's existing wire field) — no new state.** This codebase's Raft term already propagates correctly to every node via the existing heartbeat/vote term-sync path; there is no "stale/zero" class of bug here the way Req 10 had for `committedLsn` (which read an unrelated *primary-only* marker, `persist.committed_lsn`, that stays 0 off-primary by construction). `cmd_ctrlstatus` derives the displayed `electedTerm` as `elected_at_ms != 0 ? term : unknown` (corrected 2026-09-16 — see Locked Decision 3; originally `primaryName known ? term : unknown`) — purely a print-layer decision, no handler change for this half of the requirement. **Accepted trade-off:** in the narrow window where a standby has just voted for a new candidate (bumping its own `ha->term`) before its `voted_for_name`-derived primaryName reflects that same candidate, the displayed `electedTerm` could transiently be a half-step out of step with the displayed primary — the same class of transient standby-hint staleness already accepted for the original `primaryName`/`term` fields (see Architecture's "any-node hint then authoritative-from-primary" flow) and for Req 10's own documented staleness caveat. Not a regression, not new risk surface.
2. **`electedAtMs` is genuinely new state, written unconditionally (no monotonic/term compare on write).** Both promotion roads write `last_known_elected_at_ms` exactly once, at the instant each road COMMITS to `CTRL_PRIMARY` — i.e., past every refusal/eligibility gate, never at function entry (see Research Items below for the exact confirm-at-implementation lines). The heartbeat-receive site writes it only when already processing a heartbeat from the node's currently-recognized leader — the same precondition `ha_observe_heartbeat_committed`'s existing call site already relies on for `committed_lsn`/`committed_config_tag`. Because every write site is already gated by an existing freshness precondition, no second "which term does my stored value correspond to" comparison is needed. **Simplification vs. Req 10's pattern:** since there is no pre-existing primary-only marker analogous to `persist.committed_lsn` for election time, `ctrlstatus_fill` needs **no role-branch** for this field — one unconditional read of `last_known_elected_at_ms` correctly serves both the primary (self-stamped) and a standby (heartbeat-learned) case.
3. **Unknown gating happens at output time, not by proactively zeroing state.** Both `electedTerm` and `electedAtMs` are gated in `cmd_ctrlstatus` on `elected_at_ms == 0` (see live-E2E correction below), not on the primaryName-known/unknown derivation Req 2.2/2.3 computes. A stale `last_known_elected_at_ms` left over from a since-lost primary is never surfaced because a fresh promotion or heartbeat-learn always overwrites it before it could be observed as stale — the reader never sees an in-between state, so no separate staleness check is needed beyond the zero-check itself.
   > **Live-E2E correction (2026-09-16, task 7.6 finding):** this decision originally gated on the SAME primaryName-known/unknown derivation Req 2.2/2.3 already computes, reasoning it would mirror how the original design treats a stale `voted_for_name`/`primary_host`. A live 3-node cluster run proved this wrong: `primaryName` is *unconditionally* empty on every non-primary responder (Req 2's own `ctrlstatus_fill` only self-reports the primary's name when the responder itself is primary — `ha->primary_host` is vestigial, per the `ade0081` design note above), so Req 2.3's "standby knows the primary" branch is dead code in the current build via the plain `ctrlstatus` command. Gating on `primaryName` therefore made `electedTerm`/`electedAt` print "unknown" on *every* standby, permanently — even though `elected_at_ms` was correctly, non-zero, and convergently populated the whole time (independently proven live: `term`/`committedLsn`/`appliedLsn`, sourced the identical unconditional way, DID correctly display and converge on the same standby query). Gating on `status->elected_at_ms == 0` instead is both necessary and sufficient: it stays 0 until a real promotion stamps it (either road) or a heartbeat carries a learned value, and — unlike `term`, which can be legitimately nonzero from mere vote/candidacy activity before any primary is actually elected — `elected_at_ms` only ever becomes nonzero via an actual election. This is a `cmd_ctrlstatus`-only correction; `ctrlstatus_fill`'s own unconditional, no-role-branch read (Locked Decision 2) is unaffected and was already correct.
4. **Wire type: `electedAtMs @14: UInt64` epoch-milliseconds** — mirrors Req 11's `responderTimeMs` numeric-not-Text precedent exactly (machine-comparable; the client formats it via the same existing `ctrlstatus_format_time_ms`).

### Research items carried to implementation
1. Confirm the exact statement in `raft_cb_become_leader` (`ha.c:1615`) where `ha->role` is finally stored as `CTRL_PRIMARY` — past the `ha_promotion_gate_decision` eligibility gate (later in the function than this delta's own discovery reached) — and stamp `last_known_elected_at_ms` there. Do **not** stamp at the earlier `last_known_committed_lsn` self-observe point (`ha.c:1665`, inside the `if (promo_ectx)` block): that point precedes the eligibility gate and can still be followed by a promotion refusal, which must not produce a bogus election time.
2. Confirm which of `ha_observe_heartbeat_committed`'s 9 existing call sites corresponds to processing a genuinely RECEIVED heartbeat (as opposed to a local/test-only invocation) and add the unconditional `last_known_elected_at_ms` store immediately alongside that call, reading the new `electedAtMs` field off the same decoded `heartbeat_msg_t`. Also confirm the outgoing side — where `primary_heartbeat_fn` (`ha.c:9791`) populates the `heartbeat_msg_t` it sends via `xce_ha_heartbeat_req` — and set `electedAtMs = last_known_elected_at_ms` there.
3. Confirm `ha_do_promote` (`ha.c:2927`, the function `ha_promote_to_primary`'s own comment describes as owning "the term increment + sets role=PRIMARY" for the non-raft road) is the correct single stamp site for that road; its body was outside this delta's discovery window.

### Constraints checked
- **Async I/O invariant (ha-async-io.md):** every write added here is an atomic store or a `clock_gettime(CLOCK_REALTIME)` call on the existing promotion/heartbeat control-flow paths — no new `connect()`, file I/O, or other blocking call is introduced; identical shape to Req 11's already-accepted `CLOCK_REALTIME` stamp.
- **Persistence / save-restore completeness:** not applicable — `last_known_elected_at_ms` is ephemeral in-memory HA state, never written to `ha_term_file_t`/WAL/any durable format, and is fully re-derived on the next promotion or heartbeat exactly like its sibling `last_known_committed_lsn`/`last_known_committed_config_tag` (also not persisted). No save/restore path is added or modified by this delta.

### Schema delta (Cap'n Proto)
```capnp
# CtrlStatus — next free ordinal @14 (after responderTimeMs @13; append-only, never reorder)
  electedAtMs   @14: UInt64 $C.mapname("elected_at_ms");
  # epoch millis of the current primary's election, learned off-primary via
  # heartbeat; 0/unset when no primary is known (gated together with primaryName)

# HeartbeatMsg — next free ordinal @4 (after committedConfigTag @3)
  electedAtMs   @4: UInt64 $C.mapname("elected_at_ms");
  # primary's own last_known_elected_at_ms, echoed on every heartbeat (Req 12.2)
```
No change to `term @4` on `CtrlStatus` — Req 12's term half changes only the CLI's derived label/gating, not the wire field or its value.

### File Structure Plan (delta)
**Modified files:**
- `ha.h` — add `_Atomic uint64_t last_known_elected_at_ms;` to `ha_state_t`, next to `last_known_committed_lsn`/`last_known_committed_config_tag`.
- `src/schema/controller.capnp` — add `electedAtMs` to `CtrlStatus` (@14) and to `HeartbeatMsg` (@4); regen `controller.capnp.h/.c`.
- `src/controller/ha.c` — `ha_do_promote` and `raft_cb_become_leader`: stamp `last_known_elected_at_ms` from `CLOCK_REALTIME` at each road's final promotion-commit point (Research Items 1, 3); the heartbeat-receive call site (`ha_consume_heartbeat_committed`): unconditional store alongside the existing `ha_observe_heartbeat_committed` call (Research Item 2); `primary_heartbeat_fn`: read `last_known_elected_at_ms` into a local, passed through to the outgoing heartbeat (Research Item 2).
- `src/controller/ctrlstatus.c` — `ctrlstatus_fill`: read `last_known_elected_at_ms` unconditionally into `st->elected_at_ms` (no role-branch, Locked Decision 2).
- `src/harness/test.c` — `cmd_ctrlstatus`: two new printed lines, `electedTerm` (derived from `term`, gated on `elected_at_ms`) and `electedAt` (rendered via `ctrlstatus_format_time_ms`), both "unknown" when `elected_at_ms == 0` (corrected 2026-09-16 — see Locked Decision 3; originally gated on `primaryName`). **Path correction (as-built, confirmed 2026-09-15):** `cmd_ctrlstatus`/`cmd_wait_primary` now live in `src/harness/test.c`, not `src/controller/testsuite/test.c` as Reqs 1-11's original design text says — the test-client harness was consolidated into `src/harness/` by a later refactor (see the `xce-harness-component` spec) after those sections were written. Historical Reqs 1-11 text is left as-is (accurate as of when implemented); this delta targets the current, correct location.
- `src/include/request.h`, `src/lib/request.c` — **as-built addition (2026-09-16), not anticipated at design time:** `xce_ha_heartbeat_req`, not `primary_heartbeat_fn` directly, owns building the outgoing `heartbeat_msg_t` (it already threaded `term`/`committed_lsn`/`committed_config_tag` as scalar args this same way) — task 7.3 found this only once it read the real send path, and added `elected_at_ms` as a 4th scalar parameter, mechanically mirroring the 3 existing ones. No election/heartbeat *protocol* semantics changed, only one added field in an already-established parameter-threading pattern.
- `src/controller/testsuite/test_ctrlstatus.c` — extend/add unit tests (below).
- `src/controller/testsuite/test_ha_heartbeat_committed.c` — coverage for the heartbeat-carried `electedAtMs` store (the reconcile-free negative-space test, mirroring `test_observe_no_reconcile_no_other_mutation`).
- `src/controller/testsuite/test_ha_role.c` — extend `test_promote_increments_term`-style coverage to also assert `last_known_elected_at_ms` is stamped on promotion (and NOT stamped on a refused promotion attempt).
- `src/controller/testsuite/test_ha_liveness.c`, `src/controller/testsuite/test_ha_partition.c` — **as-built addition (2026-09-16):** signature-conformance updates only (these files mirror `ha_heartbeat_req_hook`'s signature for their own test doubles; the new 4th parameter had to be threaded through, no test behavior changed).
- `src/lib/testsuite/test_request.c` — **as-built addition (2026-09-16):** extends the existing `xce_ha_heartbeat_req` round-trip test to also cover `elected_at_ms`, proving the new field survives a real encode/decode cycle.

No new files (capnp fields are generated; all logic slots into existing TUs) — consistent with this spec's established pattern. **Known gap (2026-09-16 feature-level validation):** `cmd_ctrlstatus`'s print-gate logic (the exact code bug-1818 regressed in) has no checked-in unit/regression test — this print layer has never had one, for any of its lines, going back to Req 8's original implementation; coverage is live-E2E (task 7.6) plus an ephemeral scratch-program check at fix time, not a permanent regression guard. Accepted as a pre-existing, deliberate scope boundary of this spec (see Requirement 8's own design text), not a new gap introduced by this delta — flagged here for whoever next touches this function.

### Requirements Traceability (delta)
| Requirement | Summary | Components | Interfaces |
|-------------|---------|------------|------------|
| 12.1 | Report the primary's election term + time alongside its identity | `ctrlstatus_fill` (`electedAtMs` read), `cmd_ctrlstatus` (`electedTerm` derived from `term`, gated on `elected_at_ms`) | `CtrlStatus.term` (reused), `CtrlStatus.electedAtMs` (new) |
| 12.2 | Standby reports the LEARNED election term/time, not stale/unset | heartbeat-carried `electedAtMs`; existing term-sync for `electedTerm` | `HeartbeatMsg.electedAtMs`, unconditional store into `last_known_elected_at_ms` |
| 12.3 | Unknown when no primary is known | `cmd_ctrlstatus` gating on `elected_at_ms == 0` (corrected 2026-09-16, see Locked Decision 3) | independent of Req 2.2/2.3's `primaryName` derivation, which is unconditionally empty on a standby in the current build |
| 12.4 | Election time on the same clock/format as logs | `ctrlstatus_fill` (`CLOCK_REALTIME` stamp), `cmd_ctrlstatus` render | reuses `ctrlstatus_format_time_ms` (Req 11.2 precedent) |
| 12.5 | Read-only; no election/leadership behavior change | stamp sites are observational only; no gating logic altered | n/a |
| 12.6 | Test client prints both fields | `cmd_ctrlstatus` | two new printed lines |

### `ctrlstatus_fill` / promotion / heartbeat delta — Service contract
- **Req 12.1/12.2 (sourcing):** `st->elected_at_ms = ha->last_known_elected_at_ms` (atomic acquire load), unconditionally — no role-branch (Locked Decision 2).
- **Req 12.5 (read-only invariant) preserved:** the handler itself performs only atomic loads, exactly as every other field; all writes to `last_known_elected_at_ms` happen upstream, at promotion or heartbeat-processing time, never inside `ctrlstatus_fill`.
- **`cmd_ctrlstatus` (Req 12.1, 12.3, 12.6):** print
  ```
  electedTerm: <term>          (or "unknown" when elected_at_ms == 0)
  electedAt: <ctrlstatus_format_time_ms(elected_at_ms)>   (or "unknown" when elected_at_ms == 0)
  ```
  placed after the existing `Primary:` line, grouping the primary's identity with its election term/time. Gated on `elected_at_ms == 0`, not on `primaryName` (corrected 2026-09-16 — see Locked Decision 3).

### Testing Strategy (delta)
**Unit (`test_ctrlstatus.c`):**
- PRIMARY: `ctrlstatus_fill` reports `elected_at_ms == ha->last_known_elected_at_ms` (self-stamped); CLI-derived `electedTerm == st.term`.
- STANDBY with a learned `last_known_elected_at_ms` (simulating an already-processed heartbeat) and a known primaryName → reports the learned value, not 0/stale (12.2).
- STANDBY with primaryName unknown (no vote / self-vote / electing) → CLI renders `electedTerm`/`electedAt` as "unknown" regardless of any stale stored `last_known_elected_at_ms` (12.3).
- Extend the existing `test_ctrlstatus_no_mutation` snapshot/compare to also cover `last_known_elected_at_ms` (12.5).

**New/extended heartbeat unit test (sibling to `test_ha_heartbeat_committed.c`):**
- Processing a received heartbeat with a given `electedAtMs` unconditionally overwrites `last_known_elected_at_ms` and leaves `last_known_committed_lsn`/`last_known_committed_config_tag`/role/term/applied untouched — reconcile-free, mirrors `test_observe_no_reconcile_no_other_mutation` (12.2, 12.5).

**Integration:**
- Extend `test_ha_role.c`'s `test_promote_increments_term`-style coverage (non-raft road, `ha_do_promote`) to also assert `last_known_elected_at_ms` is stamped on promotion.
- Add/extend a `raft_cb_become_leader` promotion test to assert the same for the raft road, specifically confirming NO stamp occurs on a refused/gated promotion attempt (Research Item 1).
- Wire round-trip: `xcecli_ctrlstatus` decodes the new `CtrlStatus.electedAtMs` field end-to-end (codegen, no manual decode).

**E2E (live `/xce-deploy-ha`):**
- Query `ctrlstatus` on the primary right after a fresh election: `electedTerm` equals the printed `term`; `electedAt` is close to "now."
- Query a STANDBY: `electedTerm`/`electedAt` match the primary's own values (within one heartbeat interval), not "unknown" and not stale/zero — proving standby learned fidelity end-to-end (12.2).
- Kill the primary, wait for a new election, query the surviving standby immediately vs. after one heartbeat interval — demonstrates the field converges promptly via heartbeat, unlike Req 10's original join/RDHUP-only caveat (heartbeat-carried delivery here is in-boundary from day one of this delta, not a follow-up spec).

---

## Design Delta — Requirement 13 (2026-09-16)

### What changed and why
User enhancement request: `ctrlstatus`'s "Controllers (N):" list should additionally show each configured controller's connection status (ok / catching-up / dead), not just name and address, so operators can see at a glance which controllers are healthy, rejoining, or unreachable. Confirmed with the user: three-state granularity (reusing the already-tracked connected-vs-catching-up distinction) and best-effort per-responder reporting — "unknown" for peers a responder cannot itself observe, consistent with this spec's established off-primary fidelity pattern, not a primary-only-authoritative requirement. Scope of this delta = Req 13 only. Reqs 1–12 are implemented and unchanged.

### Boundary Commitments (delta)
- **This spec owns (added):** a new `CtrlPeer.status` reply field (per-peer, not per-response — this is the first Req 12+ delta to extend the nested `CtrlPeer` struct rather than `CtrlStatus` itself); `ctrlstatus_fill` populating it per peer, sourced differently by responder role (primary: matched against live connection state; non-primary: self only, "unknown" otherwise); a status-code-to-string renderer (mirroring the existing `responderRole`/`ctrlstatus_role_str` pattern); `cmd_ctrlstatus`'s peer-list print line extended to show it.
- **Out of Boundary (firm):** this spec does not change how standby connections are established, monitored, evicted, or torn down — it only reads the connection-tracking state that already exists for replication/quorum purposes (Req 13.5). It does not give a standby visibility into peers it genuinely cannot observe (confirmed at design time: this codebase's connection model is asymmetric — only the primary holds live connections to every standby; standbys do not connect to each other, and no existing mechanism lets a standby identify even its OWN primary by peer identity, only "elapsed time since last primary message") — a standby reporting anything but "unknown" for a peer other than itself would require a genuinely new cross-node identity-carrying mechanism, which is out of scope for this delta (see Locked Decision 3).
- **Allowed Dependencies (added):** the existing replication connection registry (`replicator_t.standbys[]`, `src/include/repl.h`) that already tracks each standby's live connection (`ep`), catch-up state (`in_catch_up`), and host/port — read-only, no new writes to it.
- **Revalidation Triggers (added):** any change to `standby_conn_t`'s fields or to how/when `ep`/`in_catch_up` are set (`repl_add_standby`/`repl_remove_standby`/`repl_start_catchup`/`repl_complete_catchup`) → re-check the peer-status sourcing still reads the correct signal; any change to `peers_cfg.candidates[]`'s host/port fields → re-check the host+port matching still correlates correctly to `replicator_t.standbys[]`.

### Locked design decisions
1. **Wire type: `CtrlPeer.status @3: UInt8`**, a small fixed-value code — not `Text`. Mirrors the existing `CtrlStatus.responderRole @3: UInt8` precedent exactly (backed by a C enum + a `ctrlstatus_..._str()` string renderer at the client layer), rather than introducing a new stringly-typed convention for this one field.
2. **Status values, in ordinal order:** `0 = unknown`, `1 = ok`, `2 = catching-up`, `3 = dead`. `unknown` is `0` (the default/zero value) so a peer entry that nothing ever sets defaults to the safe, honest "can't tell" state rather than an implicitly-true `ok`.
3. **Primary-side sourcing: match by host+port, not by name or position.** `standby_conn_t` (the primary's live connection-tracking slot) stores `host`/`port` but not a configured peer name or a stable index into `peers_cfg.candidates[]` — so `ctrlstatus_fill`, when the responder is primary, must scan `replicator_t.standbys[]` for a slot whose `host`/`port` match each configured peer's `hostname`/`port`. No match (or a matched slot with `ep == NULL`) → `dead`; a match with `ep != NULL && in_catch_up` → `catching-up`; a match with `ep != NULL && !in_catch_up` → `ok`. This mirrors `repl_active_standby_count`'s own existing `ep != NULL && !in_catch_up` predicate (`repl.c`) — reuse that exact condition rather than inventing a new one. **Correction (task 8.6, 2026-09-16, live-E2E finding):** the scan must NOT stop at the first `host`/`port` match. A standby that is killed and later reconnects can end up occupying a NEW `replicator_t.standbys[]` slot while its OLD slot (host/port still populated, `ep == NULL`) also still matches the same configured peer — `repl_add_standby`'s `ep`-identity idempotency check cannot reuse the old slot once its `ep` is gone. A first-match-wins scan then reports that peer `dead` PERMANENTLY, even after it has genuinely reconnected and fully caught up (live-reproduced: `ctrlstatus` stayed `dead` for 5+ minutes after a rejoin the primary's own log showed as fully caught up), violating Req 13.3's "true, currently-known" guarantee. The corrected algorithm scans ALL `MAX_STANDBYS` slots for a given peer and prefers ANY slot with `ep != NULL` (deriving `ok`/`catching-up` from that live slot) over a slot with `ep == NULL`; only falls back to `dead` when no matching slot has `ep != NULL`. See `.wolf/buglog.json` bug-1819.
4. **The responder always reports its own entry as `ok`** (Req 13.2) — matched by comparing each configured peer's name against `st->responderName`, the same identity `ctrlstatus_fill` already establishes for itself.
5. **Non-primary responder: only self is determinable; every other peer — including the primary — reports `unknown`.** This was an open question at requirements time (whether a standby could additionally report its own primary's status via the liveness signal it already tracks for failover); design-time investigation closes it: that liveness signal (`ha->last_activity`, `liveness_poll_fn`) is a bare elapsed-time scalar with no peer identity attached, and this codebase has no other mechanism (analogous to the vestigial `ha->primary_host`) that would let a standby reliably attribute it to one specific configured peer. Building one would be materially larger than this delta (a genuinely new identity-carrying signal on the heartbeat/connection-accept path) and is explicitly out of boundary here — a future spec's concern, not silently absorbed into this one.
6. **Read-only, no role-branch beyond what Decision 3/5 already requires**: `ctrlstatus_fill` performs only reads (of `peers_cfg`, and — only when the responder is primary — of `replicator_t.standbys[]` under its existing lock); no new state is written, no existing connection-management behavior changes.

### Research item carried to implementation
1. Confirm the exact field path from `cluster_t` to its `replicator_t` (repl.h's own comment says only "embedded in persist_t or cluster_t (P2-4)", ambiguous at design time — likely `c->persist.repl` given `replicator_t.ce` couples it to the commit engine which lives off `persist_t`, but confirm against the current struct definition, not this note, before wiring the read).

### Constraints checked
- **Async I/O invariant (ha-async-io.md):** the new read takes `replicator_t.lock` (a `mylock_t`) briefly to scan a small, fixed-size (`MAX_STANDBYS`) in-memory array — no network I/O, no unbounded wait, identical in shape to the already-established `repl_active_standby_count`'s own lock-and-scan. **Correction (second design-review pass, 2026-09-16):** the previous version of this note claimed "the primary-identity read already takes `ha->lock`" — that was true only of the ORIGINAL Req 1-9 design text, not the as-built code: after the `ade0081` fix (Req 5.1, primaryName made role-derived/self-only), `ctrlstatus_fill`'s current source takes NO lock anywhere — every existing field is an atomic load or a plain config read. This delta's `replicator_t.lock` acquisition is therefore the FIRST lock this function has taken since that fix, not a repeat of an existing pattern within `ctrlstatus_fill` itself. It is still safe: `ctrlstatus_fill` runs on the request-handling path (not the trigger-only delaymgr thread), and the exact same lock is already taken from `repl_active_standby_count` on the write-admission quorum path — a comparably latency-sensitive context — with no documented contention issue.
- **Persistence / save-restore completeness:** not applicable — `CtrlPeer.status` is derived entirely from existing in-memory connection state (`replicator_t.standbys[]`, already not persisted) and existing config (`peers_cfg`); nothing new is written to disk, WAL, or any snapshot format, and there is no save/restore path for this delta to audit.

### Schema delta (Cap'n Proto)
```capnp
# CtrlPeer — next free ordinal @3 (after port @2; append-only, never reorder)
struct CtrlPeer $C.mapname("xce_ctrlpeer_t") {
  name   @0: Text;
  host   @1: Text;
  port   @2: UInt32;
  status @3: UInt8 $C.mapname("status");
  # 0=unknown 1=ok 2=catching-up 3=dead (Req 13.1). Always 1 (ok) for the
  # responder's own entry (Req 13.2). On a non-primary responder, every
  # entry but the responder's own is 0 (unknown) — Req 13.4/Locked Decision 5.
}
```
No change to `CtrlStatus`'s own top-level fields — this delta extends the nested `CtrlPeer` list element, not the reply envelope.

### File Structure Plan (delta)
**Modified files:**
- `src/schema/controller.capnp` — add `status @3: UInt8` to `CtrlPeer`; regen `controller.capnp.h/.c`.
- `src/include/typemap.h` — add `uint8_t status;` to the hand-authored `xce_ctrlpeer_t` struct, matching the schema field.
- `src/controller/ctrlstatus.c` — `ctrlstatus_fill`'s peer-building loop: set each `xce_ctrlpeer_t.status` per Locked Decisions 3–5 (self → ok; primary responder → host/port-matched against `replicator_t.standbys[]`; non-primary responder, non-self → unknown). Add a small `ctrlstatus_peer_status_str()` (or similarly-named) helper alongside the existing `ctrlstatus_role_str`, in whichever header/TU that already lives in.
- `src/harness/test.c` — `cmd_ctrlstatus`: extend the existing per-peer print line (`printf("\t%s\t%s:%u\n", ...)`) to also render the status string.
- `src/controller/testsuite/test_ctrlstatus.c` — extend/add unit tests (below).

No new files (the capnp field is generated; the small status-code enum and its string renderer slot into an existing header/TU) — consistent with this spec's established pattern.

### Requirements Traceability (delta)
| Requirement | Summary | Components | Interfaces |
|-------------|---------|------------|------------|
| 13.1 | Report each peer's status as one of a well-defined value set, alongside name/address | `ctrlstatus_fill` | `CtrlPeer.status` (new) |
| 13.2 | Responder always reports its own status as ok | `ctrlstatus_fill` (self-match by name) | `CtrlPeer.status == 1` for the self entry |
| 13.3 | Primary reports the true status for every other controller | `ctrlstatus_fill` (host/port match against `replicator_t.standbys[]`) | `CtrlPeer.status` sourced from `ep`/`in_catch_up` |
| 13.4 | Non-primary reports unknown for what it cannot determine | `ctrlstatus_fill` (non-primary branch) | `CtrlPeer.status == 0` for every non-self entry |
| 13.5 | Read-only; no change to connection lifecycle | `ctrlstatus_fill` reads `replicator_t` only, under its existing lock; writes nothing | n/a |
| 13.6 | Test client prints each peer's status | `cmd_ctrlstatus` | extended per-peer print line |

### `ctrlstatus_fill` / peer-status delta — Service contract
- **Req 13.1–13.4 (sourcing):** for each `peers_cfg.candidates[i]`, set `status`:
  - `candidates[i].name == st->responderName` → `1` (ok) — unconditional, regardless of role (Req 13.2).
  - Else, `responderRole == CTRL_PRIMARY` → scan `replicator_t.standbys[]` (under its lock) for a slot whose `host`/`port` match `candidates[i].hostname`/`.port`; no match or `ep == NULL` → `3` (dead); `ep != NULL && in_catch_up` → `2` (catching-up); `ep != NULL && !in_catch_up` → `1` (ok).
  - Else (non-primary responder, non-self peer) → `0` (unknown).
- **Self-match fallback (design-review finding, 2026-09-16):** the self-match is a straightforward name comparison against `st->responderName`, the same identity `ctrlstatus_fill` already establishes for itself and already expects to be self-consistent with `peers_cfg` (this is a pre-existing assumption of the ORIGINAL Req 1-9 design, not new to this delta — a controller whose own name is absent from its own configured peer list is already a misconfiguration visible today, since it would already be missing an entry from the pre-existing `Controllers (N):` list's identity, unrelated to status). This delta does not need to invent new self-identity infrastructure to handle that pre-existing failure mode, but it MUST NOT let it produce a silently wrong status: if no `candidates[i]` matches `responderName`, `ctrlstatus_fill` shall `WARN` once per call (visible in the responder's own log, consistent with this codebase's existing fail-loud-not-silent convention) and leave every peer's status to fall through to the normal non-self branches above (i.e., no entry is force-corrected to `ok` under this condition) — an explicit, logged degradation rather than an undefined or fabricated result. This keeps Req 13.2's "always ok for self" guarantee intact for every correctly-configured cluster while making a misconfigured one loudly debuggable instead of silently misleading.
- **Req 13.5 (read-only) preserved:** only reads `peers_cfg` and, conditionally, `replicator_t.standbys[]` under its existing lock — no new writes anywhere. The `WARN` log above is observability-only, not a state mutation.
- **`cmd_ctrlstatus` (Req 13.1, 13.6):** extend the existing `Controllers (N):` per-peer line to also print the rendered status string, e.g. `\t%s\t%s:%u\t%s\n` with the fourth `%s` from the new status-string helper.

### Testing Strategy (delta)
**Unit (`test_ctrlstatus.c`):**
- PRIMARY with a full `replicator_t.standbys[]` (one slot `ep != NULL && !in_catch_up`, one `ep != NULL && in_catch_up`, one peer with no matching slot at all) → `ctrlstatus_fill` reports `ok`/`catching-up`/`dead` respectively for those three peers, and `ok` for its own entry (13.1–13.3).
- STANDBY responder → reports `ok` for its own entry and `unknown` (0) for every other configured peer, including the primary (13.2, 13.4).
- Handler mutates no state — extend the existing `test_ctrlstatus_no_mutation` to also snapshot/compare whatever `replicator_t` fields the primary-branch read touches (13.5).
- Status-string renderer: each of the 4 codes renders its expected string; an out-of-range code renders a safe fallback (not garbage), mirroring `ctrlstatus_role_str`'s own convention if it has one.
- Self-match fallback (design-review finding): a fixture whose `responderName` matches no `candidates[i].name` (simulating the misconfiguration case) still produces a defined, non-crashing result via the normal non-self branches, and a `WARN` is logged — proving the degradation is loud and deterministic, not silent or undefined (13.2).

**Integration:**
- Wire round-trip: `xcecli_ctrlstatus` decodes the new `CtrlPeer.status` field for every peer in the list (codegen, no manual decode).

**E2E (live `/xce-deploy-ha`):**
- Query `ctrlstatus` on the primary of a healthy 3-node cluster: all three peers (including itself) show `ok`.
- Kill one standby (not the primary): re-query the primary — that standby now shows `dead`, the other two still `ok`.
- Query a surviving standby directly: its own entry shows `ok`, every other entry (including the primary) shows `unknown` — proving the asymmetric-visibility design decision holds live, not just in a unit test.
- (Optional, if a clean way to hold a node in catch-up long enough to observe exists in this project's E2E tooling) restart the killed standby and query the primary again shortly after: that peer transiently shows `catching-up` before settling to `ok`.