Rongsong Shen d70bf6a402 update documents

2026-08-10 17:51:51 +08:00

33 KiB

Raw Blame History

Design Document — controller-status (`ctrlstatus`)

Overview

Purpose: add a single, cheap, read-only client→controller query — ctrlstatus, an XCE analog of LSF lsid — that returns basic cluster/controller identity and health: who is the current primary, the answering node's identity and role, the configured controllers (peers), the committed configuration version and content checksum, and a few HA health/progress fields (term, commit/apply position, fenced state, cluster name).

Users: operators inspecting a running cluster, and test/E2E authors orchestrating failover (including wait-primary, which switches its probe to this query).

Impact: today there is no status query — callers infer "is there a serving primary" indirectly (e.g. by listing jobs). This adds a new opcode by cloning the existing opcode-only read pattern of clusterconf, reads only already-maintained HA/config state (no behavior change), and crucially makes the query answerable on any node by adding it to the HA dispatch-gate allow-lists.

Goals

A new read-only opcode returning the status fields of Requirements 2–6 in one request/response (1.1, 1.4).
Answerable on any node — a standby reports a hint view (the primary it knows); an authoritative answer comes only from the primary, reached by following the redirect / rotating the master list (2.3, 7.1–7.3).
Report the loaded config version + content checksum (and the committed version) so same-version divergence is detectable (5.2, 5.3).
A ctrlstatus test-client command (8.x) and a wait-primary probe switched to an authoritative (primary-only) status query (9.x).

Non-Goals

No operator-facing CLI/REST lsid surface (Go tier) — later pass.
No change to HA election, replication, or commit behavior — the query only reads existing state.
No new configuration checksum computation — the committed-config SHA-256 already exists; this only exposes it.

Boundary Commitments

This Spec Owns

A new Cap'n Proto opcode ctrlstatus and its reply payload (CtrlStatus, CtrlPeer).
The controller handler handle_ctrlstatus that snapshots local HA/config state into the reply.
The dispatch-gate allow-list entries that let ctrlstatus answer in every controller role (incl. no-quorum).
The libxce client API (xce_ctrlstatus_req / xce_ctrlstatus / xcecli_ctrlstatus + free helper).
The ctrlstatus test-client command and the wait-primary probe switch.

Out of Boundary

The Go CLI/REST lsid surface (later spec).
Any change to how HA state (role, term, primary identity, committed config tag, commit/apply LSN, fence) is produced or maintained — this spec only reads it.
The existing clusterconf opcode and its resource/queue/worker payload (left unchanged).

Allowed Dependencies

The Cap'n Proto schema + build-time codegen (src/schema/controller.capnp → generated controller.capnp.h/.c).
The existing request/reply plumbing (init_reply_by_req, xce_reply, opcode-only *_req senders) and client failover wrapper (XCECLI_FAILOVER_CALL, is_write=false).
Read access to existing HA state (ha_state_t), config (myconf_t: committed_config_tag, config_sha256, peers_cfg), and persist accessors (persist_recovery_unrecoverable, persist_durable_lsn).

Revalidation Triggers

Any change to the CtrlStatus/CtrlPeer reply shape or the ctrlstatus opcode ordinal → re-check libxce decode + the test command + any future consumer.
Any change to the dispatch-gate role allow-lists → re-check that ctrlstatus still answers in every role and that no other opcode's gating widened.
A change to the meaning of committed_config_tag / config_sha256 → re-check the version/checksum fields.

Architecture

Existing Architecture Analysis

Opcode-only read precedent: clusterconf is sent with no request body (Request_body_nomsg), dispatched in cluster.c switch(request.opcode), handled by handle_clusterconf (reply.c:10), and returned via xce_reply. The client side is xce_clusterconf → xcecli_clusterconf (XCECLI_FAILOVER_CALL, is_write=false). ctrlstatus clones this pattern.
No-alloc state-report precedent: ha_nop_reply_state_report (ha.c:3802) reads HA atomics into a stack struct and points the reply union at it without heap allocation. handle_ctrlstatus follows this model (a small bounded peer list is the only allocation).
Dispatch gate (the critical seam): ha_dispatch_gate (ha.c:4525) is an allow-list keyed by role. The CTRL_STANDBY branch defaults to -XCE_ERR_STANDBY; CTRL_INIT/CTRL_ELECTING default to -XCE_ERR_NOT_READY; the CTRL_PRIMARY no-quorum branch allow-lists only specific reads. clusterconf is not in any standby allow-list — it is implicitly primary-only-when-serving. A status query must not inherit that; it must be explicitly allow-listed in every role branch.
Config content checksum already exists: config_version_tag = first 8 bytes of SHA-256 over the committed config blob; full digest in config_sha256[32], round-trip-verified on load (config_repl.c).

Pattern & dependency direction

Extend-in-place along the existing request path. Dependency direction (unchanged): schema → libxce (request/reply/client) → controller handler; the test client depends on libxce. No upward dependencies introduced.

graph LR
    TestClient[xce_test ctrlstatus and wait_primary] --> Libxce[libxce xcecli_ctrlstatus]
    Libxce --> Failover[XCECLI_FAILOVER_CALL is_write false]
    Failover --> Gate[ha_dispatch_gate allow_list all roles]
    Gate --> Handler[handle_ctrlstatus snapshots local state]
    Handler --> HAState[ha_state_t and myconf_t]
    Handler --> Reply[xce_reply CtrlStatus]

File Structure Plan

Modified Files

src/schema/controller.capnp — add OPCode ctrlstatus @40; new struct CtrlStatus $C.mapname("xce_ctrlstatus_t") and struct CtrlPeer $C.mapname("xce_ctrlpeer_t"); reply-union member ctrlstatus @17: CtrlStatus. (Build regenerates controller.capnp.h/.c → xce_ctrlstatus_t, RequestReply_data_ctrlstatus, free_CtrlStatus.)
src/controller/reply.c — add handle_ctrlstatus (snapshot local HA/config state into a CtrlStatus reply; small peer list; xce_reply). Mirrors handle_clusterconf/ha_nop_reply_state_report.
src/controller/cluster.c — add case OPCode_ctrlstatus: rc = handle_ctrlstatus(...) to the opcode switch.
src/controller/ha.c — add OPCode_ctrlstatus to the ha_dispatch_gate allow-lists for CTRL_PRIMARY (incl. the no-quorum branch), CTRL_STANDBY, CTRL_ELECTING, CTRL_INIT. (the requirements-critical change)
src/lib/request.c + src/include/request.h — add xce_ctrlstatus_req (opcode-only sender, clone of xce_clusterconf_req).
src/lib/xceapi.c — add xce_ctrlstatus (send req, cbuf_get status, MOVE_PTR reply union).
src/lib/client.c + src/include/client.h — add xcecli_ctrlstatus (XCECLI_FAILOVER_CALL, is_write=false), the authoritative xcecli_ctrlstatus_primary (follows redirect / rotates the master list until responderRole == PRIMARY), and the xcecli_free_ctrlstatus macro.
src/controller/testsuite/test.c — add cmd_ctrlstatus + table registration (uses xcecli_ctrlstatus, shows the responder's view); switch cmd_wait_primary's probe (currently joblist_query(...) at the deadline-loop call site) to xcecli_ctrlstatus_primary (authoritative, primary-only success).

New Files

None (capnp C types are generated; all handlers/APIs slot into existing TUs).

System Flows

Status query: any-node hint then authoritative-from-primary

sequenceDiagram
    participant C as Client
    participant S as Standby node
    participant P as Primary node
    C->>S: ctrlstatus request
    S-->>C: reply responderRole STANDBY plus known primary hint
    Note over C,S: Standby answer is a non authoritative hint view
    C->>P: re-query targeting the reported primary
    P-->>C: authoritative reply responderRole PRIMARY

The dispatch gate allow-lists ctrlstatus in every role (incl. no-quorum), so any node answers with its local view (satisfying any-node visibility). An authoritative answer comes only from the node whose responderRole == PRIMARY. When the first reply is from a non-primary, the client follows the redirect — it maps the reply's primaryName to an address via the reply's peers list (or rotates the configured master list) and re-queries — until it obtains the primary's authoritative reply or exhausts its budget. The plain ctrlstatus command may display the first (hint) reply with its "answered by" framing; wait-primary and any authoritative use require the primary's reply. Gate decision: ctrlstatus is allow-listed in every role branch and does not require quorum or a committed read — it returns the node's local view even when fenced or without quorum (precisely when an operator most wants it).

wait-primary probe (Req 9)

wait-primary keeps its existing monotonic-deadline poll loop and timeout/exit semantics; only the probe changes from joblist_query to xcecli_ctrlstatus. Success predicate: the status reply reports a non-"unknown" current primary. On success it prints the served primary's identity. (Rationale + caveat below.)

Requirements Traceability

Requirement	Summary	Components	Interfaces
1.1, 1.2, 1.4	Single-call read-only focused status	`handle_ctrlstatus`, `CtrlStatus` schema	opcode `ctrlstatus`, reply `CtrlStatus`
1.3	"unknown" for undeterminable fields	`handle_ctrlstatus`	empty Text/sentinel mapping
2.1, 2.2, 2.3	Current primary identity (unknown when unelected; standby's known primary)	`handle_ctrlstatus` (reads `ha->primary_host` under `ha->lock`)	`CtrlStatus.primaryName`
3.1, 3.2, 3.3	Answered-by node + role, distinct from primary	`handle_ctrlstatus`	`CtrlStatus.responderName`, `responderRole`
4.1, 4.2	Configured controllers + count	`handle_ctrlstatus` (reads `peers_cfg`)	`CtrlStatus.peers`, `peerCount`
5.1, 5.2, 5.3	Config version + content checksum (loaded pair + committed)	`handle_ctrlstatus` (reads loaded `config_version_tag`/`config_sha256` + `committed_config_tag`)	`CtrlStatus.configVersion`, `configChecksum`, `committedConfigVersion`
6.1–6.4	Term, commit/apply, fenced, cluster name	`handle_ctrlstatus`	`CtrlStatus.term`, `committedLsn`, `appliedLsn`, `fenced`, `clusterName`
7.1, 7.2, 7.3	Failover-aware any-node delivery	`xcecli_ctrlstatus`, dispatch-gate allow-lists	`XCECLI_FAILOVER_CALL` (is_write=false), gate entries
8.1, 8.2, 8.3	Test-client `ctrlstatus` command	`cmd_ctrlstatus`	client command + exit code
9.1–9.4	wait-primary uses an authoritative (primary-only) status probe	`cmd_wait_primary` probe swap, `xcecli_ctrlstatus_primary`	redirect-to-primary, `responderRole == PRIMARY` success

Components and Interfaces

Component	Layer	Intent	Req Coverage	Key Dependencies	Contracts
`CtrlStatus`/`CtrlPeer` schema	Schema	Reply payload for the status query	2–6	capnp codegen (P0)	API (reply)
`handle_ctrlstatus`	Controller	Snapshot local HA/config state into the reply	1–6	`ha_state_t`, `myconf_t`, persist accessors (P0)	Service
dispatch-gate allow-lists	Controller HA	Permit `ctrlstatus` in every role	7	`ha_dispatch_gate` (P0)	Service
`xcecli_ctrlstatus` / `xcecli_ctrlstatus_primary`	libxce	Failover-aware read (any-node) + authoritative reach-primary read + free	1, 7, 9	failover wrapper, redirect hint (P0)	Service
`cmd_ctrlstatus`	Test client	Print the status (responder view)	8	`xcecli_ctrlstatus` (P0)	State
`cmd_wait_primary` probe	Test client	Authoritative serving-primary probe	9	`xcecli_ctrlstatus_primary` (P0)	State

Schema — reply payload (Cap'n Proto)

struct CtrlPeer $C.mapname("xce_ctrlpeer_t") {
  name @0: Text;
  host @1: Text;
  port @2: UInt32;
}

struct CtrlStatus $C.mapname("xce_ctrlstatus_t") {
  clusterName       @0:  Text;
  primaryName       @1:  Text;    # empty ⇒ "unknown" (no elected/known primary)
  responderName     @2:  Text;    # the node that answered
  responderRole     @3:  UInt8;   # ctrl_role_t: INIT/ELECTING/PRIMARY/STANDBY
  term              @4:  UInt64;
  # Loaded config (consistent pair: version is the 8-byte prefix of the checksum)
  configVersion     @5:  UInt64;  # loaded config_version_tag (8-byte SHA-256 prefix)
  configChecksum    @6:  Data;    # loaded config_sha256 (full 32 bytes)
  # Committed config version (may differ from loaded mid config-change)
  committedConfigVersion @7: UInt64;  # committed_config_tag
  committedLsn      @8:  UInt64;
  appliedLsn        @9:  UInt64;
  fenced            @10: Bool;    # in_catch_up OR recovery_unrecoverable
  peers             @11: List(CtrlPeer) $C.mapname("peers.list") $C.maplistcount("peers.count");
  peerCount         @12: UInt32;
}

Config fields (Req 5): configVersion/configChecksum are the loaded config's tag and full SHA-256 — a self-consistent pair (the version is the first 8 bytes of the checksum), so two nodes with identical loaded content match on both and divergent content differs on both (5.3). committedConfigVersion is the separately-tracked committed tag, exposing "what I run" vs "what is committed"; in steady state all three agree.
New OPCode ordinal: ctrlstatus @40 (next free after preVoteReply @39). New reply-union member: ctrlstatus @17 (next free after walackreplymsg @16). Field ordinals are append-only — never reuse/reorder.

`handle_ctrlstatus` — Service contract

Field	Detail
Intent	Build a `CtrlStatus` from the responding node's local HA/config state and reply
Requirements	1.1–1.4, 2.1–2.3, 3.1–3.3, 4.1–4.2, 5.1–5.3, 6.1–6.4

Responsibilities & Constraints

Read-only: must not modify any HA, persist, or config state (1.2).
Field sourcing: clusterName/responderName from c->myconf/own peerinfo; responderRole, term, appliedLsn, committedLsn, committedConfigVersion (committed_config_tag) via atomic_load_explicit(..., acquire); configVersion from the loaded config_version_tag and configChecksum from the loaded config_sha256[32] (a self-consistent pair — Req 5.3); primaryName/peers as below; fenced = in_catch_up || persist_recovery_unrecoverable(...).
Locking: primary_host/primary_port are guarded by ha->lock — take it for a consistent snapshot, copy out, release. All other fields are atomics or immutable config.
"unknown" mapping (1.3, 2.2, 2.3): if primary_host[0] == '\0' (no known primary), emit an empty primaryName, which the client renders as "unknown".
Peers (4.x): iterate c->myconf->peers_cfg.candidates[0..count-1] → one CtrlPeer{name,host,port} each; set peerCount.
Answerable in every role; returns the local view even when fenced/no-quorum.

Dependencies

Inbound: cluster.c opcode switch — routes the request (P0).
Outbound: ha_state_t, myconf_t, persist accessors — read-only snapshot (P0).
Outbound: init_reply_by_req, xce_reply — reply encode/send (P0).

Dispatch-gate allow-lists — Service contract

Add OPCode_ctrlstatus to the allow-list in ha_dispatch_gate for CTRL_PRIMARY (including the no-quorum branch), CTRL_STANDBY, CTRL_ELECTING, and CTRL_INIT (7.1, 7.2).
Constraint: the change is scoped strictly to ctrlstatus; no other opcode's gating may widen. ctrlstatus never requires quorum or a committed read.

libxce client API — Service contract

/* opcode-only request sender (clone of xce_clusterconf_req) */
int  xce_ctrlstatus_req(endpoint_t* ep);
/* send + await reply; on success MOVE_PTR the reply union out */
int  xce_ctrlstatus(xce_ctx_t* ctx, xce_ctrlstatus_t** status);
/* failover-aware read (is_write=false): returns whatever node answers (may be a
 * standby hint view). Use for the operator command. */
int  xcecli_ctrlstatus(xcecli_ctx_t c, xce_ctrlstatus_t** status);
/* AUTHORITATIVE read: returns only a reply whose responderRole == PRIMARY. If the
 * first answer is from a non-primary, follow the redirect — target the reported
 * primaryName via the reply's peers list, else rotate the configured master list
 * — until the primary answers or the failover budget is exhausted. */
int  xcecli_ctrlstatus_primary(xcecli_ctx_t c, xce_ctrlstatus_t** status);
#define xcecli_free_ctrlstatus(s)  xce_free_ctrlstatus(s)

Preconditions: valid client context.
Postconditions: on success *status owns a decoded CtrlStatus (caller frees); on failure returns a typed negative (7.3) and *status untouched. For xcecli_ctrlstatus_primary, success additionally guarantees responderRole == PRIMARY (an authoritative answer).
Invariants: is_write=false — retryable/rotatable across the configured controllers.

`cmd_ctrlstatus` (test client) — State contract

Calls xcecli_ctrlstatus; on success prints the fields in human-readable form (8.1) — Primary, answered by <responderName> (<role>) distinct from Primary: (3.3), peers + count, configVersion and committedConfigVersion, configChecksum as hex, term, commit/apply, fenced, cluster name; renders empty primaryName as unknown. Returns 0 on success, <0 on failure with the error printed (8.2, 8.3). Frees the reply.

`cmd_wait_primary` probe swap — State contract

Replace the single joblist_query(...) probe call in the deadline loop with xcecli_ctrlstatus_primary (the authoritative variant).
Success predicate (9.1, 9.2): the probe obtains an authoritative reply with responderRole == PRIMARY (the answer came from the serving primary). On success, complete and print that primary's identity. A standby-only/hint answer is NOT success — the probe follows the redirect to the primary (or keeps polling) until the primary answers or the deadline elapses.
Decision (resolves review Issue 1): require the answer to come from the primary, not merely "a primary name was reported by some node." This avoids a false success on a standby's stale primary_host after a primary death (the false-positive class removed from the durability soak) and faithfully meets Req 9.1's "serving primary." The any-node hint view remains available for the operator ctrlstatus command via xcecli_ctrlstatus.
Preserved (9.4): the timeout-argument parsing, monotonic deadline, poll interval, log-level quieting, and success/failure exit codes are unchanged; only the probe call and the success message change.

Data Models

The only new data is the reply payload above. CtrlStatus is a flat value object (no persistence, no aggregate); CtrlPeer is a bounded value list (≤ configured controller count, typically 3). configChecksum is the loaded config's 32-byte SHA-256 and configVersion is its own 8-byte prefix (a self-consistent pair); committedConfigVersion is the separately-tracked committed tag. Two nodes with identical loaded config content produce identical configVersion + configChecksum; differing content differs in both (5.3). A node mid config-change may show configVersion != committedConfigVersion.

Error Handling

No reachable controller (7.3): xcecli_ctrlstatus returns the failover wrapper's typed negative (e.g. no-master/timeout); cmd_ctrlstatus prints it and returns non-zero (8.3); wait-primary keeps polling until its deadline (9.3).
Undeterminable field (1.3): emit the defined "unknown"/empty value, never uninitialized data.
Standby with no known primary (2.3): primaryName empty → "unknown"; the call still succeeds (the status itself is valid).
Read-only guarantee (1.2): the handler takes only read locks/atomic loads; no state mutation on any path.

Testing Strategy

Unit / handler

handle_ctrlstatus on a PRIMARY returns primaryName==self, responderRole==PRIMARY, populated peers/version/checksum (2.1, 3.x, 4.x, 5.x).
handle_ctrlstatus on a STANDBY with a known primary returns that primary and responderRole==STANDBY; with no known primary returns empty primaryName (2.2, 2.3).
fenced reflects in_catch_up/recovery_unrecoverable (6.3); the handler mutates no state (1.2).
configVersion/configChecksum are a self-consistent loaded pair — equal across two nodes with identical loaded config, differ when content differs; committedConfigVersion is exposed separately (5.1, 5.2, 5.3).

Integration

Dispatch-gate: a ctrlstatus request is answered (not -XCE_ERR_STANDBY/-XCE_ERR_NOT_READY) on STANDBY, ELECTING, INIT, and PRIMARY-no-quorum nodes; no other opcode's gating changed (7.1, 7.2).
libxce round-trip: xcecli_ctrlstatus decodes all fields; is_write=false rotates across configured controllers; no-reachable-controller returns a typed failure (7.3).
Authoritative read: xcecli_ctrlstatus_primary, when the first answer is from a standby, follows the redirect to the primary and returns a reply with responderRole == PRIMARY; it never returns a standby/hint reply as the authoritative result (9.1, 9.2).

E2E (live `/xce-deploy-ha`)

ctrlstatus against a healthy 3-node cluster prints the primary, peers, version+checksum; querying a standby still reports the primary with answered by <standby> (2.3, 8.1).
wait-primary (authoritative probe) returns immediately on a healthy cluster and names the primary; after a primary kill, blocks until the new primary itself answers, then names it; does NOT false-succeed on a standby's stale primary view; times out when no primary serves (9.1–9.4). Directly useful to the ha-commit-durability Phase 6 liveness/>60s-election investigation.

Security Considerations

ctrlstatus exposes cluster topology, role, term, and config version/checksum — the same trust level as the existing clusterconf/state-report reads (no secrets, no config content, no job data). It is read-only and goes through the same authenticated client path as other client ops; no new authorization surface is introduced. Broadening the dispatch gate is scoped strictly to this one read opcode.

Design Delta — Requirements 10 & 11 (2026-06-20)

Merge note (as-built correction). The handler shipped as ctrlstatus_fill + handle_ctrlstatus in src/controller/ctrlstatus.c (not reply.c); configChecksum is carried as Text lowercase-hex (the Data capnp type maps to the project's structured xce_data_t); primaryName is role-derived (primary → self, else empty) after E2E fix ade0081 — ha->primary_host is vestigial. The Req 10/11 changes are written against this as-built reality; the Reqs 1–9 design above stands as the original record.

What changed and why

Live E2E (2026-06-20): a standby's ctrlstatus reported committedLsn: 0 / committedConfigVersion: 0 while appliedLsn: 38 — an apparent commit/apply inversion — because ctrlstatus_fill reads the primary-only markers persist.committed_lsn (ctrlstatus.c:130) and ha->committed_config_tag (:109), which stay 0 on a node that never led. And there was no responder timestamp to line a captured status up against ctrl-N.log. Req 10 fixes the off-primary committed sourcing; Req 11 adds a responder snapshot timestamp.

Boundary Commitments (delta)

This spec owns (added): the off-primary field-sourcing fix inside ctrlstatus_fill (read the learned committed values when not primary); a new read-only CtrlStatus.responderTimeMs field stamped at snapshot time; its render in cmd_ctrlstatus.
Out of Boundary (firm): making the off-primary committed value live-fresh — i.e. consuming HeartbeatMsg.committedLsn / extending OP_nop to refresh last_known_committed_lsn on every heartbeat — is a heartbeat/replication-protocol change. It violates Req 10.4 and is routed to ha-heartbeat-liveness / ha-repl-hardening. controller-status reports the honest best-known value only. (Note: superseded by ha-heartbeat-committed-propagation — that spec implemented exactly this, adding ha_consume_heartbeat_committed/ha_observe_heartbeat_committed, invoked from the primary's heartbeat in cluster.c, which now refreshes last_known_committed_lsn/last_known_committed_config_tag on every heartbeat, term-guarded and monotonic. See that spec's design.md.)
Allowed Dependencies (added): read access to ha->last_known_committed_lsn (_Atomic uint64_t, ha.h:183) and ha->last_known_committed_config_tag (ha.h:185); clock_gettime(CLOCK_REALTIME); the existing xcelog.h wall-clock format for client rendering. (Note: last_known_committed_config_tag is now also _Atomic uint64_t — migrated by ha-heartbeat-committed-propagation from the plain uint64_t this design was written against.)
Revalidation Triggers (added): any change to where/when last_known_committed_lsn/last_known_committed_config_tag are written (today: cluster_apply_ctrlhello_reply at join/RDHUP, ha_observe_heartbeat_committed on every heartbeat, and the election safe-commit path) → re-check the off-primary committed reporting and the staleness caveat. (Note: the heartbeat writer was added by ha-heartbeat-committed-propagation, superseding the "join/RDHUP only" premise below.)

Locked design decisions (from research items)

Req 10 sourcing → role-branch (not uniform max()). When responderRole == CTRL_PRIMARY, keep the node's own authoritative live markers (persist.committed_lsn, ha->committed_config_tag). Otherwise report the learned pair (ha->last_known_committed_lsn, ha->last_known_committed_config_tag). Rationale: a standby's learned value is its only honest "committed" knowledge; a primary's own marker is authoritative and must not be shadowed by a possibly-stale learned value. (Mirrors the last_known_* read precedent at ha.c:5319 (ha_vote_self_freshness), ha.c:6732.)
Req 10 staleness caveat (documented behavior, not a bug). last_known_committed_lsn is refreshed at CtrlHello join / RDHUP-failover, not per heartbeat, so an off-primary committed value MAY lag the live primary between joins. This is the in-boundary contract; live-fresh is the out-of-boundary item above. The 0-inversion is eliminated regardless (a standby that has joined has a non-zero learned value; a never-joined node correctly reports 0 = unknown, per Req 1.3 / 10.2). (Note: superseded by ha-heartbeat-committed-propagation — last_known_committed_lsn/_config_tag are now also refreshed on every heartbeat via ha_observe_heartbeat_committed, so the join/RDHUP-only staleness window described here no longer applies; see that spec's design.md.)
Req 11 wire type → UInt64 epoch-milliseconds. Numeric, locale-safe; the client formats it. (Pre-formatted Text was the alternative; rejected to keep the wire value machine-comparable and avoid encoding a locale into the reply.)
Req 11 clock → CLOCK_REALTIME, stamped in ctrlstatus_fill at snapshot time. NOT CLOCK_MONOTONIC (which ha_on_heartbeat_received uses for liveness) — monotonic is not wall-clock and cannot be correlated to logs. The client renders the millis to local YYYY-MM-DD HH:MM:SS,mmm using the xcelog.h convention so it is directly comparable to a log line.

Research item carried to implementation

Confirm the exact wall-clock formatter/helper in src/include/xcelog.h and reuse it (or match its YYYY-MM-DD HH:MM:SS,mmm form) in cmd_ctrlstatus so the printed timestamp is byte-comparable to ctrl-N.log. last_known_committed_config_tag was non-atomic at design time (written under lock in ha_convergence.c); the status read was planned as a best-effort aligned 8-byte load (acceptable for a reporting snapshot) — do not add locking on the read path. (Note: superseded by ha-heartbeat-committed-propagation, which migrated the field to _Atomic uint64_t at ha.h:185, written via atomic_store_explicit at ha_convergence.c:368 and ha.c:9011; the as-built ctrlstatus_fill correctly reads it with atomic_load_explicit, not a bare aligned load.)

Schema delta (Cap'n Proto)

Append one field to CtrlStatus (next free ordinal @13, after peerCount @12; append-only, never reorder):

  responderTimeMs   @13: UInt64 $C.mapname("responder_time_ms");  # CLOCK_REALTIME epoch millis at snapshot (Req 11.1)

No change to committedLsn @8 / committedConfigVersion @7 wire shape — Req 10 changes only the value source, not the field.

`ctrlstatus_fill` delta — Service contract (`src/controller/ctrlstatus.c`)

Req 10 (committed sourcing): replace the unconditional reads at ctrlstatus.c:108–110,130 with the role-branch above:
- responderRole == CTRL_PRIMARY → committedLsn = persist.committed_lsn, committedConfigVersion = ha->committed_config_tag (atomic acquire loads, unchanged).
- else → committedLsn = ha->last_known_committed_lsn (atomic acquire), committedConfigVersion = ha->last_known_committed_config_tag (plain aligned load).
- appliedLsn (ha->applied_lsn) is unchanged — it is already correct in every role.
Req 11 (timestamp): stamp st->responder_time_ms from clock_gettime(CLOCK_REALTIME) (tv_sec*1000 + tv_nsec/1e6) at the top of the snapshot, before/with the other field loads, so it marks the responder's sample moment.
Read-only invariant (10.4, 1.2) preserved: all additions are atomic/aligned loads + a clock read; no HA/persist/config mutation.

`cmd_ctrlstatus` delta — State contract (`src/controller/testsuite/test.c`)

Print one added line rendering responder_time_ms as local YYYY-MM-DD HH:MM:SS,mmm (xcelog.h form) alongside the existing fields (Req 11.3). The committed lines need no rendering change — Req 10 only changes the values they already print.

File Structure Plan (delta)

Modified files:

src/schema/controller.capnp — add responderTimeMs @13: UInt64 to CtrlStatus (regen controller.capnp.h/.c → xce_ctrlstatus_t.responder_time_ms).
src/controller/ctrlstatus.c — ctrlstatus_fill: role-branch committed sourcing (Req 10) + CLOCK_REALTIME timestamp stamp (Req 11).
src/controller/testsuite/test.c — cmd_ctrlstatus: render the responder timestamp line (Req 11.3).
src/controller/testsuite/test_ctrlstatus.c — add/extend unit tests (below).

No new files (the capnp field is generated; all logic slots into existing TUs).

Requirements Traceability (delta)

Requirement	Summary	Components	Interfaces
10.1	Off-primary reports learned committed LSN + config version	`ctrlstatus_fill` role-branch	`CtrlStatus.committedLsn`, `committedConfigVersion` (value source = `ha->last_known_committed_*`)
10.2	No learned value yet → 0/unknown	`ctrlstatus_fill`	`last_known_committed_lsn == 0` (correct unknown, per 1.3)
10.3	No spurious applied>committed inversion	`ctrlstatus_fill`	`appliedLsn` vs role-correct `committedLsn`
10.4	Read-only reporting; no production/replication change	`ctrlstatus_fill`; out-of-boundary routing	atomic/aligned loads only
11.1	Responder wall-clock at sample time	`ctrlstatus_fill` (`CLOCK_REALTIME`)	`CtrlStatus.responderTimeMs`
11.2	Same clock/format as logs	`cmd_ctrlstatus` render (xcelog.h form)	local `YYYY-MM-DD HH:MM:SS,mmm`
11.3	Test client prints the timestamp	`cmd_ctrlstatus`	added print line

Testing Strategy (delta)

Unit (test_ctrlstatus.c, TDD red→green):

STANDBY with last_known_committed_lsn = 36, persist.committed_lsn = 0, applied_lsn = 38 → reply reports committedLsn == 36 (not 0) and appliedLsn == 38 (no inversion below the learned value) (10.1, 10.3).
STANDBY pre-join (last_known_committed_lsn == 0) → reply reports committedLsn == 0 / committedConfigVersion == 0 (correct unknown) (10.2).
PRIMARY → reply reports committedLsn == persist.committed_lsn and committedConfigVersion == ha->committed_config_tag (authoritative own markers unchanged) (10.1 primary branch).
committedConfigVersion off-primary equals last_known_committed_config_tag (symmetry with the LSN) (10.1).
responder_time_ms is non-zero and within a tolerance of a test-captured CLOCK_REALTIME sample; the handler still mutates no state (11.1, 1.2/10.4).

Integration / round-trip (libxce): xcecli_ctrlstatus decodes the new responderTimeMs field end-to-end (no manual decode — codegen), proving wire compatibility.

E2E (live /xce-deploy-ha): reproduce the original finding — submit jobs so the primary commits, query a standby; assert the standby's committedLsn is non-zero and tracks (within the join-refresh caveat) rather than 0, and that the printed responder timestamp matches the standby's own ctrl-N.log line for that query. (Note: the "join-refresh caveat" is narrower than described above — ha-heartbeat-committed-propagation added per-heartbeat refresh, so tracking is now near-live rather than join/RDHUP-only; see that spec's design.md.)

33 KiB Raw Blame History Unescape Escape

Design Document — controller-status (ctrlstatus)