lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/.kiro/specs/controller-status/design.md

19 KiB
Raw Blame History

Design Document — controller-status (ctrlstatus)

Overview

Purpose: add a single, cheap, read-only client→controller query — ctrlstatus, an XCE analog of LSF lsid — that returns basic cluster/controller identity and health: who is the current primary, the answering node's identity and role, the configured controllers (peers), the committed configuration version and content checksum, and a few HA health/progress fields (term, commit/apply position, fenced state, cluster name).

Users: operators inspecting a running cluster, and test/E2E authors orchestrating failover (including wait-primary, which switches its probe to this query).

Impact: today there is no status query — callers infer "is there a serving primary" indirectly (e.g. by listing jobs). This adds a new opcode by cloning the existing opcode-only read pattern of clusterconf, reads only already-maintained HA/config state (no behavior change), and crucially makes the query answerable on any node by adding it to the HA dispatch-gate allow-lists.

Goals

  • A new read-only opcode returning the status fields of Requirements 2–6 in one request/response (1.1, 1.4).
  • Answerable on any node and across failover — a standby reports the primary it knows (2.3, 7.1–7.3).
  • Report a content checksum of the committed configuration so same-version divergence is detectable (5.2, 5.3).
  • A ctrlstatus test-client command (8.x) and a wait-primary probe switched to this query (9.x).

Non-Goals

  • No operator-facing CLI/REST lsid surface (Go tier) — later pass.
  • No change to HA election, replication, or commit behavior — the query only reads existing state.
  • No new configuration checksum computation — the committed-config SHA-256 already exists; this only exposes it.

Boundary Commitments

This Spec Owns

  • A new Cap'n Proto opcode ctrlstatus and its reply payload (CtrlStatus, CtrlPeer).
  • The controller handler handle_ctrlstatus that snapshots local HA/config state into the reply.
  • The dispatch-gate allow-list entries that let ctrlstatus answer in every controller role (incl. no-quorum).
  • The libxce client API (xce_ctrlstatus_req / xce_ctrlstatus / xcecli_ctrlstatus + free helper).
  • The ctrlstatus test-client command and the wait-primary probe switch.

Out of Boundary

  • The Go CLI/REST lsid surface (later spec).
  • Any change to how HA state (role, term, primary identity, committed config tag, commit/apply LSN, fence) is produced or maintained — this spec only reads it.
  • The existing clusterconf opcode and its resource/queue/worker payload (left unchanged).

Allowed Dependencies

  • The Cap'n Proto schema + build-time codegen (src/schema/controller.capnp → generated controller.capnp.h/.c).
  • The existing request/reply plumbing (init_reply_by_req, xce_reply, opcode-only *_req senders) and client failover wrapper (XCECLI_FAILOVER_CALL, is_write=false).
  • Read access to existing HA state (ha_state_t), config (myconf_t: committed_config_tag, config_sha256, peers_cfg), and persist accessors (persist_recovery_unrecoverable, persist_durable_lsn).

Revalidation Triggers

  • Any change to the CtrlStatus/CtrlPeer reply shape or the ctrlstatus opcode ordinal → re-check libxce decode + the test command + any future consumer.
  • Any change to the dispatch-gate role allow-lists → re-check that ctrlstatus still answers in every role and that no other opcode's gating widened.
  • A change to the meaning of committed_config_tag / config_sha256 → re-check the version/checksum fields.

Architecture

Existing Architecture Analysis

  • Opcode-only read precedent: clusterconf is sent with no request body (Request_body_nomsg), dispatched in cluster.c switch(request.opcode), handled by handle_clusterconf (reply.c:10), and returned via xce_reply. The client side is xce_clusterconf → xcecli_clusterconf (XCECLI_FAILOVER_CALL, is_write=false). ctrlstatus clones this pattern.
  • No-alloc state-report precedent: ha_nop_reply_state_report (ha.c:1784) reads HA atomics into a stack struct and points the reply union at it without heap allocation. handle_ctrlstatus follows this model (a small bounded peer list is the only allocation).
  • Dispatch gate (the critical seam): ha_dispatch_gate (ha.c:2059) is an allow-list keyed by role. The CTRL_STANDBY branch defaults to -XCE_ERR_STANDBY; CTRL_INIT/CTRL_ELECTING default to -XCE_ERR_NOT_READY; the CTRL_PRIMARY no-quorum branch allow-lists only specific reads. clusterconf is not in any standby allow-list — it is implicitly primary-only-when-serving. A status query must not inherit that; it must be explicitly allow-listed in every role branch.
  • Config content checksum already exists: config_version_tag = first 8 bytes of SHA-256 over the committed config blob; full digest in config_sha256[32], round-trip-verified on load (config_repl.c).

Pattern & dependency direction

Extend-in-place along the existing request path. Dependency direction (unchanged): schema → libxce (request/reply/client) → controller handler; the test client depends on libxce. No upward dependencies introduced.

graph LR
    TestClient[xce_test ctrlstatus / wait-primary] --> Libxce[libxce xcecli_ctrlstatus]
    Libxce --> Failover[XCECLI_FAILOVER_CALL is_write false]
    Failover --> Gate[ha_dispatch_gate allow-list all roles]
    Gate --> Handler[handle_ctrlstatus snapshot local state]
    Handler --> HAState[ha_state_t and myconf_t read-only]
    Handler --> Reply[xce_reply CtrlStatus]

File Structure Plan

Modified Files

  • src/schema/controller.capnp — add OPCode ctrlstatus @40; new struct CtrlStatus $C.mapname("xce_ctrlstatus_t") and struct CtrlPeer $C.mapname("xce_ctrlpeer_t"); reply-union member ctrlstatus @17: CtrlStatus. (Build regenerates controller.capnp.h/.c → xce_ctrlstatus_t, RequestReply_data_ctrlstatus, free_CtrlStatus.)
  • src/controller/reply.c — add handle_ctrlstatus (snapshot local HA/config state into a CtrlStatus reply; small peer list; xce_reply). Mirrors handle_clusterconf/ha_nop_reply_state_report.
  • src/controller/cluster.c — add case OPCode_ctrlstatus: rc = handle_ctrlstatus(...) to the opcode switch.
  • src/controller/ha.c — add OPCode_ctrlstatus to the ha_dispatch_gate allow-lists for CTRL_PRIMARY (incl. the no-quorum branch), CTRL_STANDBY, CTRL_ELECTING, CTRL_INIT. (the requirements-critical change)
  • src/lib/request.c + src/include/request.h — add xce_ctrlstatus_req (opcode-only sender, clone of xce_clusterconf_req).
  • src/lib/xceapi.c — add xce_ctrlstatus (send req, cbuf_get status, MOVE_PTR reply union).
  • src/lib/client.c + src/include/client.h — add xcecli_ctrlstatus (XCECLI_FAILOVER_CALL, is_write=false) and the xcecli_free_ctrlstatus macro.
  • src/controller/testsuite/test.c — add cmd_ctrlstatus + table registration; switch cmd_wait_primary's probe (currently joblist_query(...) at the deadline-loop call site) to xcecli_ctrlstatus.

New Files

  • None (capnp C types are generated; all handlers/APIs slot into existing TUs).

System Flows

Status query (any role) + dispatch gate

sequenceDiagram
    participant C as Client
    participant F as Failover wrapper
    participant N as Controller node
    participant G as ha_dispatch_gate
    participant H as handle_ctrlstatus
    C->>F: xcecli_ctrlstatus
    F->>N: ctrlstatus request (opcode-only)
    N->>G: gate(opcode, role)
    G-->>N: ALLOW (all roles, incl no-quorum)
    N->>H: snapshot local state
    H-->>C: CtrlStatus reply
    Note over C,N: On a standby, primaryName = its known primary (or "unknown"); responderRole = STANDBY

Gate decision: ctrlstatus is allow-listed in every role branch and does not require quorum or a committed read — it returns the node's local view even when fenced or without quorum (precisely when an operator most wants it).

wait-primary probe (Req 9)

wait-primary keeps its existing monotonic-deadline poll loop and timeout/exit semantics; only the probe changes from joblist_query to xcecli_ctrlstatus. Success predicate: the status reply reports a non-"unknown" current primary. On success it prints the served primary's identity. (Rationale + caveat below.)

Requirements Traceability

Requirement Summary Components Interfaces
1.1, 1.2, 1.4 Single-call read-only focused status handle_ctrlstatus, CtrlStatus schema opcode ctrlstatus, reply CtrlStatus
1.3 "unknown" for undeterminable fields handle_ctrlstatus empty Text/sentinel mapping
2.1, 2.2, 2.3 Current primary identity (unknown when unelected; standby's known primary) handle_ctrlstatus (reads ha->primary_host under ha->lock) CtrlStatus.primaryName
3.1, 3.2, 3.3 Answered-by node + role, distinct from primary handle_ctrlstatus CtrlStatus.responderName, responderRole
4.1, 4.2 Configured controllers + count handle_ctrlstatus (reads peers_cfg) CtrlStatus.peers, peerCount
5.1, 5.2, 5.3 Config version + content checksum handle_ctrlstatus (reads committed_config_tag, config_sha256) CtrlStatus.configVersion, configChecksum
6.1–6.4 Term, commit/apply, fenced, cluster name handle_ctrlstatus CtrlStatus.term, committedLsn, appliedLsn, fenced, clusterName
7.1, 7.2, 7.3 Failover-aware any-node delivery xcecli_ctrlstatus, dispatch-gate allow-lists XCECLI_FAILOVER_CALL (is_write=false), gate entries
8.1, 8.2, 8.3 Test-client ctrlstatus command cmd_ctrlstatus client command + exit code
9.1–9.4 wait-primary uses the status query cmd_wait_primary probe swap xcecli_ctrlstatus

Components and Interfaces

Component Layer Intent Req Coverage Key Dependencies Contracts
CtrlStatus/CtrlPeer schema Schema Reply payload for the status query 2–6 capnp codegen (P0) API (reply)
handle_ctrlstatus Controller Snapshot local HA/config state into the reply 1–6 ha_state_t, myconf_t, persist accessors (P0) Service
dispatch-gate allow-lists Controller HA Permit ctrlstatus in every role 7 ha_dispatch_gate (P0) Service
xce_ctrlstatus / xcecli_ctrlstatus libxce Failover-aware client call + free 1, 7 failover wrapper (P0) Service
cmd_ctrlstatus Test client Print the status 8 xcecli_ctrlstatus (P0) State
cmd_wait_primary probe Test client Use status as the serving-primary probe 9 xcecli_ctrlstatus (P0) State

Schema — reply payload (Cap'n Proto)

struct CtrlPeer $C.mapname("xce_ctrlpeer_t") {
  name @0: Text;
  host @1: Text;
  port @2: UInt32;
}

struct CtrlStatus $C.mapname("xce_ctrlstatus_t") {
  clusterName     @0:  Text;
  primaryName     @1:  Text;    # empty ⇒ "unknown" (no elected/known primary)
  responderName   @2:  Text;    # the node that answered
  responderRole   @3:  UInt8;   # ctrl_role_t: INIT/ELECTING/PRIMARY/STANDBY
  term            @4:  UInt64;
  configVersion   @5:  UInt64;  # committed_config_tag (8-byte SHA-256 prefix)
  configChecksum  @6:  Data;    # full committed-config SHA-256 (32 bytes)
  committedLsn    @7:  UInt64;
  appliedLsn      @8:  UInt64;
  fenced          @9:  Bool;    # in_catch_up OR recovery_unrecoverable
  peers           @10: List(CtrlPeer) $C.mapname("peers.list") $C.maplistcount("peers.count");
  peerCount       @11: UInt32;
}
  • New OPCode ordinal: ctrlstatus @40 (next free after preVoteReply @39). New reply-union member: ctrlstatus @17 (next free after walackreplymsg @16). Field ordinals are append-only — never reuse/reorder.

handle_ctrlstatus — Service contract

Field Detail
Intent Build a CtrlStatus from the responding node's local HA/config state and reply
Requirements 1.1–1.4, 2.1–2.3, 3.1–3.3, 4.1–4.2, 5.1–5.3, 6.1–6.4

Responsibilities & Constraints

  • Read-only: must not modify any HA, persist, or config state (1.2).
  • Field sourcing: clusterName/responderName from c->myconf/own peerinfo; responderRole, term, appliedLsn, committedLsn, committed_config_tag via atomic_load_explicit(..., acquire); primaryName/peers as below; fenced = in_catch_up || persist_recovery_unrecoverable(...); configChecksum from config_sha256[32].
  • Locking: primary_host/primary_port are guarded by ha->lock — take it for a consistent snapshot, copy out, release. All other fields are atomics or immutable config.
  • "unknown" mapping (1.3, 2.2, 2.3): if primary_host[0] == '\0' (no known primary), emit an empty primaryName, which the client renders as "unknown".
  • Peers (4.x): iterate c->myconf->peers_cfg.candidates[0..count-1] → one CtrlPeer{name,host,port} each; set peerCount.
  • Answerable in every role; returns the local view even when fenced/no-quorum.

Dependencies

  • Inbound: cluster.c opcode switch — routes the request (P0).
  • Outbound: ha_state_t, myconf_t, persist accessors — read-only snapshot (P0).
  • Outbound: init_reply_by_req, xce_reply — reply encode/send (P0).

Dispatch-gate allow-lists — Service contract

  • Add OPCode_ctrlstatus to the allow-list in ha_dispatch_gate for CTRL_PRIMARY (including the no-quorum branch), CTRL_STANDBY, CTRL_ELECTING, and CTRL_INIT (7.1, 7.2).
  • Constraint: the change is scoped strictly to ctrlstatus; no other opcode's gating may widen. ctrlstatus never requires quorum or a committed read.

libxce client API — Service contract

/* opcode-only request sender (clone of xce_clusterconf_req) */
int  xce_ctrlstatus_req(endpoint_t* ep);
/* send + await reply; on success MOVE_PTR the reply union out */
int  xce_ctrlstatus(xce_ctx_t* ctx, xce_ctrlstatus_t** status);
/* failover-aware read (is_write=false) */
int  xcecli_ctrlstatus(xcecli_ctx_t c, xce_ctrlstatus_t** status);
#define xcecli_free_ctrlstatus(s)  xce_free_ctrlstatus(s)
  • Preconditions: valid client context.
  • Postconditions: on success *status owns a decoded CtrlStatus (caller frees); on failure returns a typed negative (7.3) and *status untouched.
  • Invariants: is_write=false — retryable/rotatable across the configured controllers.

cmd_ctrlstatus (test client) — State contract

  • Calls xcecli_ctrlstatus; on success prints the fields in human-readable form (8.1), framing the responder as answered by <responderName> (<role>) distinct from Primary: (3.3); renders empty primaryName as unknown; renders configChecksum as hex. Returns 0 on success, <0 on failure with the error printed (8.2, 8.3). Frees the reply.

cmd_wait_primary probe swap — State contract

  • Replace the single joblist_query(...) probe call in the deadline loop with xcecli_ctrlstatus.
  • Success predicate (9.1, 9.2): the status reply reports a non-"unknown" primaryName. On success, complete and print the served primary's identity.
  • Preserved (9.4): the timeout-argument parsing, monotonic deadline, poll interval, log-level quieting, and success/failure exit codes are unchanged; only the probe call and the success message change.
  • Decision & caveat: the predicate is "a known primary is reported" rather than "the responder is itself the primary," because the failover read may legitimately be answered by a standby; requiring responderRole == PRIMARY could starve the predicate if a standby always answers first. The accepted caveat is a brief window where a standby may report a just-failed primary before it clears; this is acceptable for a "wait until a primary exists" gate and is strictly more informative than the prior job-list probe (it names the primary).

Data Models

The only new data is the reply payload above. CtrlStatus is a flat value object (no persistence, no aggregate); CtrlPeer is a bounded value list (≤ configured controller count, typically 3). configChecksum is the 32-byte committed-config SHA-256; configVersion is its 8-byte prefix as a UInt64. Two nodes with identical committed config content produce identical configChecksum and configVersion; differing content differs in both (5.3).

Error Handling

  • No reachable controller (7.3): xcecli_ctrlstatus returns the failover wrapper's typed negative (e.g. no-master/timeout); cmd_ctrlstatus prints it and returns non-zero (8.3); wait-primary keeps polling until its deadline (9.3).
  • Undeterminable field (1.3): emit the defined "unknown"/empty value, never uninitialized data.
  • Standby with no known primary (2.3): primaryName empty → "unknown"; the call still succeeds (the status itself is valid).
  • Read-only guarantee (1.2): the handler takes only read locks/atomic loads; no state mutation on any path.

Testing Strategy

Unit / handler

  • handle_ctrlstatus on a PRIMARY returns primaryName==self, responderRole==PRIMARY, populated peers/version/checksum (2.1, 3.x, 4.x, 5.x).
  • handle_ctrlstatus on a STANDBY with a known primary returns that primary and responderRole==STANDBY; with no known primary returns empty primaryName (2.2, 2.3).
  • fenced reflects in_catch_up/recovery_unrecoverable (6.3); the handler mutates no state (1.2).
  • configChecksum/configVersion equal across two nodes with identical config, differ when content differs (5.3).

Integration

  • Dispatch-gate: a ctrlstatus request is answered (not -XCE_ERR_STANDBY/-XCE_ERR_NOT_READY) on STANDBY, ELECTING, INIT, and PRIMARY-no-quorum nodes; no other opcode's gating changed (7.1, 7.2).
  • libxce round-trip: xcecli_ctrlstatus decodes all fields; is_write=false rotates across configured controllers; no-reachable-controller returns a typed failure (7.3).

E2E (live /xce-deploy-ha)

  • ctrlstatus against a healthy 3-node cluster prints the primary, peers, version+checksum; querying a standby still reports the primary (2.3, 8.1).
  • wait-primary (status-probe) returns immediately on a healthy cluster and names the primary; after a primary kill, blocks until a new primary is reported, then names it; times out when none (9.1–9.4). Directly useful to the ha-commit-durability Phase 6 liveness/>60s-election investigation.

Security Considerations

ctrlstatus exposes cluster topology, role, term, and config version/checksum — the same trust level as the existing clusterconf/state-report reads (no secrets, no config content, no job data). It is read-only and goes through the same authenticated client path as other client ops; no new authorization surface is introduced. Broadening the dispatch gate is scoped strictly to this one read opcode.