myxce/.kiro/specs/controller-status/requirements.md

Requirements Document

Introduction

This feature adds a controller status query (an XCE analog of LSF's lsid): a single, cheap request that returns basic cluster/controller identity and health — which node is the current primary, the answering node's identity and role, the configured controllers (peers), the committed configuration version and a content checksum, and a few HA health/progress fields. Today no such query exists: a client must infer "is there a serving primary" indirectly (for example, by listing jobs), and there is no single call that reports who the primary is, the configured controller topology, or the committed configuration version. This makes operational visibility and failover-aware test orchestration awkward.

The status query is served by any controller node — a standby reports who the primary is from the state it already tracks — and rides the existing client failover behavior, so a caller can get status without knowing which node is primary. The test client also gains a command that prints this status, and its wait-primary probe switches to this query so that waiting for a primary is cheaper and additionally reports which node became the serving primary.

Boundary Context

• In scope: a client-observable controller status query returning the current primary identity, the answering node's identity and role, the configured controller topology (peers + count), the committed configuration version and content checksum, and HA health (current term, commit/apply progress, serving/fenced state, cluster name); off-primary fidelity of the reported committed positions (a standby reports the committed position it has learned from the primary, not an unset node-local value); a responder snapshot timestamp for log correlation; failover-aware delivery of that query; a test-client command that prints the status; and wait-primary using the status query as its probe.
• Out of scope: an operator-facing CLI/REST status surface (a later pass); any change to HA election, replication, or commit behavior (this query only reads existing controller state); presentation/output formats beyond the query's structured reply and the test client's human-readable print.
• Adjacent expectations: relies on the existing client master-rotation/failover already used by other read operations; consumes the controller's existing HA state (current primary, role, term, committed configuration version, commit/apply progress, fenced state) without changing how that state is produced or maintained. The status query is a focused identity/health view and is intentionally distinct from the existing full cluster-configuration listing (which enumerates resources, queues, and workers).

Requirements

Requirement 1: Single-call controller status query

Objective: As an operator or test author, I want to retrieve basic controller and cluster status in a single request, so that I can see cluster identity and health without inferring it from unrelated operations.

Acceptance Criteria

1. When a client issues a controller status query, the Controller shall return, in a single request/response exchange, a response containing the status fields defined in Requirements 2 through 6.
2. The Controller shall treat the status query as read-only and shall not modify any cluster or controller state as a result of answering it.
3. If the responding controller cannot determine the value of a status field, then the Controller shall return a well-defined "unknown" value for that field rather than omitting it or returning uninitialized data.
4. The status response shall be limited to controller/cluster identity and health fields and shall not include the full resource, queue, or worker inventory.

Requirement 2: Current primary identity

Objective: As an operator, I want the status response to identify the current primary, so that I know which node is authoritative.

Acceptance Criteria

1. While a primary is currently elected and serving, when a client issues a status query, the Controller shall report the current primary's identity in the response.
2. If no primary has currently been elected, then the Controller shall report the primary identity as "unknown" rather than a stale or guessed value.
3. When the status query is answered by a standby node that knows the current primary, the Controller shall report that primary's identity; if the standby does not currently know a primary, the Controller shall report the primary identity as "unknown".

Requirement 3: Responding node identity and role

Objective: As an operator, I want to know which node answered and its role, so that I can interpret the response in context without confusing it with the authoritative cluster view.

Acceptance Criteria

1. When a client issues a status query, the Controller shall report the identity (name) of the node that answered the query.
2. When a client issues a status query, the Controller shall report the answering node's current role as one of: primary, standby, electing, or initializing.
3. The responding node's role shall be presented as a property of the answering node (for example, "answered by <node> (<role>)") and shall be distinct from the authoritative current-primary identity of Requirement 2, so that a transient answering-node role (such as electing or initializing) is not misread as the cluster's primary state.

Requirement 4: Configured controller topology

Objective: As an operator, I want the configured controllers, so that I can see the cluster's intended membership and size.

Acceptance Criteria

1. When a client issues a status query, the Controller shall report the list of configured controllers, each identified by its name and network address.
2. When a client issues a status query, the Controller shall report the total count of configured controllers.

Requirement 5: Committed configuration version and checksum

Objective: As an operator, I want the configuration version and a content checksum, so that I can confirm which configuration the cluster has committed and detect content divergence between nodes even when their version numbers match.

Acceptance Criteria

1. When a client issues a status query, the Controller shall report the committed configuration version currently in effect on the responding node.
2. When a client issues a status query, the Controller shall report a checksum (CRC) of the committed configuration content currently in effect on the responding node.
3. The reported configuration checksum shall be derived from the configuration content such that two nodes with identical committed configuration content report the same checksum, and two nodes with differing content report different checksums (enabling detection of same-version content divergence).

Requirement 6: HA health and progress fields

Objective: As an operator, I want basic HA health and progress, so that I can assess election state and catch-up.

Acceptance Criteria

1. When a client issues a status query, the Controller shall report the current term (election epoch).
2. When a client issues a status query, the Controller shall report its commit progress position and its apply progress position.
3. While the responding node is in a fenced or non-promotable state, when a client issues a status query, the Controller shall report that fenced state in the response.
4. When a client issues a status query, the Controller shall report the cluster name.

Requirement 7: Failover-aware, any-node delivery

Objective: As a client, I want the status query to reach a reachable controller automatically, so that I can get status even across failover or when I do not know which node is primary.

Acceptance Criteria

1. When a client issues a status query, the XCE client shall attempt the configured controllers using the same failover/rotation behavior used by other read operations until one responds or the attempts are exhausted.
2. When any configured controller, whether primary or standby, is reachable, the XCE client shall return that controller's status response.
3. If no configured controller is reachable within the client's failover budget, then the XCE client shall report a typed failure rather than reporting success with empty status.

Requirement 8: Test-client status command

Objective: As a test author, I want a client command that prints the controller status, so that I can inspect cluster identity and health from a script.

Acceptance Criteria

1. When the status command is issued in the test client, the test client shall print, in human-readable form, the current primary identity, the answering node's identity and role (as "answered by"), the configured controllers and their count, the committed configuration version, the committed configuration checksum, the current term, the commit/apply progress, the fenced state, and the cluster name.
2. When the underlying status query succeeds, the test client command shall report success (zero status).
3. If the underlying status query fails, then the test client command shall report failure (non-zero status) and shall print the error.

Requirement 9: wait-primary uses the status query

Objective: As a test author, I want wait-primary to probe via the status query, so that the wait is cheaper and reports which node became the serving primary.

Acceptance Criteria

1. When wait-primary probes the cluster, the wait-primary command shall use the controller status query, rather than a job-listing operation, to determine whether a serving primary is present.
2. When the status query reports a serving primary within the timeout, the wait-primary command shall complete successfully and shall report the identity of the serving primary.
3. If no serving primary is reported within the timeout, then the wait-primary command shall fail.
4. The change to the wait-primary probe shall preserve its existing timeout-argument handling and its success/failure exit semantics, adding only the served-primary identity to its success output.

Requirement 10: Off-primary committed-state fidelity

Objective: As an operator, I want a standby's reported committed positions to reflect what it has actually learned from the primary, so that the status does not show a misleading committed value of zero while the node has clearly applied replicated entries.

Acceptance Criteria

1. While the responding node is not the current primary and has learned a committed position from the primary, when a client issues a status query, the Controller shall report the committed configuration version and commit progress position that it has learned from the primary, rather than a node-local committed value that remains unset on a node that has never served as primary.
2. If the responding node is not the current primary and has not yet learned any committed position from a primary, then the Controller shall report the committed configuration version and commit progress position as the well-defined "unknown"/zero value (consistent with Requirement 1.3), rather than a stale or guessed value.
3. While the responding node is actively applying replicated entries, the Controller shall source the reported commit progress position and apply progress position consistently, so that the reported commit position is not spuriously lower than the position the node has actually committed or learned — avoiding an apparent "applied greater than committed" inversion that does not reflect the node's real state.
4. The committed-state fidelity in this requirement shall be a read-only reporting concern only and shall not change how committed positions are produced, replicated, or persisted.

Requirement 11: Responder snapshot timestamp for log correlation

Objective: As an operator or test author, I want each status response to carry the responding node's clock time at the moment it sampled its state, so that I can line a captured status up against that node's logs.

Acceptance Criteria

1. When a client issues a status query, the Controller shall include in the response the responding node's wall-clock time at the moment the status fields were sampled.
2. The reported timestamp shall be expressed on the same clock and in a form that an operator can directly compare against the responding controller's log timestamps.
3. When the status command is issued in the test client, the test client shall print the responder snapshot timestamp in human-readable form alongside the other status fields.