docs(controller-status): add Requirement 13 delta (per-controller connection status)
Adds requirements, design, and research for showing each configured controller's connection status (ok/catching-up/dead/unknown) in ctrlstatus's peer list, with the same off-primary fidelity pattern already established (primary reports the true picture; a standby reports unknown for what it cannot itself observe). Design review found and fixed one gap before implementation: no defined fallback if the responder's own name doesn't match any configured peer entry (a pre-existing misconfiguration class, not new to this delta) -- now an explicit, logged (WARN) degradation instead of a silently wrong status. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
This commit is contained in:
parent
2d7990aaa7
commit
40882c6359
4 changed files with 181 additions and 13 deletions
|
|
@ -409,3 +409,92 @@ No new files (capnp fields are generated; all logic slots into existing TUs) —
|
|||
- Query `ctrlstatus` on the primary right after a fresh election: `electedTerm` equals the printed `term`; `electedAt` is close to "now."
|
||||
- Query a STANDBY: `electedTerm`/`electedAt` match the primary's own values (within one heartbeat interval), not "unknown" and not stale/zero — proving standby learned fidelity end-to-end (12.2).
|
||||
- Kill the primary, wait for a new election, query the surviving standby immediately vs. after one heartbeat interval — demonstrates the field converges promptly via heartbeat, unlike Req 10's original join/RDHUP-only caveat (heartbeat-carried delivery here is in-boundary from day one of this delta, not a follow-up spec).
|
||||
|
||||
---
|
||||
|
||||
## Design Delta — Requirement 13 (2026-09-16)
|
||||
|
||||
### What changed and why
|
||||
User enhancement request: `ctrlstatus`'s "Controllers (N):" list should additionally show each configured controller's connection status (ok / catching-up / dead), not just name and address, so operators can see at a glance which controllers are healthy, rejoining, or unreachable. Confirmed with the user: three-state granularity (reusing the already-tracked connected-vs-catching-up distinction) and best-effort per-responder reporting — "unknown" for peers a responder cannot itself observe, consistent with this spec's established off-primary fidelity pattern, not a primary-only-authoritative requirement. Scope of this delta = Req 13 only. Reqs 1–12 are implemented and unchanged.
|
||||
|
||||
### Boundary Commitments (delta)
|
||||
- **This spec owns (added):** a new `CtrlPeer.status` reply field (per-peer, not per-response — this is the first Req 12+ delta to extend the nested `CtrlPeer` struct rather than `CtrlStatus` itself); `ctrlstatus_fill` populating it per peer, sourced differently by responder role (primary: matched against live connection state; non-primary: self only, "unknown" otherwise); a status-code-to-string renderer (mirroring the existing `responderRole`/`ctrlstatus_role_str` pattern); `cmd_ctrlstatus`'s peer-list print line extended to show it.
|
||||
- **Out of Boundary (firm):** this spec does not change how standby connections are established, monitored, evicted, or torn down — it only reads the connection-tracking state that already exists for replication/quorum purposes (Req 13.5). It does not give a standby visibility into peers it genuinely cannot observe (confirmed at design time: this codebase's connection model is asymmetric — only the primary holds live connections to every standby; standbys do not connect to each other, and no existing mechanism lets a standby identify even its OWN primary by peer identity, only "elapsed time since last primary message") — a standby reporting anything but "unknown" for a peer other than itself would require a genuinely new cross-node identity-carrying mechanism, which is out of scope for this delta (see Locked Decision 3).
|
||||
- **Allowed Dependencies (added):** the existing replication connection registry (`replicator_t.standbys[]`, `src/include/repl.h`) that already tracks each standby's live connection (`ep`), catch-up state (`in_catch_up`), and host/port — read-only, no new writes to it.
|
||||
- **Revalidation Triggers (added):** any change to `standby_conn_t`'s fields or to how/when `ep`/`in_catch_up` are set (`repl_add_standby`/`repl_remove_standby`/`repl_start_catchup`/`repl_complete_catchup`) → re-check the peer-status sourcing still reads the correct signal; any change to `peers_cfg.candidates[]`'s host/port fields → re-check the host+port matching still correlates correctly to `replicator_t.standbys[]`.
|
||||
|
||||
### Locked design decisions
|
||||
1. **Wire type: `CtrlPeer.status @3: UInt8`**, a small fixed-value code — not `Text`. Mirrors the existing `CtrlStatus.responderRole @3: UInt8` precedent exactly (backed by a C enum + a `ctrlstatus_..._str()` string renderer at the client layer), rather than introducing a new stringly-typed convention for this one field.
|
||||
2. **Status values, in ordinal order:** `0 = unknown`, `1 = ok`, `2 = catching-up`, `3 = dead`. `unknown` is `0` (the default/zero value) so a peer entry that nothing ever sets defaults to the safe, honest "can't tell" state rather than an implicitly-true `ok`.
|
||||
3. **Primary-side sourcing: match by host+port, not by name or position.** `standby_conn_t` (the primary's live connection-tracking slot) stores `host`/`port` but not a configured peer name or a stable index into `peers_cfg.candidates[]` — so `ctrlstatus_fill`, when the responder is primary, must scan `replicator_t.standbys[]` for a slot whose `host`/`port` match each configured peer's `hostname`/`port`. No match (or a matched slot with `ep == NULL`) → `dead`; a match with `ep != NULL && in_catch_up` → `catching-up`; a match with `ep != NULL && !in_catch_up` → `ok`. This mirrors `repl_active_standby_count`'s own existing `ep != NULL && !in_catch_up` predicate (`repl.c`) — reuse that exact condition rather than inventing a new one.
|
||||
4. **The responder always reports its own entry as `ok`** (Req 13.2) — matched by comparing each configured peer's name against `st->responderName`, the same identity `ctrlstatus_fill` already establishes for itself.
|
||||
5. **Non-primary responder: only self is determinable; every other peer — including the primary — reports `unknown`.** This was an open question at requirements time (whether a standby could additionally report its own primary's status via the liveness signal it already tracks for failover); design-time investigation closes it: that liveness signal (`ha->last_activity`, `liveness_poll_fn`) is a bare elapsed-time scalar with no peer identity attached, and this codebase has no other mechanism (analogous to the vestigial `ha->primary_host`) that would let a standby reliably attribute it to one specific configured peer. Building one would be materially larger than this delta (a genuinely new identity-carrying signal on the heartbeat/connection-accept path) and is explicitly out of boundary here — a future spec's concern, not silently absorbed into this one.
|
||||
6. **Read-only, no role-branch beyond what Decision 3/5 already requires**: `ctrlstatus_fill` performs only reads (of `peers_cfg`, and — only when the responder is primary — of `replicator_t.standbys[]` under its existing lock); no new state is written, no existing connection-management behavior changes.
|
||||
|
||||
### Research item carried to implementation
|
||||
1. Confirm the exact field path from `cluster_t` to its `replicator_t` (repl.h's own comment says only "embedded in persist_t or cluster_t (P2-4)", ambiguous at design time — likely `c->persist.repl` given `replicator_t.ce` couples it to the commit engine which lives off `persist_t`, but confirm against the current struct definition, not this note, before wiring the read).
|
||||
|
||||
### Constraints checked
|
||||
- **Async I/O invariant (ha-async-io.md):** the new read takes `replicator_t.lock` (a `mylock_t`) briefly to scan a small, fixed-size (`MAX_STANDBYS`) in-memory array — no network I/O, no unbounded wait, identical in shape to the already-established `repl_active_standby_count`'s own lock-and-scan. `ctrlstatus_fill` runs on the request-handling path (not the trigger-only delaymgr thread), so a short, bounded lock acquisition here is consistent with the rest of this handler's existing reads (e.g. the primary-identity read already takes `ha->lock` in the original Req 1-9 design).
|
||||
- **Persistence / save-restore completeness:** not applicable — `CtrlPeer.status` is derived entirely from existing in-memory connection state (`replicator_t.standbys[]`, already not persisted) and existing config (`peers_cfg`); nothing new is written to disk, WAL, or any snapshot format, and there is no save/restore path for this delta to audit.
|
||||
|
||||
### Schema delta (Cap'n Proto)
|
||||
```capnp
|
||||
# CtrlPeer — next free ordinal @3 (after port @2; append-only, never reorder)
|
||||
struct CtrlPeer $C.mapname("xce_ctrlpeer_t") {
|
||||
name @0: Text;
|
||||
host @1: Text;
|
||||
port @2: UInt32;
|
||||
status @3: UInt8 $C.mapname("status");
|
||||
# 0=unknown 1=ok 2=catching-up 3=dead (Req 13.1). Always 1 (ok) for the
|
||||
# responder's own entry (Req 13.2). On a non-primary responder, every
|
||||
# entry but the responder's own is 0 (unknown) — Req 13.4/Locked Decision 5.
|
||||
}
|
||||
```
|
||||
No change to `CtrlStatus`'s own top-level fields — this delta extends the nested `CtrlPeer` list element, not the reply envelope.
|
||||
|
||||
### File Structure Plan (delta)
|
||||
**Modified files:**
|
||||
- `src/schema/controller.capnp` — add `status @3: UInt8` to `CtrlPeer`; regen `controller.capnp.h/.c`.
|
||||
- `src/include/typemap.h` — add `uint8_t status;` to the hand-authored `xce_ctrlpeer_t` struct, matching the schema field.
|
||||
- `src/controller/ctrlstatus.c` — `ctrlstatus_fill`'s peer-building loop: set each `xce_ctrlpeer_t.status` per Locked Decisions 3–5 (self → ok; primary responder → host/port-matched against `replicator_t.standbys[]`; non-primary responder, non-self → unknown). Add a small `ctrlstatus_peer_status_str()` (or similarly-named) helper alongside the existing `ctrlstatus_role_str`, in whichever header/TU that already lives in.
|
||||
- `src/harness/test.c` — `cmd_ctrlstatus`: extend the existing per-peer print line (`printf("\t%s\t%s:%u\n", ...)`) to also render the status string.
|
||||
- `src/controller/testsuite/test_ctrlstatus.c` — extend/add unit tests (below).
|
||||
|
||||
No new files (the capnp field is generated; the small status-code enum and its string renderer slot into an existing header/TU) — consistent with this spec's established pattern.
|
||||
|
||||
### Requirements Traceability (delta)
|
||||
| Requirement | Summary | Components | Interfaces |
|
||||
|-------------|---------|------------|------------|
|
||||
| 13.1 | Report each peer's status as one of a well-defined value set, alongside name/address | `ctrlstatus_fill` | `CtrlPeer.status` (new) |
|
||||
| 13.2 | Responder always reports its own status as ok | `ctrlstatus_fill` (self-match by name) | `CtrlPeer.status == 1` for the self entry |
|
||||
| 13.3 | Primary reports the true status for every other controller | `ctrlstatus_fill` (host/port match against `replicator_t.standbys[]`) | `CtrlPeer.status` sourced from `ep`/`in_catch_up` |
|
||||
| 13.4 | Non-primary reports unknown for what it cannot determine | `ctrlstatus_fill` (non-primary branch) | `CtrlPeer.status == 0` for every non-self entry |
|
||||
| 13.5 | Read-only; no change to connection lifecycle | `ctrlstatus_fill` reads `replicator_t` only, under its existing lock; writes nothing | n/a |
|
||||
| 13.6 | Test client prints each peer's status | `cmd_ctrlstatus` | extended per-peer print line |
|
||||
|
||||
### `ctrlstatus_fill` / peer-status delta — Service contract
|
||||
- **Req 13.1–13.4 (sourcing):** for each `peers_cfg.candidates[i]`, set `status`:
|
||||
- `candidates[i].name == st->responderName` → `1` (ok) — unconditional, regardless of role (Req 13.2).
|
||||
- Else, `responderRole == CTRL_PRIMARY` → scan `replicator_t.standbys[]` (under its lock) for a slot whose `host`/`port` match `candidates[i].hostname`/`.port`; no match or `ep == NULL` → `3` (dead); `ep != NULL && in_catch_up` → `2` (catching-up); `ep != NULL && !in_catch_up` → `1` (ok).
|
||||
- Else (non-primary responder, non-self peer) → `0` (unknown).
|
||||
- **Self-match fallback (design-review finding, 2026-09-16):** the self-match is a straightforward name comparison against `st->responderName`, the same identity `ctrlstatus_fill` already establishes for itself and already expects to be self-consistent with `peers_cfg` (this is a pre-existing assumption of the ORIGINAL Req 1-9 design, not new to this delta — a controller whose own name is absent from its own configured peer list is already a misconfiguration visible today, since it would already be missing an entry from the pre-existing `Controllers (N):` list's identity, unrelated to status). This delta does not need to invent new self-identity infrastructure to handle that pre-existing failure mode, but it MUST NOT let it produce a silently wrong status: if no `candidates[i]` matches `responderName`, `ctrlstatus_fill` shall `WARN` once per call (visible in the responder's own log, consistent with this codebase's existing fail-loud-not-silent convention) and leave every peer's status to fall through to the normal non-self branches above (i.e., no entry is force-corrected to `ok` under this condition) — an explicit, logged degradation rather than an undefined or fabricated result. This keeps Req 13.2's "always ok for self" guarantee intact for every correctly-configured cluster while making a misconfigured one loudly debuggable instead of silently misleading.
|
||||
- **Req 13.5 (read-only) preserved:** only reads `peers_cfg` and, conditionally, `replicator_t.standbys[]` under its existing lock — no new writes anywhere. The `WARN` log above is observability-only, not a state mutation.
|
||||
- **`cmd_ctrlstatus` (Req 13.1, 13.6):** extend the existing `Controllers (N):` per-peer line to also print the rendered status string, e.g. `\t%s\t%s:%u\t%s\n` with the fourth `%s` from the new status-string helper.
|
||||
|
||||
### Testing Strategy (delta)
|
||||
**Unit (`test_ctrlstatus.c`):**
|
||||
- PRIMARY with a full `replicator_t.standbys[]` (one slot `ep != NULL && !in_catch_up`, one `ep != NULL && in_catch_up`, one peer with no matching slot at all) → `ctrlstatus_fill` reports `ok`/`catching-up`/`dead` respectively for those three peers, and `ok` for its own entry (13.1–13.3).
|
||||
- STANDBY responder → reports `ok` for its own entry and `unknown` (0) for every other configured peer, including the primary (13.2, 13.4).
|
||||
- Handler mutates no state — extend the existing `test_ctrlstatus_no_mutation` to also snapshot/compare whatever `replicator_t` fields the primary-branch read touches (13.5).
|
||||
- Status-string renderer: each of the 4 codes renders its expected string; an out-of-range code renders a safe fallback (not garbage), mirroring `ctrlstatus_role_str`'s own convention if it has one.
|
||||
- Self-match fallback (design-review finding): a fixture whose `responderName` matches no `candidates[i].name` (simulating the misconfiguration case) still produces a defined, non-crashing result via the normal non-self branches, and a `WARN` is logged — proving the degradation is loud and deterministic, not silent or undefined (13.2).
|
||||
|
||||
**Integration:**
|
||||
- Wire round-trip: `xcecli_ctrlstatus` decodes the new `CtrlPeer.status` field for every peer in the list (codegen, no manual decode).
|
||||
|
||||
**E2E (live `/xce-deploy-ha`):**
|
||||
- Query `ctrlstatus` on the primary of a healthy 3-node cluster: all three peers (including itself) show `ok`.
|
||||
- Kill one standby (not the primary): re-query the primary — that standby now shows `dead`, the other two still `ok`.
|
||||
- Query a surviving standby directly: its own entry shows `ok`, every other entry (including the primary) shows `unknown` — proving the asymmetric-visibility design decision holds live, not just in a unit test.
|
||||
- (Optional, if a clean way to hold a node in catch-up long enough to observe exists in this project's E2E tooling) restart the killed standby and query the primary again shortly after: that peer transiently shows `catching-up` before settling to `ok`.
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ The status query is served by **any** controller node — a standby reports who
|
|||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: a client-observable controller status query returning the current primary identity, the answering node's identity and role, the configured controller topology (peers + count), the committed configuration version and content checksum, and HA health (current term, commit/apply progress, serving/fenced state, cluster name); off-primary fidelity of the reported committed positions (a standby reports the committed position it has learned from the primary, not an unset node-local value); a responder snapshot timestamp for log correlation; the current primary's election term and election time, with the same off-primary fidelity (a standby reports what it has learned from the primary, not an unset or stale local value); failover-aware delivery of that query; a test-client command that prints the status; and `wait-primary` using the status query as its probe.
|
||||
- **In scope**: a client-observable controller status query returning the current primary identity, the answering node's identity and role, the configured controller topology (peers + count), the committed configuration version and content checksum, and HA health (current term, commit/apply progress, serving/fenced state, cluster name); off-primary fidelity of the reported committed positions (a standby reports the committed position it has learned from the primary, not an unset node-local value); a responder snapshot timestamp for log correlation; the current primary's election term and election time, with the same off-primary fidelity (a standby reports what it has learned from the primary, not an unset or stale local value); each configured controller's connection status, with the same off-primary fidelity pattern (the primary reports a true picture of every controller; a standby reports "unknown" for whatever it cannot itself observe); failover-aware delivery of that query; a test-client command that prints the status; and `wait-primary` using the status query as its probe.
|
||||
- **Out of scope**: an operator-facing CLI/REST status surface (a later pass); any change to HA election, replication, or commit behavior (this query only **reads** existing controller state); presentation/output formats beyond the query's structured reply and the test client's human-readable print.
|
||||
- **Adjacent expectations**: relies on the existing client master-rotation/failover already used by other read operations; consumes the controller's existing HA state (current primary, role, term, committed configuration version, commit/apply progress, fenced state) without changing how that state is produced or maintained. The status query is a **focused identity/health view** and is intentionally distinct from the existing full cluster-configuration listing (which enumerates resources, queues, and workers).
|
||||
|
||||
|
|
@ -127,3 +127,15 @@ The status query is served by **any** controller node — a standby reports who
|
|||
4. The reported election time shall be expressed on the same clock and in a form that an operator can directly compare against controller log timestamps, consistent with Requirement 11.2.
|
||||
5. Reporting the election term and election time shall be a read-only reporting concern and shall not change how elections are conducted, terms are advanced, or leadership is determined, consistent with Requirement 1.2.
|
||||
6. When the status command is issued in the test client, the test client shall print the current primary's election term and election time, in human-readable form, alongside the other status fields.
|
||||
|
||||
### Requirement 13: Per-controller connection status
|
||||
|
||||
**Objective:** As an operator, I want to see each configured controller's connection status alongside its identity in the status response, so that I can tell at a glance which controllers are healthy, rejoining, or unreachable without checking logs on every node.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When a client issues a status query, the Controller shall report each configured controller's connection status as one of a well-defined set of values — "ok" (connected and caught up), "catching-up" (connected but not yet caught up with the cluster), "dead" (not connected), or "unknown" (the responder cannot determine it) — reported alongside that controller's existing name and address (Requirement 4.1).
|
||||
2. The responding node shall always report its own connection status as "ok".
|
||||
3. While the responding node is currently the primary, when a client issues a status query, the Controller shall report the true, currently-known connection status for every other configured controller, since the primary is the only node with direct connection visibility to every standby.
|
||||
4. While the responding node is not the primary, when a client issues a status query, the Controller shall report "unknown" as the connection status for any configured controller it cannot reliably determine the connection state of, rather than a stale or guessed value.
|
||||
5. Reporting per-controller connection status shall be a read-only reporting concern and shall not change how connections are established, monitored, or torn down.
|
||||
6. When the status command is issued in the test client, the test client shall print each configured controller's connection status alongside its existing name and address.
|
||||
|
|
|
|||
|
|
@ -228,3 +228,70 @@ strictly correct.
|
|||
read suffices here since there is no pre-existing primary-only marker to disambiguate against)
|
||||
— both concrete instances of preferring the smaller design that still satisfies every
|
||||
acceptance criterion.
|
||||
|
||||
---
|
||||
|
||||
## Gap Analysis — Delta for Requirement 13 (2026-09-16)
|
||||
|
||||
Reopened by a user enhancement request: `ctrlstatus`'s "Controllers (N):" peer list should show
|
||||
each configured controller's connection status (ok/catching-up/dead), not just name and address.
|
||||
User-confirmed at requirements time: three-state granularity, and best-effort per-responder
|
||||
reporting ("unknown" for peers a responder can't observe) rather than a primary-only-authoritative
|
||||
requirement. Scope of this delta = Req 13 only. Reqs 1–12 are implemented and unchanged.
|
||||
|
||||
### Requirement-to-Asset Map
|
||||
|
||||
| Need (Req) | Existing asset | Gap |
|
||||
|---|---|---|
|
||||
| 13.1 well-defined status value set, per peer | `CtrlPeer` struct (`controller.capnp`, `xce_ctrlpeer_t` in `typemap.h`) already has `name`/`host`/`port`, next free ordinal `@3` | **Missing**: a `status` field on `CtrlPeer` — this is the first Req 12+ delta to extend the nested list-element struct rather than `CtrlStatus` itself |
|
||||
| 13.2 self always `ok` | `ctrlstatus_fill` already establishes `st->responderName` for its own identity | **No gap** — trivial name-match against the peer loop's own `candidates[i].name` |
|
||||
| 13.3 primary reports true status for every peer | `replicator_t.standbys[]` (`src/include/repl.h`) already tracks each standby's live connection (`ep`), catch-up state (`in_catch_up`), host, and port — built and maintained for replication/quorum purposes, unrelated to this spec | **Missing**: a lookup from a configured peer (`peers_cfg.candidates[i]`, identified by name+host+port) to its corresponding `replicator_t.standbys[]` slot. **Constraint discovered**: no positional/index correlation exists between the two arrays — `standby_conn_t` carries `host`/`port` but no configured-peer name or stable index, so the lookup must match by host+port value, not by name or array position. `repl_active_standby_count` (`repl.c`) already implements the exact `ep != NULL && !in_catch_up` predicate this delta needs to reuse for the "ok" case. |
|
||||
| 13.4 non-primary reports unknown for what it can't determine | `ha->last_activity`/`liveness_poll_fn` (a standby's own liveness signal about its primary) | **Constraint discovered (closes an open requirements-time question)**: this liveness signal is a bare elapsed-time scalar with no peer identity attached — there is no mechanism (analogous to `ha->primary_host`, itself already known to be vestigial per Req 5.1's `ade0081` finding) that would let a standby attribute it to one specific configured peer. A standby can determine ONLY its own entry; every other entry, including the primary's, must report `unknown`. Building peer-identity-carrying liveness data would be materially larger than this delta (a new signal on the heartbeat/connection-accept path) — explicitly out of boundary, not silently absorbed here. |
|
||||
| 13.5 read-only | n/a | **No gap** — the new read is read-only by construction (a lock-scoped array scan), same shape as the existing `ctrlstatus_fill` reads |
|
||||
| 13.6 test client prints it | `cmd_ctrlstatus`'s existing per-peer print line (`src/harness/test.c`) | **Missing**: one more `%s` in the existing printf, fed by a new status-string renderer |
|
||||
|
||||
### Implementation Approach Options
|
||||
|
||||
**Req 13.3 sourcing — Option A (host+port match against `replicator_t.standbys[]`) — RECOMMENDED.**
|
||||
The only asset that exists; matches the exact predicate `repl_active_standby_count` already uses.
|
||||
*Trade-off*: ✅ reuses proven, already-correct logic; no new state. ❌ an O(candidates × MAX_STANDBYS)
|
||||
scan per query — trivially cheap given both are small, fixed-size (peer counts are single-digit
|
||||
in practice, `MAX_STANDBYS` is a small compile-time bound), not a concern at this request rate.
|
||||
|
||||
**Req 13.3 sourcing — Option B (add a stable peer-index field to `standby_conn_t`, matching
|
||||
`standby_ep_ctx_t.peer_idx`'s existing pattern) — REJECTED (for this delta).** Would remove the
|
||||
host/port string-compare, but requires touching `repl_add_standby`'s call sites (production
|
||||
connection-establishment code, not a read-only reporting path) to thread the index through —
|
||||
real scope creep for a purely cosmetic optimization over Option A, which is already O(small×small)
|
||||
and correct. Revisit only if a future need requires this correlation elsewhere too.
|
||||
|
||||
**Req 13.4 non-primary visibility — Option A (self-only, unknown otherwise) — RECOMMENDED (per
|
||||
requirements-time user confirmation, closed further by this gap analysis).** The only option that
|
||||
doesn't require new cross-node signaling.
|
||||
|
||||
**Req 13.4 non-primary visibility — Option B (thread the primary's peer identity through the
|
||||
heartbeat so a standby CAN report its own primary's status) — REJECTED (out of boundary).**
|
||||
Genuinely useful, but a real protocol change (a new heartbeat-carried field identifying the
|
||||
sender, analogous in spirit to how Req 12's `electedAtMs` rides the heartbeat) — bigger than a
|
||||
"show status" delta, deserves its own requirements pass if ever pursued, not a silent scope
|
||||
absorption here.
|
||||
|
||||
### Effort & Risk
|
||||
|
||||
- **Req 13, combined: S (1–2 days).** No HA-critical path touched at all — unlike Req 12, this
|
||||
delta adds zero new promotion-road or heartbeat-protocol code; it is a pure additional read
|
||||
inside the already-read-only `ctrlstatus_fill`, the lowest-risk category of change this spec has
|
||||
made so far. **Risk: Low.** The one genuine unknown flagged as a Research Item (the exact
|
||||
`cluster_t` → `replicator_t` field path) is a one-line confirmation, not a design fork.
|
||||
|
||||
### Design Synthesis (decisions resolved for design.md)
|
||||
|
||||
1. **Generalization**: none needed — this is a single, self-contained addition (one new field,
|
||||
one new lookup, one new renderer); no broader capability is being generalized out of it.
|
||||
2. **Build vs. Adopt**: ADOPT `replicator_t.standbys[]` and `repl_active_standby_count`'s exact
|
||||
`ep != NULL && !in_catch_up` predicate wholesale — zero new tracking state; BUILD only the
|
||||
lookup-by-host/port loop, the wire field, and the string renderer.
|
||||
3. **Simplification**: rejected Option B for both 13.3 (index-based correlation) and 13.4
|
||||
(heartbeat-carried primary identity) — both would introduce real new production-path coupling
|
||||
to serve a read-only reporting feature; the smaller, already-available-data option satisfies
|
||||
every acceptance criterion the user confirmed.
|
||||
|
|
|
|||
|
|
@ -1,9 +1,9 @@
|
|||
{
|
||||
"feature_name": "controller-status",
|
||||
"created_at": "2026-06-19T00:00:00Z",
|
||||
"updated_at": "2026-09-16T09:20:00Z",
|
||||
"updated_at": "2026-09-16T19:35:00Z",
|
||||
"language": "en",
|
||||
"phase": "implementation-complete",
|
||||
"phase": "design-generated",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
|
|
@ -11,7 +11,7 @@
|
|||
},
|
||||
"design": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
"approved": false
|
||||
},
|
||||
"tasks": {
|
||||
"generated": true,
|
||||
|
|
@ -25,16 +25,16 @@
|
|||
"notes": "ctrlstatus opcode + handler + all-roles dispatch gate + libxce any-node & authoritative reads + ctrlstatus command + wait-primary swap, all reviewed and committed; E2E-verified live (primary self-reports, standbys answer with answered-by, config SHA-256 checksum identical across nodes, authoritative wait-primary names the primary). E2E found+fixed the primaryName-from-vestigial-primary_host bug. Out-of-boundary upstream finding: libxce failover returns EINVAL on a dead-first-master (affects joblist equally) -> route to libxce-failover."
|
||||
},
|
||||
"delta": {
|
||||
"reopened_at": "2026-09-15T00:00:00Z",
|
||||
"reason": "User enhancement request (2026-09-15): the xce_test ctrlstatus command should additionally report the term in which the current primary was elected and the wall-clock time at which it was elected, so operators can tell how long the current primary has served and correlate its election with a specific event in the logs. Standby-side fidelity is explicitly in scope (mirrors Req 10's off-primary fidelity pattern): a standby that already knows the current primary's identity (Req 2.3) must report the primary's learned election term/time, not an unset or stale local value.",
|
||||
"new_requirements": "12 (current primary's election term and time)",
|
||||
"needs": "design + tasks regen for Req 12 only; prior tasks 1.1-6.4 (Req 1-11) remain implemented",
|
||||
"impl_status": "7.1-7.6 all implemented + reviewed + committed and complete. Live E2E (7.6) found + fixed a real bug (bug-1818): cmd_ctrlstatus gated electedTerm/electedAt on NULLSTR(primaryName), which is unconditionally empty off-primary in this build, permanently hiding the correct, already-converging elected_at_ms data on every standby. Fixed by gating on elected_at_ms == 0 instead (commit 80839661); design.md's Locked Decision 3 corrected to match. Retry confirmed live convergence across a real primary kill/re-election.",
|
||||
"reopened_at": "2026-09-16T00:00:00Z",
|
||||
"reason": "User enhancement request (2026-09-16): the xce_test ctrlstatus command should additionally show each configured controller's connection status (ok/catching-up/dead/unknown), not just its name and address, so operators can see at a glance which controllers are healthy, rejoining, or unreachable without checking logs on every node. Confirmed with the user: three-state granularity (ok/catching-up/dead, using the already-tracked in_catch_up distinction), and best-effort per-responder reporting with 'unknown' for peers a standby cannot observe (consistent with this spec's established off-primary fidelity pattern) -- not a primary-only-authoritative requirement.",
|
||||
"new_requirements": "13 (per-controller connection status)",
|
||||
"needs": "design + tasks regen for Req 13 only; prior tasks 1.1-7.6 (Req 1-12) remain implemented",
|
||||
"impl_status": "not started",
|
||||
"prior_delta": {
|
||||
"reopened_at": "2026-06-20T00:00:00Z",
|
||||
"reason": "Live E2E finding (2026-06-20): ctrlstatus answered by a standby reports committedLsn=0 / committedConfigVersion=0 while appliedLsn=38 — it reads the primary-only raw markers (persist.committed_lsn, ha->committed_config_tag) instead of the standby's learned committed position (ha->last_known_committed_lsn / last_known_committed_config_tag). Plus an observability gap: no responder timestamp to correlate a captured status against ctrl-N.log.",
|
||||
"new_requirements": "10 (off-primary committed-state fidelity), 11 (responder snapshot timestamp for log correlation)",
|
||||
"impl_status": "6.1-6.4 all implemented + reviewed + committed and complete, validated GO on 2026-07-17 (see validate_impl below)."
|
||||
"reopened_at": "2026-09-15T00:00:00Z",
|
||||
"reason": "User enhancement request (2026-09-15): the xce_test ctrlstatus command should additionally report the term in which the current primary was elected and the wall-clock time at which it was elected, so operators can tell how long the current primary has served and correlate its election with a specific event in the logs. Standby-side fidelity is explicitly in scope (mirrors Req 10's off-primary fidelity pattern): a standby that already knows the current primary's identity (Req 2.3) must report the primary's learned election term/time, not an unset or stale local value.",
|
||||
"new_requirements": "12 (current primary's election term and time)",
|
||||
"impl_status": "7.1-7.6 all implemented + reviewed + committed and complete, validated GO on 2026-09-16 (see validate_impl below)."
|
||||
}
|
||||
},
|
||||
"validate_impl": {
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue