docs(controller-status): add Requirement 12 delta (primary election term/time)
Adds requirements, design, research, and tasks for reporting the term and wall-clock time the current primary was elected, including standby-side learned fidelity (mirroring the Req 10 pattern), following the existing controller-status spec's delta convention (Req 10/11). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
This commit is contained in:
parent
63de947305
commit
7556856e5c
5 changed files with 245 additions and 8 deletions
|
|
@ -311,3 +311,97 @@ No new files (the capnp field is generated; all logic slots into existing TUs).
|
|||
**Integration / round-trip (libxce):** `xcecli_ctrlstatus` decodes the new `responderTimeMs` field end-to-end (no manual decode — codegen), proving wire compatibility.
|
||||
|
||||
**E2E (live `/xce-deploy-ha`):** reproduce the original finding — submit jobs so the primary commits, query a **standby**; assert the standby's `committedLsn` is non-zero and tracks (within the join-refresh caveat) rather than `0`, and that the printed `responder timestamp` matches the standby's own `ctrl-N.log` line for that query. (Note: the "join-refresh caveat" is narrower than described above — ha-heartbeat-committed-propagation added per-heartbeat refresh, so tracking is now near-live rather than join/RDHUP-only; see that spec's design.md.)
|
||||
|
||||
---
|
||||
|
||||
## Design Delta — Requirement 12 (2026-09-15)
|
||||
|
||||
### What changed and why
|
||||
User enhancement request: `ctrlstatus` should additionally report the term in which the current primary was elected and the wall-clock time it was elected, so operators can tell how long the current primary has served and correlate its election with the logs — with the same off-primary (standby) fidelity Req 10 already established for committed position, not a bare "unknown" off-primary. Scope of this delta = Req 12 only. Reqs 1–11 are implemented and unchanged.
|
||||
|
||||
### Boundary Commitments (delta)
|
||||
- **This spec owns (added):** a new `ha_state_t.last_known_elected_at_ms` field (`_Atomic uint64_t`, declared alongside `last_known_committed_lsn`/`last_known_committed_config_tag` in `ha.h`), self-stamped by BOTH promotion roads at the exact point each commits to `CTRL_PRIMARY`; a new `HeartbeatMsg.electedAtMs` wire field the primary echoes on every outgoing heartbeat; the receiving standby's unconditional store of that value into its own `last_known_elected_at_ms`; a new `CtrlStatus.electedAtMs` reply field; `ctrlstatus_fill` populating it (one unconditional read, no role-branch — see Locked Decision 2); `cmd_ctrlstatus`'s two new printed lines (`electedTerm`, `electedAt`), gated on the same primaryName-known/unknown derivation Req 2 already computes.
|
||||
- **Out of Boundary (firm):** `electedTerm` gets **no new field** — it reuses the existing `CtrlStatus.term` (Req 6.1) with CLI-side "unknown" gating (Locked Decision 1); this spec does not change how elections are conducted, how `ha->term` advances, or how/when `ha->role` transitions — every stamp site added here is purely observational at an already-decided promotion instant (Req 12.5).
|
||||
- **Allowed Dependencies (added):** `ha->term` and the existing primaryName derivation (Req 2/3, unchanged); `clock_gettime(CLOCK_REALTIME)` (existing precedent, Req 11); `ctrlstatus_format_time_ms` (`src/include/xce/ctrlstatus_fmt.h`) reused verbatim for rendering — no new formatting code.
|
||||
- **Revalidation Triggers (added):** any change to where `ha->role` finally becomes `CTRL_PRIMARY` on either promotion road (`ha_do_promote` or `raft_cb_become_leader`) → re-check the `last_known_elected_at_ms` stamp site is still past every refusal/eligibility gate; any change to the `HeartbeatMsg`/`heartbeat_msg_t` shape → re-check the `electedAtMs` ordinal and decode; any change to the primaryName-unknown derivation (Req 2) → re-check the electedTerm/electedAtMs "unknown" gating that reuses it.
|
||||
|
||||
### Locked design decisions
|
||||
1. **`electedTerm` reuses `ha->term` (Req 6.1's existing wire field) — no new state.** This codebase's Raft term already propagates correctly to every node via the existing heartbeat/vote term-sync path; there is no "stale/zero" class of bug here the way Req 10 had for `committedLsn` (which read an unrelated *primary-only* marker, `persist.committed_lsn`, that stays 0 off-primary by construction). `cmd_ctrlstatus` derives the displayed `electedTerm` as `primaryName known ? term : unknown` — purely a print-layer decision, no handler change for this half of the requirement. **Accepted trade-off:** in the narrow window where a standby has just voted for a new candidate (bumping its own `ha->term`) before its `voted_for_name`-derived primaryName reflects that same candidate, the displayed `electedTerm` could transiently be a half-step out of step with the displayed primary — the same class of transient standby-hint staleness already accepted for the original `primaryName`/`term` fields (see Architecture's "any-node hint then authoritative-from-primary" flow) and for Req 10's own documented staleness caveat. Not a regression, not new risk surface.
|
||||
2. **`electedAtMs` is genuinely new state, written unconditionally (no monotonic/term compare on write).** Both promotion roads write `last_known_elected_at_ms` exactly once, at the instant each road COMMITS to `CTRL_PRIMARY` — i.e., past every refusal/eligibility gate, never at function entry (see Research Items below for the exact confirm-at-implementation lines). The heartbeat-receive site writes it only when already processing a heartbeat from the node's currently-recognized leader — the same precondition `ha_observe_heartbeat_committed`'s existing call site already relies on for `committed_lsn`/`committed_config_tag`. Because every write site is already gated by an existing freshness precondition, no second "which term does my stored value correspond to" comparison is needed. **Simplification vs. Req 10's pattern:** since there is no pre-existing primary-only marker analogous to `persist.committed_lsn` for election time, `ctrlstatus_fill` needs **no role-branch** for this field — one unconditional read of `last_known_elected_at_ms` correctly serves both the primary (self-stamped) and a standby (heartbeat-learned) case.
|
||||
3. **Unknown gating happens at output time, not by proactively zeroing state.** Both `electedTerm` and `electedAtMs` are gated in `ctrlstatus_fill`/`cmd_ctrlstatus` on the SAME primaryName-known/unknown derivation Req 2.2/2.3 already computes. A stale `last_known_elected_at_ms` left over from a since-lost primary is simply never surfaced once primaryName itself reports "unknown" — mirrors how the original design already treats a stale `voted_for_name`/`primary_host`.
|
||||
4. **Wire type: `electedAtMs @14: UInt64` epoch-milliseconds** — mirrors Req 11's `responderTimeMs` numeric-not-Text precedent exactly (machine-comparable; the client formats it via the same existing `ctrlstatus_format_time_ms`).
|
||||
|
||||
### Research items carried to implementation
|
||||
1. Confirm the exact statement in `raft_cb_become_leader` (`ha.c:1615`) where `ha->role` is finally stored as `CTRL_PRIMARY` — past the `ha_promotion_gate_decision` eligibility gate (later in the function than this delta's own discovery reached) — and stamp `last_known_elected_at_ms` there. Do **not** stamp at the earlier `last_known_committed_lsn` self-observe point (`ha.c:1665`, inside the `if (promo_ectx)` block): that point precedes the eligibility gate and can still be followed by a promotion refusal, which must not produce a bogus election time.
|
||||
2. Confirm which of `ha_observe_heartbeat_committed`'s 9 existing call sites corresponds to processing a genuinely RECEIVED heartbeat (as opposed to a local/test-only invocation) and add the unconditional `last_known_elected_at_ms` store immediately alongside that call, reading the new `electedAtMs` field off the same decoded `heartbeat_msg_t`. Also confirm the outgoing side — where `primary_heartbeat_fn` (`ha.c:9791`) populates the `heartbeat_msg_t` it sends via `xce_ha_heartbeat_req` — and set `electedAtMs = last_known_elected_at_ms` there.
|
||||
3. Confirm `ha_do_promote` (`ha.c:2927`, the function `ha_promote_to_primary`'s own comment describes as owning "the term increment + sets role=PRIMARY" for the non-raft road) is the correct single stamp site for that road; its body was outside this delta's discovery window.
|
||||
|
||||
### Constraints checked
|
||||
- **Async I/O invariant (ha-async-io.md):** every write added here is an atomic store or a `clock_gettime(CLOCK_REALTIME)` call on the existing promotion/heartbeat control-flow paths — no new `connect()`, file I/O, or other blocking call is introduced; identical shape to Req 11's already-accepted `CLOCK_REALTIME` stamp.
|
||||
- **Persistence / save-restore completeness:** not applicable — `last_known_elected_at_ms` is ephemeral in-memory HA state, never written to `ha_term_file_t`/WAL/any durable format, and is fully re-derived on the next promotion or heartbeat exactly like its sibling `last_known_committed_lsn`/`last_known_committed_config_tag` (also not persisted). No save/restore path is added or modified by this delta.
|
||||
|
||||
### Schema delta (Cap'n Proto)
|
||||
```capnp
|
||||
# CtrlStatus — next free ordinal @14 (after responderTimeMs @13; append-only, never reorder)
|
||||
electedAtMs @14: UInt64 $C.mapname("elected_at_ms");
|
||||
# epoch millis of the current primary's election, learned off-primary via
|
||||
# heartbeat; 0/unset when no primary is known (gated together with primaryName)
|
||||
|
||||
# HeartbeatMsg — next free ordinal @4 (after committedConfigTag @3)
|
||||
electedAtMs @4: UInt64 $C.mapname("elected_at_ms");
|
||||
# primary's own last_known_elected_at_ms, echoed on every heartbeat (Req 12.2)
|
||||
```
|
||||
No change to `term @4` on `CtrlStatus` — Req 12's term half changes only the CLI's derived label/gating, not the wire field or its value.
|
||||
|
||||
### File Structure Plan (delta)
|
||||
**Modified files:**
|
||||
- `ha.h` — add `_Atomic uint64_t last_known_elected_at_ms;` to `ha_state_t`, next to `last_known_committed_lsn`/`last_known_committed_config_tag`.
|
||||
- `src/schema/controller.capnp` — add `electedAtMs` to `CtrlStatus` (@14) and to `HeartbeatMsg` (@4); regen `controller.capnp.h/.c`.
|
||||
- `src/controller/ha.c` — `ha_do_promote` and `raft_cb_become_leader`: stamp `last_known_elected_at_ms` from `CLOCK_REALTIME` at each road's final promotion-commit point (Research Items 1, 3); the heartbeat-receive call site: unconditional store alongside the existing `ha_observe_heartbeat_committed` call (Research Item 2); `primary_heartbeat_fn`: populate the outgoing `heartbeat_msg_t.elected_at_ms` from `last_known_elected_at_ms` (Research Item 2).
|
||||
- `src/controller/ctrlstatus.c` — `ctrlstatus_fill`: read `last_known_elected_at_ms` unconditionally into `st->elected_at_ms` (no role-branch, Locked Decision 2).
|
||||
- `src/harness/test.c` — `cmd_ctrlstatus`: two new printed lines, `electedTerm` (derived from `term`+`primaryName`) and `electedAt` (rendered via `ctrlstatus_format_time_ms`), both "unknown" when `primaryName` is unknown. **Path correction (as-built, confirmed 2026-09-15):** `cmd_ctrlstatus`/`cmd_wait_primary` now live in `src/harness/test.c`, not `src/controller/testsuite/test.c` as Reqs 1-11's original design text says — the test-client harness was consolidated into `src/harness/` by a later refactor (see the `xce-harness-component` spec) after those sections were written. Historical Reqs 1-11 text is left as-is (accurate as of when implemented); this delta targets the current, correct location.
|
||||
- `src/controller/testsuite/test_ctrlstatus.c` — extend/add unit tests (below).
|
||||
- `src/controller/testsuite/test_ha_heartbeat_committed.c` (or a new sibling test file) — coverage for the heartbeat-carried `electedAtMs` store.
|
||||
- `src/controller/testsuite/test_ha_role.c` — extend `test_promote_increments_term`-style coverage to also assert `last_known_elected_at_ms` is stamped on promotion.
|
||||
|
||||
No new files (capnp fields are generated; all logic slots into existing TUs) — consistent with this spec's established pattern.
|
||||
|
||||
### Requirements Traceability (delta)
|
||||
| Requirement | Summary | Components | Interfaces |
|
||||
|-------------|---------|------------|------------|
|
||||
| 12.1 | Report the primary's election term + time alongside its identity | `ctrlstatus_fill` (`electedAtMs` read), `cmd_ctrlstatus` (`electedTerm` derived from `term`+`primaryName`) | `CtrlStatus.term` (reused), `CtrlStatus.electedAtMs` (new) |
|
||||
| 12.2 | Standby reports the LEARNED election term/time, not stale/unset | heartbeat-carried `electedAtMs`; existing term-sync for `electedTerm` | `HeartbeatMsg.electedAtMs`, unconditional store into `last_known_elected_at_ms` |
|
||||
| 12.3 | Unknown when no primary is known | `ctrlstatus_fill`/`cmd_ctrlstatus` gating on primaryName | shares Req 2.2/2.3's existing "unknown" derivation |
|
||||
| 12.4 | Election time on the same clock/format as logs | `ctrlstatus_fill` (`CLOCK_REALTIME` stamp), `cmd_ctrlstatus` render | reuses `ctrlstatus_format_time_ms` (Req 11.2 precedent) |
|
||||
| 12.5 | Read-only; no election/leadership behavior change | stamp sites are observational only; no gating logic altered | n/a |
|
||||
| 12.6 | Test client prints both fields | `cmd_ctrlstatus` | two new printed lines |
|
||||
|
||||
### `ctrlstatus_fill` / promotion / heartbeat delta — Service contract
|
||||
- **Req 12.1/12.2 (sourcing):** `st->elected_at_ms = ha->last_known_elected_at_ms` (atomic acquire load), unconditionally — no role-branch (Locked Decision 2).
|
||||
- **Req 12.5 (read-only invariant) preserved:** the handler itself performs only atomic loads, exactly as every other field; all writes to `last_known_elected_at_ms` happen upstream, at promotion or heartbeat-processing time, never inside `ctrlstatus_fill`.
|
||||
- **`cmd_ctrlstatus` (Req 12.1, 12.3, 12.6):** print
|
||||
```
|
||||
electedTerm: <term> (or "unknown" when primaryName is unknown)
|
||||
electedAt: <ctrlstatus_format_time_ms(elected_at_ms)> (or "unknown")
|
||||
```
|
||||
placed after the existing `Primary:` line, grouping the primary's identity with its election term/time.
|
||||
|
||||
### Testing Strategy (delta)
|
||||
**Unit (`test_ctrlstatus.c`):**
|
||||
- PRIMARY: `ctrlstatus_fill` reports `elected_at_ms == ha->last_known_elected_at_ms` (self-stamped); CLI-derived `electedTerm == st.term`.
|
||||
- STANDBY with a learned `last_known_elected_at_ms` (simulating an already-processed heartbeat) and a known primaryName → reports the learned value, not 0/stale (12.2).
|
||||
- STANDBY with primaryName unknown (no vote / self-vote / electing) → CLI renders `electedTerm`/`electedAt` as "unknown" regardless of any stale stored `last_known_elected_at_ms` (12.3).
|
||||
- Extend the existing `test_ctrlstatus_no_mutation` snapshot/compare to also cover `last_known_elected_at_ms` (12.5).
|
||||
|
||||
**New/extended heartbeat unit test (sibling to `test_ha_heartbeat_committed.c`):**
|
||||
- Processing a received heartbeat with a given `electedAtMs` unconditionally overwrites `last_known_elected_at_ms` and leaves `last_known_committed_lsn`/`last_known_committed_config_tag`/role/term/applied untouched — reconcile-free, mirrors `test_observe_no_reconcile_no_other_mutation` (12.2, 12.5).
|
||||
|
||||
**Integration:**
|
||||
- Extend `test_ha_role.c`'s `test_promote_increments_term`-style coverage (non-raft road, `ha_do_promote`) to also assert `last_known_elected_at_ms` is stamped on promotion.
|
||||
- Add/extend a `raft_cb_become_leader` promotion test to assert the same for the raft road, specifically confirming NO stamp occurs on a refused/gated promotion attempt (Research Item 1).
|
||||
- Wire round-trip: `xcecli_ctrlstatus` decodes the new `CtrlStatus.electedAtMs` field end-to-end (codegen, no manual decode).
|
||||
|
||||
**E2E (live `/xce-deploy-ha`):**
|
||||
- Query `ctrlstatus` on the primary right after a fresh election: `electedTerm` equals the printed `term`; `electedAt` is close to "now."
|
||||
- Query a STANDBY: `electedTerm`/`electedAt` match the primary's own values (within one heartbeat interval), not "unknown" and not stale/zero — proving standby learned fidelity end-to-end (12.2).
|
||||
- Kill the primary, wait for a new election, query the surviving standby immediately vs. after one heartbeat interval — demonstrates the field converges promptly via heartbeat, unlike Req 10's original join/RDHUP-only caveat (heartbeat-carried delivery here is in-boundary from day one of this delta, not a follow-up spec).
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ The status query is served by **any** controller node — a standby reports who
|
|||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: a client-observable controller status query returning the current primary identity, the answering node's identity and role, the configured controller topology (peers + count), the committed configuration version and content checksum, and HA health (current term, commit/apply progress, serving/fenced state, cluster name); off-primary fidelity of the reported committed positions (a standby reports the committed position it has learned from the primary, not an unset node-local value); a responder snapshot timestamp for log correlation; failover-aware delivery of that query; a test-client command that prints the status; and `wait-primary` using the status query as its probe.
|
||||
- **In scope**: a client-observable controller status query returning the current primary identity, the answering node's identity and role, the configured controller topology (peers + count), the committed configuration version and content checksum, and HA health (current term, commit/apply progress, serving/fenced state, cluster name); off-primary fidelity of the reported committed positions (a standby reports the committed position it has learned from the primary, not an unset node-local value); a responder snapshot timestamp for log correlation; the current primary's election term and election time, with the same off-primary fidelity (a standby reports what it has learned from the primary, not an unset or stale local value); failover-aware delivery of that query; a test-client command that prints the status; and `wait-primary` using the status query as its probe.
|
||||
- **Out of scope**: an operator-facing CLI/REST status surface (a later pass); any change to HA election, replication, or commit behavior (this query only **reads** existing controller state); presentation/output formats beyond the query's structured reply and the test client's human-readable print.
|
||||
- **Adjacent expectations**: relies on the existing client master-rotation/failover already used by other read operations; consumes the controller's existing HA state (current primary, role, term, committed configuration version, commit/apply progress, fenced state) without changing how that state is produced or maintained. The status query is a **focused identity/health view** and is intentionally distinct from the existing full cluster-configuration listing (which enumerates resources, queues, and workers).
|
||||
|
||||
|
|
@ -115,3 +115,15 @@ The status query is served by **any** controller node — a standby reports who
|
|||
1. When a client issues a status query, the Controller shall include in the response the responding node's wall-clock time at the moment the status fields were sampled.
|
||||
2. The reported timestamp shall be expressed on the same clock and in a form that an operator can directly compare against the responding controller's log timestamps.
|
||||
3. When the status command is issued in the test client, the test client shall print the responder snapshot timestamp in human-readable form alongside the other status fields.
|
||||
|
||||
### Requirement 12: Current primary's election term and time
|
||||
|
||||
**Objective:** As an operator or test author, I want the status response to report the term in which the current primary was elected and the wall-clock time at which it was elected, so that I can tell how long the current primary has been serving and correlate its election with a specific event in the logs.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. While a primary is currently elected and serving, when a client issues a status query, the Controller shall report the term in which that primary was elected and the wall-clock time at which it was elected, alongside the current primary identity reported by Requirement 2.
|
||||
2. When the status query is answered by a standby node that knows the current primary's identity (Requirement 2.3), the Controller shall report that primary's election term and election time as it has learned them from the primary, rather than an unset or stale local value.
|
||||
3. If no primary has currently been elected, or the responding standby does not currently know a primary (Requirement 2.2), then the Controller shall report the election term and election time as the well-defined "unknown"/zero value, rather than a stale or guessed value.
|
||||
4. The reported election time shall be expressed on the same clock and in a form that an operator can directly compare against controller log timestamps, consistent with Requirement 11.2.
|
||||
5. Reporting the election term and election time shall be a read-only reporting concern and shall not change how elections are conducted, terms are advanced, or leadership is determined, consistent with Requirement 1.2.
|
||||
6. When the status command is issued in the test client, the test client shall print the current primary's election term and election time, in human-readable form, alongside the other status fields.
|
||||
|
|
|
|||
|
|
@ -151,3 +151,80 @@ work, mirrors the `configChecksum`-as-Text precedent). Recommend numeric epoch-m
|
|||
2. Decide Req 11 wire type: `UInt64` epoch-millis (recommended) vs `Text`.
|
||||
3. Decide Req 10 sourcing: role-branch (`persist.*` on primary, `last_known_*` off-primary) vs uniform `max()`; keep LSN and config-tag symmetric.
|
||||
4. State the **staleness caveat** in design (learned value refreshes at join, not per heartbeat) and explicitly route live-fresh-committed to **ha-heartbeat-liveness / ha-repl-hardening** as out-of-boundary.
|
||||
|
||||
---
|
||||
|
||||
## Gap Analysis — Delta for Requirement 12 (2026-09-15)
|
||||
|
||||
Reopened by a user enhancement request: `ctrlstatus` should report the term in which the
|
||||
current primary was elected and the wall-clock time it was elected, with standby-side
|
||||
fidelity (learned, not "unknown") explicitly requested — mirroring Req 10's off-primary
|
||||
fidelity pattern rather than the simpler primary-only alternative. Scope of this delta =
|
||||
Req 12 only. Reqs 1–11 are implemented and unchanged.
|
||||
|
||||
### Requirement-to-Asset Map
|
||||
|
||||
| Need (Req) | Existing asset | Gap |
|
||||
|---|---|---|
|
||||
| 12.1 term of the current primary's election | `ha->term` (`ha.h`, already reported as `CtrlStatus.term`, Req 6.1) | **No gap** — this codebase's Raft term already propagates correctly to every node via the existing heartbeat/vote term-sync path (`heartbeat_msg_t.term`, `src/include/typemap.h:613`). Reuse, don't add a field. |
|
||||
| 12.1 wall-clock time of the current primary's election | none — `ha_state_t` has no election timestamp of any kind (unlike `last_known_committed_lsn`/`last_known_committed_config_tag`, which already exist for committed position) | **Missing**: a new `ha_state_t` field, a new promotion-time stamp (both roads), a new heartbeat-carried field, and a new `CtrlStatus` reply field |
|
||||
| 12.2 off-primary (standby) fidelity for election time | `ha_observe_heartbeat_committed` (`ha.c:9085`) is the existing precedent for learning a primary-sourced value via heartbeat, monotonic-on-LSN | **Missing**: `heartbeat_msg_t` (`typemap.h:611-616`) carries `term`/`committed_lsn`/`committed_config_tag` but no election timestamp — must add `electedAtMs` to both the C struct and its capnp schema (`HeartbeatMsg`, next free ordinal after `committedConfigTag@3`) |
|
||||
| 12.2 primary's own promotion-time self-observation | `raft_cb_become_leader` (`ha.c:1615`) already self-observes `last_known_committed_lsn` at its promotion instant (`ha.c:1665`, inside `if (promo_ectx)`); `ha_do_promote` (`ha.c:2927`) is the non-raft road's promotion function | **Gap (needs implementation-time confirmation, not just reuse)**: the committed-LSN self-observe point in `raft_cb_become_leader` sits BEFORE the function's later `ha_promotion_gate_decision` eligibility gate, which can still refuse promotion — the NEW election-time stamp must sit AFTER that gate, at the point the function is irrevocably committed to `CTRL_PRIMARY`, a different line than the existing precedent. This delta's own codegraph exploration did not reach that later line (function body was truncated by explore-tool budget); flagged as a Research Item in design.md rather than guessed. |
|
||||
| 12.3 unknown when no primary known | existing primaryName-unknown derivation (Req 2.2/2.3) | **No gap** — gate the new fields' *output* on this existing derivation; no new "unknown" logic needed |
|
||||
| 12.4 same clock/format as Req 11 | `ctrlstatus_format_time_ms` (`src/include/xce/ctrlstatus_fmt.h:33`) already extracted as a shared, reusable formatter (confirmed via the discovery-phase CodeGraph hint — it is already used by both `ctrlstatus.c` and `joblist_display.c`) | **No gap** — reuse verbatim, zero new formatting code |
|
||||
| 12.6 test client prints both fields | `cmd_ctrlstatus` (`src/controller/testsuite/test.c`) | **Missing**: two print lines |
|
||||
|
||||
### Implementation Approach Options
|
||||
|
||||
**Req 12 term half — Option A (reuse `ha->term`, CLI-side gate) — RECOMMENDED.** No new field;
|
||||
`cmd_ctrlstatus` derives the displayed `electedTerm` as `primaryName known ? term : unknown`.
|
||||
*Trade-off*: ✅ zero new state, zero schema change for this half. ❌ accepts the same narrow
|
||||
standby-hint transient-staleness window already accepted elsewhere in this spec (a standby that
|
||||
just voted for a new candidate, before its primaryName catches up) — judged acceptable, not a
|
||||
regression, and explicitly called out as a trade-off in design.md rather than glossed over.
|
||||
|
||||
**Req 12 term half — Option B (a dedicated learned `last_known_elected_term`, symmetric with
|
||||
`last_known_committed_lsn`) — REJECTED.** Would be fully symmetric with Req 10's own pattern, but
|
||||
Req 10 needed a *learned* channel specifically because the field it replaced
|
||||
(`persist.committed_lsn`) is genuinely primary-only and wrong (0) off-primary — no such
|
||||
pre-existing wrong-value bug exists for `term`, which is already correctly synced cluster-wide.
|
||||
Adding a parallel field here would be unjustified duplication (design-synthesis Simplification
|
||||
lens): same value, two storage locations, extra room for the two to drift.
|
||||
|
||||
**Req 12 time half — Option A (new `last_known_elected_at_ms`, unconditional heartbeat-carried
|
||||
overwrite) — RECOMMENDED.** Mirrors `ha_observe_heartbeat_committed`'s existing shape (new
|
||||
`heartbeat_msg_t` field, primary-side self-observe at promotion, standby-side store on receive)
|
||||
but does NOT need a monotonic/value compare on write, because every write site is already gated
|
||||
by an existing freshness precondition (see design.md Locked Decision 2). *Trade-off*: ✅ small,
|
||||
in-boundary, symmetric with the existing learned-state pattern. ❌ two new promotion-road stamp
|
||||
sites to get exactly right (see the gate-ordering gap above) — the one non-trivial risk in this
|
||||
delta, called out explicitly as Research Items in design.md rather than assumed correct.
|
||||
|
||||
**Req 12 time half — Option B (derive an approximate elected time from the WAL_NOOP own-term
|
||||
gate entry's timestamp, `ha_arm_own_term_gate`, `ha.c:5079`) — REJECTED.** The own-term no-op is
|
||||
appended AFTER promotion completes and is not guaranteed to exist on every promotion road in
|
||||
every configuration (comment at `ha.c:5088-5093` shows it can fail and leave the gate "UNSET,
|
||||
degraded-but-safe" without aborting promotion) — an unreliable proxy for "the moment of
|
||||
election." A direct, dedicated stamp at the actual role-transition instant is simpler and
|
||||
strictly correct.
|
||||
|
||||
### Effort & Risk
|
||||
|
||||
- **Req 12 term half**: **XS** (a few hours) — one `cmd_ctrlstatus` print-layer change, no schema/handler change. **Risk: Low.**
|
||||
- **Req 12 time half**: **S** (1–2 days) — one new `ha_state_t` field, two promotion-road stamp sites (needs the exact post-eligibility-gate line confirmed per-road, see Research Items), one heartbeat schema field + send/receive wiring, one `ctrlstatus_fill` read, one print block, TDD coverage across a unit test file, a heartbeat sibling test, and a promotion-road integration test. **Risk: Low–Medium** — the only real risk is stamping the election time at the WRONG point in `raft_cb_become_leader` (before its final eligibility gate, like the existing `last_known_committed_lsn` self-observe does), which would show a bogus election time for a promotion attempt that was actually refused; mitigated by an explicit Research Item + a dedicated "no stamp on a refused promotion" test.
|
||||
- **Combined**: **S**. No heartbeat/replication *protocol* semantics change (the heartbeat message only gains one more echoed field, exactly like Req 10/11's own precedent) — stays in-boundary, no need to route anything to ha-heartbeat-liveness/ha-repl-hardening this time (unlike Req 10, which did need that routing for its "live-fresh" out-of-boundary item).
|
||||
|
||||
### Design Synthesis (decisions resolved for design.md)
|
||||
|
||||
1. **Generalization**: `electedAtMs`'s heartbeat-carry + promotion-self-observe shape generalizes
|
||||
the SAME pattern `committedLsn`/`committedConfigTag` already established — no new abstraction
|
||||
invented, the existing `heartbeat_msg_t`/`HeartbeatMsg` extension point is reused directly.
|
||||
2. **Build vs. Adopt**: ADOPT the existing heartbeat-carried learned-state pattern wholesale
|
||||
(no alternative transport considered — piggybacking on the already-flowing heartbeat is
|
||||
strictly simpler than any new channel); BUILD only the one new field + two stamp sites + one
|
||||
receive-side store. `electedTerm` needs no build at all (pure reuse).
|
||||
3. **Simplification**: rejected a parallel `last_known_elected_term` field (Option B above) and
|
||||
rejected a role-branch in `ctrlstatus_fill` for `electedAtMs` (unlike Req 10, one unconditional
|
||||
read suffices here since there is no pre-existing primary-only marker to disambiguate against)
|
||||
— both concrete instances of preferring the smaller design that still satisfies every
|
||||
acceptance criterion.
|
||||
|
|
|
|||
|
|
@ -1,9 +1,9 @@
|
|||
{
|
||||
"feature_name": "controller-status",
|
||||
"created_at": "2026-06-19T00:00:00Z",
|
||||
"updated_at": "2026-07-17T11:15:00Z",
|
||||
"updated_at": "2026-09-15T20:33:00Z",
|
||||
"language": "en",
|
||||
"phase": "implementation-complete",
|
||||
"phase": "tasks-generated",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
|
|
@ -25,11 +25,17 @@
|
|||
"notes": "ctrlstatus opcode + handler + all-roles dispatch gate + libxce any-node & authoritative reads + ctrlstatus command + wait-primary swap, all reviewed and committed; E2E-verified live (primary self-reports, standbys answer with answered-by, config SHA-256 checksum identical across nodes, authoritative wait-primary names the primary). E2E found+fixed the primaryName-from-vestigial-primary_host bug. Out-of-boundary upstream finding: libxce failover returns EINVAL on a dead-first-master (affects joblist equally) -> route to libxce-failover."
|
||||
},
|
||||
"delta": {
|
||||
"reopened_at": "2026-06-20T00:00:00Z",
|
||||
"reason": "Live E2E finding (2026-06-20): ctrlstatus answered by a standby reports committedLsn=0 / committedConfigVersion=0 while appliedLsn=38 — it reads the primary-only raw markers (persist.committed_lsn, ha->committed_config_tag) instead of the standby's learned committed position (ha->last_known_committed_lsn / last_known_committed_config_tag). Plus an observability gap: no responder timestamp to correlate a captured status against ctrl-N.log.",
|
||||
"new_requirements": "10 (off-primary committed-state fidelity), 11 (responder snapshot timestamp for log correlation)",
|
||||
"needs": "design + tasks regen for Req 10-11 only; prior tasks 1.1-5.1 remain implemented",
|
||||
"impl_status": "6.1-6.4 all implemented + reviewed + committed and complete. 6.4 (live-cluster E2E), previously blocked on the sandbox overlayfs fdatasync issue (bug-795), was VERIFIED 2026-07-02 on a real-fs (~/tmp, ext4) 3-node deploy per tasks.md's own Implementation Notes -- this field was stale until this validation pass corrected it."
|
||||
"reopened_at": "2026-09-15T00:00:00Z",
|
||||
"reason": "User enhancement request (2026-09-15): the xce_test ctrlstatus command should additionally report the term in which the current primary was elected and the wall-clock time at which it was elected, so operators can tell how long the current primary has served and correlate its election with a specific event in the logs. Standby-side fidelity is explicitly in scope (mirrors Req 10's off-primary fidelity pattern): a standby that already knows the current primary's identity (Req 2.3) must report the primary's learned election term/time, not an unset or stale local value.",
|
||||
"new_requirements": "12 (current primary's election term and time)",
|
||||
"needs": "design + tasks regen for Req 12 only; prior tasks 1.1-6.4 (Req 1-11) remain implemented",
|
||||
"impl_status": "not started",
|
||||
"prior_delta": {
|
||||
"reopened_at": "2026-06-20T00:00:00Z",
|
||||
"reason": "Live E2E finding (2026-06-20): ctrlstatus answered by a standby reports committedLsn=0 / committedConfigVersion=0 while appliedLsn=38 — it reads the primary-only raw markers (persist.committed_lsn, ha->committed_config_tag) instead of the standby's learned committed position (ha->last_known_committed_lsn / last_known_committed_config_tag). Plus an observability gap: no responder timestamp to correlate a captured status against ctrl-N.log.",
|
||||
"new_requirements": "10 (off-primary committed-state fidelity), 11 (responder snapshot timestamp for log correlation)",
|
||||
"impl_status": "6.1-6.4 all implemented + reviewed + committed and complete, validated GO on 2026-07-17 (see validate_impl below)."
|
||||
}
|
||||
},
|
||||
"validate_impl": {
|
||||
"decision": "GO",
|
||||
|
|
|
|||
|
|
@ -89,6 +89,54 @@
|
|||
- _Requirements: 10.1, 10.3, 11.2, 11.3_
|
||||
- _Depends: 6.2, 6.3_
|
||||
|
||||
## 7. Delta: primary election term + time (Req 12)
|
||||
> Sequential — the election-time stamp touches the same HA promotion/heartbeat control flow as tasks 2.2/6.2 (ha.c); no `(P)`. Read-only reporting only: this task must not change how elections are conducted, how term advances, or how/when role transitions to primary — every write here is an additional observational stamp at an already-decided instant. `electedTerm` needs no new field (design.md Locked Decision 1); it is a CLI-side derivation added in 7.5, not new state.
|
||||
|
||||
- [ ] 7.1 Add the election-time schema fields and controller state
|
||||
- Add a wall-clock election-time field to the status reply payload (next free ordinal, append-only) and to the heartbeat message the primary already sends to standbys (next free ordinal).
|
||||
- Add the corresponding piece of HA state that holds the currently-known primary's election time, alongside the existing learned-committed-position state.
|
||||
- Add the matching fields to the hand-authored C structs backing these two payloads (this schema's `$C.mapname` types are NOT auto-generated — the generated codec references hand-written typedefs in `typemap.h`, per tasks 1.1/2.1/6.1's established convention) and verify each new field field-for-field against the regenerated codec.
|
||||
- Observable: the build regenerates the reply and heartbeat codecs clean with the new fields present, the hand-authored C structs match the codec field-for-field, a codec round-trip carries each new field's value unchanged, and the new HA state field exists and defaults to zero/unknown before any election.
|
||||
- _Requirements: 12.1_
|
||||
- _Boundary: schema + typemap + ha_state_t_
|
||||
|
||||
- [ ] 7.2 Stamp the election time when a node becomes primary
|
||||
- TDD (red first): a promotion via the non-election-protocol path and a promotion via the leader-election path both record a non-zero election-time at the moment the node commits to the primary role; a promotion attempt that is refused (fenced, ineligible, or otherwise rejected) records NO election-time and leaves any prior value untouched.
|
||||
- Stamp the wall-clock election time exactly once, at the point each promotion path irrevocably commits to the primary role — after any eligibility/refusal checks that path performs, never before.
|
||||
- Observable: the new tests go red → green; a refused promotion attempt (either path) leaves the election-time state unchanged, proving the stamp is not placed before the refusal checks.
|
||||
- _Requirements: 12.1, 12.5_
|
||||
- _Boundary: promotion (ha_do_promote, raft_cb_become_leader)_
|
||||
- _Depends: 7.1_
|
||||
|
||||
- [ ] 7.3 Carry the primary's election time to standbys over the heartbeat
|
||||
- TDD (red first): a standby that processes a heartbeat carrying the primary's election time stores it as its own learned election-time and leaves every other learned/produced HA field (committed position, role, term, applied position) untouched (reconcile-free).
|
||||
- The primary includes its own election time in every heartbeat it sends; a standby that receives a heartbeat stores the carried election time as its learned value.
|
||||
- Observable: the reconcile-free test passes — only the learned election-time changes, nothing else moves; a live/loopback heartbeat round-trip shows the standby's learned election-time matching the primary's.
|
||||
- _Requirements: 12.2_
|
||||
- _Boundary: heartbeat (send + receive)_
|
||||
- _Depends: 7.1_
|
||||
|
||||
- [ ] 7.4 Source the election time in the status handler
|
||||
- TDD (red first): the status handler reports a non-zero election-time equal to the responder's own learned value (self-stamped on a primary, heartbeat-learned on a standby); a standby seeded with a learned election-time (simulating an already-processed heartbeat) reports that learned value, not zero; the handler still mutates no state.
|
||||
- Read the learned election-time into the status reply unconditionally (no role-based branching, no "unknown" gating in the handler itself — that display decision belongs to the client command in 7.5, mirroring how the handler already reports raw term unconditionally regardless of whether a primary is currently known).
|
||||
- Observable: the new unit tests go red → green; a standby seeded with a learned election-time reports that value instead of zero; the no-mutation assertion holds.
|
||||
- _Requirements: 12.1, 12.2, 12.5_
|
||||
- _Boundary: ctrlstatus_fill_
|
||||
- _Depends: 7.1_
|
||||
|
||||
- [ ] 7.5 Render the election term and time in the `ctrlstatus` client command
|
||||
- Print the primary's election term and election time next to the primary's identity in the client command's output, rendering the election time on the same clock/format already used for the responder snapshot timestamp; render both as "unknown" whenever the primary identity itself is reported as unknown.
|
||||
- Observable: piping `ctrlstatus` through the client prints the election term and a log-comparable election-time string alongside the existing fields when a primary is known, and prints "unknown" for both when it is not.
|
||||
- _Requirements: 12.1, 12.3, 12.4, 12.6_
|
||||
- _Boundary: cmd_ctrlstatus_
|
||||
- _Depends: 7.4_
|
||||
|
||||
- [ ] 7.6 End-to-end validation on a live cluster
|
||||
- Run against a live 3-node cluster: query the primary right after a fresh election and confirm its printed election term matches its current term and its election time is close to "now"; query a standby and confirm it reports the SAME election term and election time as the primary (not "unknown", not stale) once at least one heartbeat has been exchanged; kill the primary, wait for a new election, and confirm the surviving standby's reported election term/time updates to the new primary's values within about one heartbeat interval.
|
||||
- Observable: the live run shows matching election term/time between the primary and a standby, and shows the standby's values converge to the new primary's after a kill within one heartbeat interval; manual environment prerequisite (live cluster), same status as the other live-E2E tasks (5.1, 6.4).
|
||||
- _Requirements: 12.1, 12.2, 12.4, 12.6_
|
||||
- _Depends: 7.2, 7.3, 7.4, 7.5_
|
||||
|
||||
## Implementation Notes (5.1 E2E outcome)
|
||||
- 5.1 ran live on a 3-node `/xce-deploy-ha`-style cluster (ext2/3 fs). PASSED in-scope: `ctrlstatus` shows the primary self-reporting (`Primary: ctrl-1` on the primary), standbys honestly report `Primary: unknown` with `answered by ctrl-2 (standby)` (the dispatch-gate lets standbys answer — Req 7); peers + count correct; `configChecksum` = the real 64-hex SHA-256 (matched the deploy tag prefix) and is IDENTICAL across all 3 nodes with identical config (Req 5.3); `configVersion`==`committedConfigVersion`; authoritative `wait-primary` names the primary in <1s on a serveable cluster (Req 9.1/9.2).
|
||||
- 5.1 FOUND + FIXED a real 2.1 bug: `ha->primary_host` is VESTIGIAL (never written anywhere in the controller), and `voted_for_name` is the last-vote (often self), so neither names the current primary off-primary. Corrected: PRIMARY self-reports its own name; STANDBY/ELECTING/INIT report empty → "unknown" (the authoritative client path reaches the real primary by rotation). Commit ade0081.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue