lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(controller-status): add Requirement 12 delta (primary election term/time)

Browse source 
Adds requirements, design, research, and tasks for reporting the term
and wall-clock time the current primary was elected, including
standby-side learned fidelity (mirroring the Req 10 pattern), following
the existing controller-status spec's delta convention (Req 10/11).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
This commit is contained in:
Rongsong Shen 2026-09-16 03:47:45 +00:00
parent 63de947305
commit 7556856e5c
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
5 changed files with 245 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

94
.kiro/specs/controller-status/design.md
View file

@ -311,3 +311,97 @@ No new files (the capnp field is generated; all logic slots into existing TUs).
**Integration / round-trip (libxce):** `xcecli_ctrlstatus` decodes the new `responderTimeMs` field end-to-end (no manual decode — codegen), proving wire compatibility.
**E2E (live `/xce-deploy-ha`):** reproduce the original finding — submit jobs so the primary commits, query a **standby**; assert the standby's `committedLsn` is non-zero and tracks (within the join-refresh caveat) rather than `0`, and that the printed `responder timestamp` matches the standby's own `ctrl-N.log` line for that query. (Note: the "join-refresh caveat" is narrower than described above — ha-heartbeat-committed-propagation added per-heartbeat refresh, so tracking is now near-live rather than join/RDHUP-only; see that spec's design.md.)
---
## Design Delta — Requirement 12 (2026-09-15)
### What changed and why
User enhancement request: `ctrlstatus` should additionally report the term in which the current primary was elected and the wall-clock time it was elected, so operators can tell how long the current primary has served and correlate its election with the logs — with the same off-primary (standby) fidelity Req 10 already established for committed position, not a bare "unknown" off-primary. Scope of this delta = Req 12 only. Reqs 1–11 are implemented and unchanged.
### Boundary Commitments (delta)
- **This spec owns (added):** a new `ha_state_t.last_known_elected_at_ms` field (`_Atomic uint64_t`, declared alongside `last_known_committed_lsn`/`last_known_committed_config_tag` in `ha.h`), self-stamped by BOTH promotion roads at the exact point each commits to `CTRL_PRIMARY`; a new `HeartbeatMsg.electedAtMs` wire field the primary echoes on every outgoing heartbeat; the receiving standby's unconditional store of that value into its own `last_known_elected_at_ms`; a new `CtrlStatus.electedAtMs` reply field; `ctrlstatus_fill` populating it (one unconditional read, no role-branch — see Locked Decision 2); `cmd_ctrlstatus`'s two new printed lines (`electedTerm`, `electedAt`), gated on the same primaryName-known/unknown derivation Req 2 already computes.
- **Out of Boundary (firm):** `electedTerm` gets **no new field** — it reuses the existing `CtrlStatus.term` (Req 6.1) with CLI-side "unknown" gating (Locked Decision 1); this spec does not change how elections are conducted, how `ha->term` advances, or how/when `ha->role` transitions — every stamp site added here is purely observational at an already-decided promotion instant (Req 12.5).
- **Allowed Dependencies (added):** `ha->term` and the existing primaryName derivation (Req 2/3, unchanged); `clock_gettime(CLOCK_REALTIME)` (existing precedent, Req 11); `ctrlstatus_format_time_ms` (`src/include/xce/ctrlstatus_fmt.h`) reused verbatim for rendering — no new formatting code.
- **Revalidation Triggers (added):** any change to where `ha->role` finally becomes `CTRL_PRIMARY` on either promotion road (`ha_do_promote` or `raft_cb_become_leader`) → re-check the `last_known_elected_at_ms` stamp site is still past every refusal/eligibility gate; any change to the `HeartbeatMsg`/`heartbeat_msg_t` shape → re-check the `electedAtMs` ordinal and decode; any change to the primaryName-unknown derivation (Req 2) → re-check the electedTerm/electedAtMs "unknown" gating that reuses it.
### Locked design decisions
1. **`electedTerm` reuses `ha->term` (Req 6.1's existing wire field) — no new state.** This codebase's Raft term already propagates correctly to every node via the existing heartbeat/vote term-sync path; there is no "stale/zero" class of bug here the way Req 10 had for `committedLsn` (which read an unrelated *primary-only* marker, `persist.committed_lsn`, that stays 0 off-primary by construction). `cmd_ctrlstatus` derives the displayed `electedTerm` as `primaryName known ? term : unknown` — purely a print-layer decision, no handler change for this half of the requirement. **Accepted trade-off:** in the narrow window where a standby has just voted for a new candidate (bumping its own `ha->term`) before its `voted_for_name`-derived primaryName reflects that same candidate, the displayed `electedTerm` could transiently be a half-step out of step with the displayed primary — the same class of transient standby-hint staleness already accepted for the original `primaryName`/`term` fields (see Architecture's "any-node hint then authoritative-from-primary" flow) and for Req 10's own documented staleness caveat. Not a regression, not new risk surface.
2. **`electedAtMs` is genuinely new state, written unconditionally (no monotonic/term compare on write).** Both promotion roads write `last_known_elected_at_ms` exactly once, at the instant each road COMMITS to `CTRL_PRIMARY` — i.e., past every refusal/eligibility gate, never at function entry (see Research Items below for the exact confirm-at-implementation lines). The heartbeat-receive site writes it only when already processing a heartbeat from the node's currently-recognized leader — the same precondition `ha_observe_heartbeat_committed`'s existing call site already relies on for `committed_lsn`/`committed_config_tag`. Because every write site is already gated by an existing freshness precondition, no second "which term does my stored value correspond to" comparison is needed. **Simplification vs. Req 10's pattern:** since there is no pre-existing primary-only marker analogous to `persist.committed_lsn` for election time, `ctrlstatus_fill` needs **no role-branch** for this field — one unconditional read of `last_known_elected_at_ms` correctly serves both the primary (self-stamped) and a standby (heartbeat-learned) case.
3. **Unknown gating happens at output time, not by proactively zeroing state.** Both `electedTerm` and `electedAtMs` are gated in `ctrlstatus_fill`/`cmd_ctrlstatus` on the SAME primaryName-known/unknown derivation Req 2.2/2.3 already computes. A stale `last_known_elected_at_ms` left over from a since-lost primary is simply never surfaced once primaryName itself reports "unknown" — mirrors how the original design already treats a stale `voted_for_name`/`primary_host`.
4. **Wire type: `electedAtMs @14: UInt64` epoch-milliseconds** — mirrors Req 11's `responderTimeMs` numeric-not-Text precedent exactly (machine-comparable; the client formats it via the same existing `ctrlstatus_format_time_ms`).
### Research items carried to implementation
1. Confirm the exact statement in `raft_cb_become_leader` (`ha.c:1615`) where `ha->role` is finally stored as `CTRL_PRIMARY` — past the `ha_promotion_gate_decision` eligibility gate (later in the function than this delta's own discovery reached) — and stamp `last_known_elected_at_ms` there. Do **not** stamp at the earlier `last_known_committed_lsn` self-observe point (`ha.c:1665`, inside the `if (promo_ectx)` block): that point precedes the eligibility gate and can still be followed by a promotion refusal, which must not produce a bogus election time.
2. Confirm which of `ha_observe_heartbeat_committed`'s 9 existing call sites corresponds to processing a genuinely RECEIVED heartbeat (as opposed to a local/test-only invocation) and add the unconditional `last_known_elected_at_ms` store immediately alongside that call, reading the new `electedAtMs` field off the same decoded `heartbeat_msg_t`. Also confirm the outgoing side — where `primary_heartbeat_fn` (`ha.c:9791`) populates the `heartbeat_msg_t` it sends via `xce_ha_heartbeat_req` — and set `electedAtMs = last_known_elected_at_ms` there.
3. Confirm `ha_do_promote` (`ha.c:2927`, the function `ha_promote_to_primary`'s own comment describes as owning "the term increment + sets role=PRIMARY" for the non-raft road) is the correct single stamp site for that road; its body was outside this delta's discovery window.
### Constraints checked
- **Async I/O invariant (ha-async-io.md):** every write added here is an atomic store or a `clock_gettime(CLOCK_REALTIME)` call on the existing promotion/heartbeat control-flow paths — no new `connect()`, file I/O, or other blocking call is introduced; identical shape to Req 11's already-accepted `CLOCK_REALTIME` stamp.
- **Persistence / save-restore completeness:** not applicable — `last_known_elected_at_ms` is ephemeral in-memory HA state, never written to `ha_term_file_t`/WAL/any durable format, and is fully re-derived on the next promotion or heartbeat exactly like its sibling `last_known_committed_lsn`/`last_known_committed_config_tag` (also not persisted). No save/restore path is added or modified by this delta.
### Schema delta (Cap'n Proto)
```capnp
# CtrlStatus — next free ordinal @14 (after responderTimeMs @13; append-only, never reorder)
electedAtMs @14: UInt64 $C.mapname("elected_at_ms");
# epoch millis of the current primary's election, learned off-primary via
# heartbeat; 0/unset when no primary is known (gated together with primaryName)
# HeartbeatMsg — next free ordinal @4 (after committedConfigTag @3)
electedAtMs @4: UInt64 $C.mapname("elected_at_ms");
# primary's own last_known_elected_at_ms, echoed on every heartbeat (Req 12.2)
```
No change to `term @4` on `CtrlStatus` — Req 12's term half changes only the CLI's derived label/gating, not the wire field or its value.
### File Structure Plan (delta)
**Modified files:**
- `ha.h` — add `_Atomic uint64_t last_known_elected_at_ms;` to `ha_state_t`, next to `last_known_committed_lsn`/`last_known_committed_config_tag`.
- `src/schema/controller.capnp` — add `electedAtMs` to `CtrlStatus` (@14) and to `HeartbeatMsg` (@4); regen `controller.capnp.h/.c`.
- `src/controller/ha.c` — `ha_do_promote` and `raft_cb_become_leader`: stamp `last_known_elected_at_ms` from `CLOCK_REALTIME` at each road's final promotion-commit point (Research Items 1, 3); the heartbeat-receive call site: unconditional store alongside the existing `ha_observe_heartbeat_committed` call (Research Item 2); `primary_heartbeat_fn`: populate the outgoing `heartbeat_msg_t.elected_at_ms` from `last_known_elected_at_ms` (Research Item 2).
- `src/controller/ctrlstatus.c` — `ctrlstatus_fill`: read `last_known_elected_at_ms` unconditionally into `st->elected_at_ms` (no role-branch, Locked Decision 2).
- `src/harness/test.c` — `cmd_ctrlstatus`: two new printed lines, `electedTerm` (derived from `term`+`primaryName`) and `electedAt` (rendered via `ctrlstatus_format_time_ms`), both "unknown" when `primaryName` is unknown. **Path correction (as-built, confirmed 2026-09-15):** `cmd_ctrlstatus`/`cmd_wait_primary` now live in `src/harness/test.c`, not `src/controller/testsuite/test.c` as Reqs 1-11's original design text says — the test-client harness was consolidated into `src/harness/` by a later refactor (see the `xce-harness-component` spec) after those sections were written. Historical Reqs 1-11 text is left as-is (accurate as of when implemented); this delta targets the current, correct location.
- `src/controller/testsuite/test_ctrlstatus.c` — extend/add unit tests (below).
- `src/controller/testsuite/test_ha_heartbeat_committed.c` (or a new sibling test file) — coverage for the heartbeat-carried `electedAtMs` store.
- `src/controller/testsuite/test_ha_role.c` — extend `test_promote_increments_term`-style coverage to also assert `last_known_elected_at_ms` is stamped on promotion.
No new files (capnp fields are generated; all logic slots into existing TUs) — consistent with this spec's established pattern.
### Requirements Traceability (delta)
| Requirement | Summary | Components | Interfaces |
|-------------|---------|------------|------------|
| 12.1 | Report the primary's election term + time alongside its identity | `ctrlstatus_fill` (`electedAtMs` read), `cmd_ctrlstatus` (`electedTerm` derived from `term`+`primaryName`) | `CtrlStatus.term` (reused), `CtrlStatus.electedAtMs` (new) |
| 12.2 | Standby reports the LEARNED election term/time, not stale/unset | heartbeat-carried `electedAtMs`; existing term-sync for `electedTerm` | `HeartbeatMsg.electedAtMs`, unconditional store into `last_known_elected_at_ms` |
| 12.3 | Unknown when no primary is known | `ctrlstatus_fill`/`cmd_ctrlstatus` gating on primaryName | shares Req 2.2/2.3's existing "unknown" derivation |
| 12.4 | Election time on the same clock/format as logs | `ctrlstatus_fill` (`CLOCK_REALTIME` stamp), `cmd_ctrlstatus` render | reuses `ctrlstatus_format_time_ms` (Req 11.2 precedent) |
| 12.5 | Read-only; no election/leadership behavior change | stamp sites are observational only; no gating logic altered | n/a |
| 12.6 | Test client prints both fields | `cmd_ctrlstatus` | two new printed lines |
### `ctrlstatus_fill` / promotion / heartbeat delta — Service contract
- **Req 12.1/12.2 (sourcing):** `st->elected_at_ms = ha->last_known_elected_at_ms` (atomic acquire load), unconditionally — no role-branch (Locked Decision 2).
- **Req 12.5 (read-only invariant) preserved:** the handler itself performs only atomic loads, exactly as every other field; all writes to `last_known_elected_at_ms` happen upstream, at promotion or heartbeat-processing time, never inside `ctrlstatus_fill`.
- **`cmd_ctrlstatus` (Req 12.1, 12.3, 12.6):** print
```
electedTerm: <term> (or "unknown" when primaryName is unknown)
electedAt: <ctrlstatus_format_time_ms(elected_at_ms)> (or "unknown")
```
placed after the existing `Primary:` line, grouping the primary's identity with its election term/time.
### Testing Strategy (delta)
**Unit (`test_ctrlstatus.c`):**
- PRIMARY: `ctrlstatus_fill` reports `elected_at_ms == ha->last_known_elected_at_ms` (self-stamped); CLI-derived `electedTerm == st.term`.
- STANDBY with a learned `last_known_elected_at_ms` (simulating an already-processed heartbeat) and a known primaryName → reports the learned value, not 0/stale (12.2).
- STANDBY with primaryName unknown (no vote / self-vote / electing) → CLI renders `electedTerm`/`electedAt` as "unknown" regardless of any stale stored `last_known_elected_at_ms` (12.3).
- Extend the existing `test_ctrlstatus_no_mutation` snapshot/compare to also cover `last_known_elected_at_ms` (12.5).
**New/extended heartbeat unit test (sibling to `test_ha_heartbeat_committed.c`):**
- Processing a received heartbeat with a given `electedAtMs` unconditionally overwrites `last_known_elected_at_ms` and leaves `last_known_committed_lsn`/`last_known_committed_config_tag`/role/term/applied untouched — reconcile-free, mirrors `test_observe_no_reconcile_no_other_mutation` (12.2, 12.5).
**Integration:**
- Extend `test_ha_role.c`'s `test_promote_increments_term`-style coverage (non-raft road, `ha_do_promote`) to also assert `last_known_elected_at_ms` is stamped on promotion.
- Add/extend a `raft_cb_become_leader` promotion test to assert the same for the raft road, specifically confirming NO stamp occurs on a refused/gated promotion attempt (Research Item 1).
- Wire round-trip: `xcecli_ctrlstatus` decodes the new `CtrlStatus.electedAtMs` field end-to-end (codegen, no manual decode).
**E2E (live `/xce-deploy-ha`):**
- Query `ctrlstatus` on the primary right after a fresh election: `electedTerm` equals the printed `term`; `electedAt` is close to "now."
- Query a STANDBY: `electedTerm`/`electedAt` match the primary's own values (within one heartbeat interval), not "unknown" and not stale/zero — proving standby learned fidelity end-to-end (12.2).
- Kill the primary, wait for a new election, query the surviving standby immediately vs. after one heartbeat interval — demonstrates the field converges promptly via heartbeat, unlike Req 10's original join/RDHUP-only caveat (heartbeat-carried delivery here is in-boundary from day one of this delta, not a follow-up spec).