checkpoint: bug-851 root cause + raft TLA+ conformance spec family
Investigation & fixes (failover committed-job loss): - root-cause bug-851 on an owned 3-node cluster: chaotic failover causes silent committed-prefix divergence (WAL hdr.term always 0; two leaders per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION. - live-validated fixes 848/849/850 (recovery-on-promotion, advertise real applied_lsn, persist-layer contiguity guard); add waldump WAL inspector. Specs (strict raft.tla conformance — library owns the log): - TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6. - raft-log-replication, raft-commit-safety, raft-sim-conformance: extend src/lib/raft to own log/commit/verification via callbacks (specs generated, cross-spec review passed). - ha-term-safety: controller-side adapter + config marker (R1-R7); design done. - config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857). - roadmap + buglog (851/854/857) + .wolf knowledge updated.
This commit is contained in:
parent
7d70820aac
commit
20e59d0b42
85 changed files with 10525 additions and 1993 deletions
|
|
@ -237,3 +237,77 @@ The only new data is the reply payload above. `CtrlStatus` is a flat value objec
|
|||
## Security Considerations
|
||||
|
||||
`ctrlstatus` exposes cluster topology, role, term, and config version/checksum — the same trust level as the existing `clusterconf`/state-report reads (no secrets, no config content, no job data). It is read-only and goes through the same authenticated client path as other client ops; no new authorization surface is introduced. Broadening the dispatch gate is scoped strictly to this one read opcode.
|
||||
|
||||
---
|
||||
|
||||
## Design Delta — Requirements 10 & 11 (2026-06-20)
|
||||
|
||||
> **Merge note (as-built correction).** The handler shipped as `ctrlstatus_fill` + `handle_ctrlstatus` in **`src/controller/ctrlstatus.c`** (not `reply.c`); `configChecksum` is carried as **Text** lowercase-hex (the `Data` capnp type maps to the project's structured `xce_data_t`); `primaryName` is **role-derived** (primary → self, else empty) after E2E fix `ade0081` — `ha->primary_host` is vestigial. The Req 10/11 changes are written against this as-built reality; the Reqs 1–9 design above stands as the original record.
|
||||
|
||||
### What changed and why
|
||||
Live E2E (2026-06-20): a standby's `ctrlstatus` reported `committedLsn: 0` / `committedConfigVersion: 0` while `appliedLsn: 38` — an apparent commit/apply inversion — because `ctrlstatus_fill` reads the **primary-only** markers `persist.committed_lsn` (`ctrlstatus.c:130`) and `ha->committed_config_tag` (`:109`), which stay `0` on a node that never led. And there was no responder timestamp to line a captured status up against `ctrl-N.log`. Req 10 fixes the off-primary committed sourcing; Req 11 adds a responder snapshot timestamp.
|
||||
|
||||
### Boundary Commitments (delta)
|
||||
- **This spec owns (added):** the off-primary field-sourcing fix inside `ctrlstatus_fill` (read the *learned* committed values when not primary); a new read-only `CtrlStatus.responderTimeMs` field stamped at snapshot time; its render in `cmd_ctrlstatus`.
|
||||
- **Out of Boundary (firm):** making the off-primary committed value **live-fresh** — i.e. consuming `HeartbeatMsg.committedLsn` / extending `OP_nop` to refresh `last_known_committed_lsn` on every heartbeat — is a heartbeat/replication-protocol change. It violates Req 10.4 and is routed to **ha-heartbeat-liveness** / **ha-repl-hardening**. controller-status reports the honest best-known value only.
|
||||
- **Allowed Dependencies (added):** read access to `ha->last_known_committed_lsn` (`_Atomic uint64_t`, `ha.h:170`) and `ha->last_known_committed_config_tag` (`uint64_t`, `ha.h:172`); `clock_gettime(CLOCK_REALTIME)`; the existing `xcelog.h` wall-clock format for client rendering.
|
||||
- **Revalidation Triggers (added):** any change to where/when `last_known_committed_lsn`/`last_known_committed_config_tag` are written (today: `cluster_apply_ctrlhello_reply` at join/RDHUP, and the election safe-commit path) → re-check the off-primary committed reporting and the staleness caveat.
|
||||
|
||||
### Locked design decisions (from research items)
|
||||
1. **Req 10 sourcing → role-branch (not uniform `max()`).** When `responderRole == CTRL_PRIMARY`, keep the node's own authoritative live markers (`persist.committed_lsn`, `ha->committed_config_tag`). Otherwise report the learned pair (`ha->last_known_committed_lsn`, `ha->last_known_committed_config_tag`). Rationale: a standby's learned value is its only honest "committed" knowledge; a primary's own marker is authoritative and must not be shadowed by a possibly-stale learned value. (Mirrors the `last_known_*` read precedent at `ha.c:2570`, `ha.c:4126`.)
|
||||
2. **Req 10 staleness caveat (documented behavior, not a bug).** `last_known_committed_lsn` is refreshed at **CtrlHello join / RDHUP-failover**, not per heartbeat, so an off-primary committed value MAY lag the live primary between joins. This is the in-boundary contract; live-fresh is the out-of-boundary item above. The `0`-inversion is eliminated regardless (a standby that has joined has a non-zero learned value; a never-joined node correctly reports `0` = unknown, per Req 1.3 / 10.2).
|
||||
3. **Req 11 wire type → `UInt64` epoch-milliseconds.** Numeric, locale-safe; the client formats it. (Pre-formatted `Text` was the alternative; rejected to keep the wire value machine-comparable and avoid encoding a locale into the reply.)
|
||||
4. **Req 11 clock → `CLOCK_REALTIME`, stamped in `ctrlstatus_fill` at snapshot time.** NOT `CLOCK_MONOTONIC` (which `ha_on_heartbeat_received` uses for liveness) — monotonic is not wall-clock and cannot be correlated to logs. The client renders the millis to local `YYYY-MM-DD HH:MM:SS,mmm` using the `xcelog.h` convention so it is directly comparable to a log line.
|
||||
|
||||
### Research item carried to implementation
|
||||
- Confirm the exact wall-clock formatter/helper in `src/include/xcelog.h` and reuse it (or match its `YYYY-MM-DD HH:MM:SS,mmm` form) in `cmd_ctrlstatus` so the printed timestamp is byte-comparable to `ctrl-N.log`. `last_known_committed_config_tag` is non-atomic (`ha.h:172`, written under lock at `ha_convergence.c:377`); the status read is a best-effort aligned 8-byte load (acceptable for a reporting snapshot) — do not add locking on the read path.
|
||||
|
||||
### Schema delta (Cap'n Proto)
|
||||
Append one field to `CtrlStatus` (next free ordinal `@13`, after `peerCount @12`; append-only, never reorder):
|
||||
```capnp
|
||||
responderTimeMs @13: UInt64 $C.mapname("responder_time_ms"); # CLOCK_REALTIME epoch millis at snapshot (Req 11.1)
|
||||
```
|
||||
No change to `committedLsn @8` / `committedConfigVersion @7` wire shape — Req 10 changes only the **value source**, not the field.
|
||||
|
||||
### `ctrlstatus_fill` delta — Service contract (`src/controller/ctrlstatus.c`)
|
||||
- **Req 10 (committed sourcing):** replace the unconditional reads at `ctrlstatus.c:108–110,130` with the role-branch above:
|
||||
- `responderRole == CTRL_PRIMARY` → `committedLsn = persist.committed_lsn`, `committedConfigVersion = ha->committed_config_tag` (atomic acquire loads, unchanged).
|
||||
- else → `committedLsn = ha->last_known_committed_lsn` (atomic acquire), `committedConfigVersion = ha->last_known_committed_config_tag` (plain aligned load).
|
||||
- `appliedLsn` (`ha->applied_lsn`) is unchanged — it is already correct in every role.
|
||||
- **Req 11 (timestamp):** stamp `st->responder_time_ms` from `clock_gettime(CLOCK_REALTIME)` (`tv_sec*1000 + tv_nsec/1e6`) at the top of the snapshot, before/with the other field loads, so it marks the responder's sample moment.
|
||||
- **Read-only invariant (10.4, 1.2) preserved:** all additions are atomic/aligned loads + a clock read; no HA/persist/config mutation.
|
||||
|
||||
### `cmd_ctrlstatus` delta — State contract (`src/controller/testsuite/test.c`)
|
||||
- Print one added line rendering `responder_time_ms` as local `YYYY-MM-DD HH:MM:SS,mmm` (xcelog.h form) alongside the existing fields (Req 11.3). The committed lines need no rendering change — Req 10 only changes the values they already print.
|
||||
|
||||
### File Structure Plan (delta)
|
||||
**Modified files:**
|
||||
- `src/schema/controller.capnp` — add `responderTimeMs @13: UInt64` to `CtrlStatus` (regen `controller.capnp.h/.c` → `xce_ctrlstatus_t.responder_time_ms`).
|
||||
- `src/controller/ctrlstatus.c` — `ctrlstatus_fill`: role-branch committed sourcing (Req 10) + `CLOCK_REALTIME` timestamp stamp (Req 11).
|
||||
- `src/controller/testsuite/test.c` — `cmd_ctrlstatus`: render the responder timestamp line (Req 11.3).
|
||||
- `src/controller/testsuite/test_ctrlstatus.c` — add/extend unit tests (below).
|
||||
|
||||
No new files (the capnp field is generated; all logic slots into existing TUs).
|
||||
|
||||
### Requirements Traceability (delta)
|
||||
| Requirement | Summary | Components | Interfaces |
|
||||
|-------------|---------|------------|------------|
|
||||
| 10.1 | Off-primary reports *learned* committed LSN + config version | `ctrlstatus_fill` role-branch | `CtrlStatus.committedLsn`, `committedConfigVersion` (value source = `ha->last_known_committed_*`) |
|
||||
| 10.2 | No learned value yet → 0/unknown | `ctrlstatus_fill` | `last_known_committed_lsn == 0` (correct unknown, per 1.3) |
|
||||
| 10.3 | No spurious applied>committed inversion | `ctrlstatus_fill` | `appliedLsn` vs role-correct `committedLsn` |
|
||||
| 10.4 | Read-only reporting; no production/replication change | `ctrlstatus_fill`; out-of-boundary routing | atomic/aligned loads only |
|
||||
| 11.1 | Responder wall-clock at sample time | `ctrlstatus_fill` (`CLOCK_REALTIME`) | `CtrlStatus.responderTimeMs` |
|
||||
| 11.2 | Same clock/format as logs | `cmd_ctrlstatus` render (xcelog.h form) | local `YYYY-MM-DD HH:MM:SS,mmm` |
|
||||
| 11.3 | Test client prints the timestamp | `cmd_ctrlstatus` | added print line |
|
||||
|
||||
### Testing Strategy (delta)
|
||||
**Unit (`test_ctrlstatus.c`, TDD red→green):**
|
||||
- STANDBY with `last_known_committed_lsn = 36`, `persist.committed_lsn = 0`, `applied_lsn = 38` → reply reports `committedLsn == 36` (not 0) and `appliedLsn == 38` (no inversion below the learned value) (10.1, 10.3).
|
||||
- STANDBY pre-join (`last_known_committed_lsn == 0`) → reply reports `committedLsn == 0` / `committedConfigVersion == 0` (correct unknown) (10.2).
|
||||
- PRIMARY → reply reports `committedLsn == persist.committed_lsn` and `committedConfigVersion == ha->committed_config_tag` (authoritative own markers unchanged) (10.1 primary branch).
|
||||
- `committedConfigVersion` off-primary equals `last_known_committed_config_tag` (symmetry with the LSN) (10.1).
|
||||
- `responder_time_ms` is non-zero and within a tolerance of a test-captured `CLOCK_REALTIME` sample; the handler still mutates no state (11.1, 1.2/10.4).
|
||||
|
||||
**Integration / round-trip (libxce):** `xcecli_ctrlstatus` decodes the new `responderTimeMs` field end-to-end (no manual decode — codegen), proving wire compatibility.
|
||||
|
||||
**E2E (live `/xce-deploy-ha`):** reproduce the original finding — submit jobs so the primary commits, query a **standby**; assert the standby's `committedLsn` is non-zero and tracks (within the join-refresh caveat) rather than `0`, and that the printed `responder timestamp` matches the standby's own `ctrl-N.log` line for that query.
|
||||
|
|
|
|||
|
|
@ -1,31 +1,117 @@
|
|||
# Requirements Document
|
||||
|
||||
## Project Description (Input)
|
||||
## Introduction
|
||||
|
||||
Add an LSF-`lsid`-style controller status opcode (C tier only) so a client can query basic cluster/controller identity and health in one cheap call.
|
||||
This feature adds a **controller status query** (an XCE analog of LSF's `lsid`): a single, cheap request that returns basic cluster/controller identity and health — which node is the current primary, the answering node's identity and role, the configured controllers (peers), the committed configuration version and a content checksum, and a few HA health/progress fields. Today no such query exists: a client must infer "is there a serving primary" indirectly (for example, by listing jobs), and there is no single call that reports who the primary is, the configured controller topology, or the committed configuration version. This makes operational visibility and failover-aware test orchestration awkward.
|
||||
|
||||
**Returns:**
|
||||
- Which node is the current PRIMARY (name; empty/none if mid-election)
|
||||
- The responding node's role (PRIMARY/STANDBY/ELECTING/INIT)
|
||||
- The configured controllers/peers (name + host:port list and count)
|
||||
- The configuration version (the HA committed `config_tag`)
|
||||
- Additional status: current term (election epoch), `committed_lsn`/`applied_lsn` (catch-up/health), cluster name, serving/fenced state
|
||||
The status query is served by **any** controller node — a standby reports who the primary is from the state it already tracks — and rides the existing client failover behavior, so a caller can get status without knowing which node is primary. The test client also gains a command that prints this status, and its `wait-primary` probe switches to this query so that waiting for a primary is cheaper and additionally reports which node became the serving primary.
|
||||
|
||||
Answerable by ANY node (a standby reports who the primary is from its ctrlhello/heartbeat state), so `lsid` works even when the query doesn't land on the primary; it is a failover-aware READ like the existing `clusterconf` op. All the underlying data already exists in `src/controller/ha.h` (`ha->role`, `ha->term`, `committed_config_tag`, `committed_lsn`, peer/candidate list).
|
||||
## Boundary Context
|
||||
|
||||
**Components (C tier):**
|
||||
1. `src/schema/controller.capnp` — new `CtrlInfo`/`CtrlStatus` struct + request opcode + reply-union member, modeled on the existing `clusterconf` opcode (`controller.capnp:489`).
|
||||
2. Controller `handle_ctrlinfo` handler mirroring `handle_clusterconf`.
|
||||
3. libxce `xce_ctrlinfo_req` / `xce_ctrlinfo` / `xcecli_ctrlinfo` (+ free helper) as a failover-aware read.
|
||||
4. `src/controller/testsuite/test.c` — a `ctrlinfo`/`lsid` client command that prints the status, AND switch `cmd_wait_primary`'s probe from `joblist_query` to this lighter, more-direct status op. (`wait-primary` currently piggybacks on `joblist` purely to avoid a new opcode; with this op it becomes a cheap "is a serving primary present + who" probe that can also report the elected primary and term — better diagnostics, directly useful for the `ha-commit-durability` Phase 6 liveness / >60s-election investigation.)
|
||||
|
||||
**KEY DESIGN DECISION the spec must resolve:** whether to EXTEND the existing `clusterconf` opcode (which already returns much richer cluster detail — name/resources/queues/workers) with these HA-status fields, OR add a separate lean status op. `lsid` wants a fast focused identity/health view, not the full resource/queue/worker dump, so the leaning is a dedicated lean op, but the spec should weigh reuse vs separation explicitly.
|
||||
|
||||
**IN SCOPE:** C tier only (controller opcode + libxce + `xce_test` command + `wait-primary` integration).
|
||||
|
||||
**OUT OF SCOPE:** the Go CLI/REST `xce lsid` surface (a later pass); changing HA/election/replication behavior (this only READS existing HA state); structured/JSON output beyond the capnp reply.
|
||||
|
||||
This is the "controller status/probe opcode" that the `xce-test-e2e-primitives` spec explicitly deferred as out-of-scope.
|
||||
- **In scope**: a client-observable controller status query returning the current primary identity, the answering node's identity and role, the configured controller topology (peers + count), the committed configuration version and content checksum, and HA health (current term, commit/apply progress, serving/fenced state, cluster name); off-primary fidelity of the reported committed positions (a standby reports the committed position it has learned from the primary, not an unset node-local value); a responder snapshot timestamp for log correlation; failover-aware delivery of that query; a test-client command that prints the status; and `wait-primary` using the status query as its probe.
|
||||
- **Out of scope**: an operator-facing CLI/REST status surface (a later pass); any change to HA election, replication, or commit behavior (this query only **reads** existing controller state); presentation/output formats beyond the query's structured reply and the test client's human-readable print.
|
||||
- **Adjacent expectations**: relies on the existing client master-rotation/failover already used by other read operations; consumes the controller's existing HA state (current primary, role, term, committed configuration version, commit/apply progress, fenced state) without changing how that state is produced or maintained. The status query is a **focused identity/health view** and is intentionally distinct from the existing full cluster-configuration listing (which enumerates resources, queues, and workers).
|
||||
|
||||
## Requirements
|
||||
<!-- Will be generated in /kiro-spec-requirements phase -->
|
||||
|
||||
### Requirement 1: Single-call controller status query
|
||||
|
||||
**Objective:** As an operator or test author, I want to retrieve basic controller and cluster status in a single request, so that I can see cluster identity and health without inferring it from unrelated operations.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When a client issues a controller status query, the Controller shall return, in a single request/response exchange, a response containing the status fields defined in Requirements 2 through 6.
|
||||
2. The Controller shall treat the status query as read-only and shall not modify any cluster or controller state as a result of answering it.
|
||||
3. If the responding controller cannot determine the value of a status field, then the Controller shall return a well-defined "unknown" value for that field rather than omitting it or returning uninitialized data.
|
||||
4. The status response shall be limited to controller/cluster identity and health fields and shall not include the full resource, queue, or worker inventory.
|
||||
|
||||
### Requirement 2: Current primary identity
|
||||
|
||||
**Objective:** As an operator, I want the status response to identify the current primary, so that I know which node is authoritative.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. While a primary is currently elected and serving, when a client issues a status query, the Controller shall report the current primary's identity in the response.
|
||||
2. If no primary has currently been elected, then the Controller shall report the primary identity as "unknown" rather than a stale or guessed value.
|
||||
3. When the status query is answered by a standby node that knows the current primary, the Controller shall report that primary's identity; if the standby does not currently know a primary, the Controller shall report the primary identity as "unknown".
|
||||
|
||||
### Requirement 3: Responding node identity and role
|
||||
|
||||
**Objective:** As an operator, I want to know which node answered and its role, so that I can interpret the response in context without confusing it with the authoritative cluster view.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When a client issues a status query, the Controller shall report the identity (name) of the node that answered the query.
|
||||
2. When a client issues a status query, the Controller shall report the answering node's current role as one of: primary, standby, electing, or initializing.
|
||||
3. The responding node's role shall be presented as a property of the answering node (for example, "answered by <node> (<role>)") and shall be distinct from the authoritative current-primary identity of Requirement 2, so that a transient answering-node role (such as electing or initializing) is not misread as the cluster's primary state.
|
||||
|
||||
### Requirement 4: Configured controller topology
|
||||
|
||||
**Objective:** As an operator, I want the configured controllers, so that I can see the cluster's intended membership and size.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When a client issues a status query, the Controller shall report the list of configured controllers, each identified by its name and network address.
|
||||
2. When a client issues a status query, the Controller shall report the total count of configured controllers.
|
||||
|
||||
### Requirement 5: Committed configuration version and checksum
|
||||
|
||||
**Objective:** As an operator, I want the configuration version and a content checksum, so that I can confirm which configuration the cluster has committed and detect content divergence between nodes even when their version numbers match.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When a client issues a status query, the Controller shall report the committed configuration version currently in effect on the responding node.
|
||||
2. When a client issues a status query, the Controller shall report a checksum (CRC) of the committed configuration content currently in effect on the responding node.
|
||||
3. The reported configuration checksum shall be derived from the configuration content such that two nodes with identical committed configuration content report the same checksum, and two nodes with differing content report different checksums (enabling detection of same-version content divergence).
|
||||
|
||||
### Requirement 6: HA health and progress fields
|
||||
|
||||
**Objective:** As an operator, I want basic HA health and progress, so that I can assess election state and catch-up.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When a client issues a status query, the Controller shall report the current term (election epoch).
|
||||
2. When a client issues a status query, the Controller shall report its commit progress position and its apply progress position.
|
||||
3. While the responding node is in a fenced or non-promotable state, when a client issues a status query, the Controller shall report that fenced state in the response.
|
||||
4. When a client issues a status query, the Controller shall report the cluster name.
|
||||
|
||||
### Requirement 7: Failover-aware, any-node delivery
|
||||
|
||||
**Objective:** As a client, I want the status query to reach a reachable controller automatically, so that I can get status even across failover or when I do not know which node is primary.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When a client issues a status query, the XCE client shall attempt the configured controllers using the same failover/rotation behavior used by other read operations until one responds or the attempts are exhausted.
|
||||
2. When any configured controller, whether primary or standby, is reachable, the XCE client shall return that controller's status response.
|
||||
3. If no configured controller is reachable within the client's failover budget, then the XCE client shall report a typed failure rather than reporting success with empty status.
|
||||
|
||||
### Requirement 8: Test-client status command
|
||||
|
||||
**Objective:** As a test author, I want a client command that prints the controller status, so that I can inspect cluster identity and health from a script.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When the status command is issued in the test client, the test client shall print, in human-readable form, the current primary identity, the answering node's identity and role (as "answered by"), the configured controllers and their count, the committed configuration version, the committed configuration checksum, the current term, the commit/apply progress, the fenced state, and the cluster name.
|
||||
2. When the underlying status query succeeds, the test client command shall report success (zero status).
|
||||
3. If the underlying status query fails, then the test client command shall report failure (non-zero status) and shall print the error.
|
||||
|
||||
### Requirement 9: wait-primary uses the status query
|
||||
|
||||
**Objective:** As a test author, I want `wait-primary` to probe via the status query, so that the wait is cheaper and reports which node became the serving primary.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When `wait-primary` probes the cluster, the wait-primary command shall use the controller status query, rather than a job-listing operation, to determine whether a serving primary is present.
|
||||
2. When the status query reports a serving primary within the timeout, the wait-primary command shall complete successfully and shall report the identity of the serving primary.
|
||||
3. If no serving primary is reported within the timeout, then the wait-primary command shall fail.
|
||||
4. The change to the wait-primary probe shall preserve its existing timeout-argument handling and its success/failure exit semantics, adding only the served-primary identity to its success output.
|
||||
|
||||
### Requirement 10: Off-primary committed-state fidelity
|
||||
|
||||
**Objective:** As an operator, I want a standby's reported committed positions to reflect what it has actually learned from the primary, so that the status does not show a misleading committed value of zero while the node has clearly applied replicated entries.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. While the responding node is not the current primary and has learned a committed position from the primary, when a client issues a status query, the Controller shall report the committed configuration version and commit progress position that it has learned from the primary, rather than a node-local committed value that remains unset on a node that has never served as primary.
|
||||
2. If the responding node is not the current primary and has not yet learned any committed position from a primary, then the Controller shall report the committed configuration version and commit progress position as the well-defined "unknown"/zero value (consistent with Requirement 1.3), rather than a stale or guessed value.
|
||||
3. While the responding node is actively applying replicated entries, the Controller shall source the reported commit progress position and apply progress position consistently, so that the reported commit position is not spuriously lower than the position the node has actually committed or learned — avoiding an apparent "applied greater than committed" inversion that does not reflect the node's real state.
|
||||
4. The committed-state fidelity in this requirement shall be a read-only reporting concern only and shall not change how committed positions are produced, replicated, or persisted.
|
||||
|
||||
### Requirement 11: Responder snapshot timestamp for log correlation
|
||||
|
||||
**Objective:** As an operator or test author, I want each status response to carry the responding node's clock time at the moment it sampled its state, so that I can line a captured status up against that node's logs.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When a client issues a status query, the Controller shall include in the response the responding node's wall-clock time at the moment the status fields were sampled.
|
||||
2. The reported timestamp shall be expressed on the same clock and in a form that an operator can directly compare against the responding controller's log timestamps.
|
||||
3. When the status command is issued in the test client, the test client shall print the responder snapshot timestamp in human-readable form alongside the other status fields.
|
||||
|
|
|
|||
|
|
@ -91,3 +91,63 @@ Two review issues resolved with the user; design.md updated:
|
|||
2. **Config fields → loaded pair + committed tag (user decision).** Report `configVersion`+`configChecksum` from the LOADED config (self-consistent: version = first 8 bytes of the checksum) AND `committedConfigVersion` (`committed_config_tag`) separately — "what I run" vs "what is committed". Avoids the earlier mismatched committed-tag + loaded-checksum pairing.
|
||||
|
||||
Also fixed: mermaid render issues — removed `/`, parentheses, and quotes from diagram labels/messages (the LR architecture graph + the new redirect sequence diagram).
|
||||
|
||||
---
|
||||
|
||||
## Gap Analysis — Delta for Requirements 10 & 11 (2026-06-20)
|
||||
|
||||
Reopened after a live HA-deploy E2E finding: a standby answering `ctrlstatus` showed
|
||||
`committedLsn: 0` / `committedConfigVersion: 0` while `appliedLsn: 38` (an apparent
|
||||
applied-greater-than-committed inversion), and there was no way to correlate a captured
|
||||
status against the per-node logs. Scope of this delta = Req 10 (off-primary committed
|
||||
fidelity) + Req 11 (responder snapshot timestamp). Reqs 1–9 are implemented and unchanged.
|
||||
|
||||
### Requirement-to-Asset Map
|
||||
|
||||
| Need (Req) | Existing asset | Gap |
|
||||
|---|---|---|
|
||||
| 10.1 standby reports *learned* committed LSN | `ctrlstatus_fill` reads `c->persist.committed_lsn` (`ctrlstatus.c:130`) — a **primary-only** marker, 0 on a node that never led | **Constraint**: wrong source field; the learned value `ha->last_known_committed_lsn` (`ha.h:170`) exists but is unused here |
|
||||
| 10.1 standby reports *learned* committed config tag | `ctrlstatus_fill` reads `ha->committed_config_tag` (`ctrlstatus.c:109`) — also primary-only | **Constraint**: learned `ha->last_known_committed_config_tag` (`ha.h:172`) exists but is unused here |
|
||||
| 10.1 freshness of the learned value | `last_known_committed_lsn` written only at `cluster_apply_ctrlhello_reply` (`cluster.c:291`, `ha_convergence.c:375`) and an election safe-commit path (`ha.c:925`) | **Constraint (key)**: refreshed only at **CtrlHello join / RDHUP-failover handshake**, NOT per heartbeat. The `OP_nop` heartbeat (`cluster.c:300`) carries no committed payload; the richer `HeartbeatMsg{committedLsn@2, committedConfigTag@3}` (`controller.capnp:761`) is **defined but never consumed** on receive → the learned value can LAG between joins |
|
||||
| 10.2 genuinely-unknown → 0/unknown | existing `NULLSTR`/0 conventions (Req 1.3) | none — `last_known_committed_lsn` is 0 before first learn, which is the correct "unknown" |
|
||||
| 11.1 responder wall-clock at sample time | none on `CtrlStatus` (schema `controller.capnp:496`, fields @0–@12; next free ordinal **@13**) | **Missing**: new field + a `CLOCK_REALTIME` stamp in `ctrlstatus_fill`. Note `ha_on_heartbeat_received` uses `CLOCK_MONOTONIC` — **not** reusable for log correlation |
|
||||
| 11.2 same clock/format as logs | logger header `src/include/xcelog.h` produces `[YYYY-MM-DD HH:MM:SS,mmm]` | **Research Needed**: confirm the exact formatter/helper in `xcelog.h` to reuse (or match) for byte-comparable output |
|
||||
| 11.3 test client prints timestamp | `cmd_ctrlstatus` in `src/controller/testsuite/test.c` (C `xce_test` client; print block ~line 1297) — the "client" is xce_test, not Go | **Missing**: one print line; libxce decode is codegen from schema (auto after regen) |
|
||||
|
||||
### Implementation Approach Options
|
||||
|
||||
**Req 10 — Option A (Extend `ctrlstatus_fill`) — RECOMMENDED.** When `responderRole != CTRL_PRIMARY`,
|
||||
source `committedLsn` from `ha->last_known_committed_lsn` and `committedConfigVersion` from
|
||||
`ha->last_known_committed_config_tag`; keep the primary's own `persist.committed_lsn` /
|
||||
`committed_config_tag` when it IS primary (those are the authoritative live values). Read-only,
|
||||
satisfies Req 10.4, mirrors the existing `last_known_*` usage in the RDHUP/freshness paths
|
||||
(`ha.c:2570`, `ha.c:4126`). A simpler variant — report `max(persist.committed_lsn,
|
||||
last_known_committed_lsn)` uniformly (no role branch) — also kills the 0-inversion; design picks.
|
||||
*Trade-off*: ✅ tiny, in-boundary, fixes the inversion. ❌ value can lag between joins (see below).
|
||||
|
||||
**Req 10 — out-of-boundary (do NOT pull into this spec).** Making the off-primary committed
|
||||
value *live-fresh* requires consuming `HeartbeatMsg.committedLsn` (or extending `OP_nop`) and
|
||||
storing it into `last_known_committed_lsn` on **every** heartbeat — a heartbeat/replication
|
||||
protocol change. That violates Req 10.4 ("shall not change how committed positions are produced,
|
||||
replicated, or persisted") and belongs to **ha-heartbeat-liveness** / **ha-repl-hardening**.
|
||||
controller-status should report the honest best-known value and the design must state the lag caveat.
|
||||
|
||||
**Req 11 — Option A (Extend) — RECOMMENDED.** Add `responderTime @13: UInt64` (epoch millis) to
|
||||
`CtrlStatus`; stamp it in `ctrlstatus_fill` from `CLOCK_REALTIME`; render in `cmd_ctrlstatus`
|
||||
formatted via the `xcelog.h` convention so it lines up with `ctrl-N.log`. *Design decision*:
|
||||
numeric epoch-millis (client formats — locale-safe, clean) vs pre-formatted `Text` (zero client
|
||||
work, mirrors the `configChecksum`-as-Text precedent). Recommend numeric epoch-millis.
|
||||
*Trade-off*: ✅ one field + one stamp + one print line. ❌ adds a schema ordinal (codegen regen).
|
||||
|
||||
### Effort & Risk
|
||||
|
||||
- **Req 10**: **S** (≤1 day) — one function's field sourcing + TDD (standby-known-primary asserts learned value; pre-learn asserts 0). **Risk Low** — read-only, no HA-state change.
|
||||
- **Req 11**: **S** (1–2 days) — schema field + `ctrlstatus_fill` stamp + `cmd_ctrlstatus` print + codegen regen + round-trip decode test. **Risk Low**.
|
||||
- **Combined**: **S**. The one **Medium-risk temptation to refuse**: expanding into heartbeat committed-propagation — keep it out of this spec.
|
||||
|
||||
### Research Items to Carry to Design
|
||||
|
||||
1. Confirm the exact wall-clock formatter/helper in `src/include/xcelog.h`; reuse it (or document the chosen format) so Req 11's value is byte-comparable to log lines.
|
||||
2. Decide Req 11 wire type: `UInt64` epoch-millis (recommended) vs `Text`.
|
||||
3. Decide Req 10 sourcing: role-branch (`persist.*` on primary, `last_known_*` off-primary) vs uniform `max()`; keep LSN and config-tag symmetric.
|
||||
4. State the **staleness caveat** in design (learned value refreshes at join, not per heartbeat) and explicitly route live-fresh-committed to **ha-heartbeat-liveness / ha-repl-hardening** as out-of-boundary.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue