lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

checkpoint: bug-851 root cause + raft TLA+ conformance spec family

Browse source 
Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
This commit is contained in:
Rongsong Shen 2026-06-21 14:02:15 +00:00
parent 7d70820aac
commit 20e59d0b42
85 changed files with 10525 additions and 1993 deletions
Download patch file Download diff file Expand all files Collapse all files

74
.kiro/specs/controller-status/design.md
View file

@ -237,3 +237,77 @@ The only new data is the reply payload above. `CtrlStatus` is a flat value objec
## Security Considerations
`ctrlstatus` exposes cluster topology, role, term, and config version/checksum — the same trust level as the existing `clusterconf`/state-report reads (no secrets, no config content, no job data). It is read-only and goes through the same authenticated client path as other client ops; no new authorization surface is introduced. Broadening the dispatch gate is scoped strictly to this one read opcode.
---
## Design Delta — Requirements 10 & 11 (2026-06-20)
> **Merge note (as-built correction).** The handler shipped as `ctrlstatus_fill` + `handle_ctrlstatus` in **`src/controller/ctrlstatus.c`** (not `reply.c`); `configChecksum` is carried as **Text** lowercase-hex (the `Data` capnp type maps to the project's structured `xce_data_t`); `primaryName` is **role-derived** (primary → self, else empty) after E2E fix `ade0081` — `ha->primary_host` is vestigial. The Req 10/11 changes are written against this as-built reality; the Reqs 1–9 design above stands as the original record.
### What changed and why
Live E2E (2026-06-20): a standby's `ctrlstatus` reported `committedLsn: 0` / `committedConfigVersion: 0` while `appliedLsn: 38` — an apparent commit/apply inversion — because `ctrlstatus_fill` reads the **primary-only** markers `persist.committed_lsn` (`ctrlstatus.c:130`) and `ha->committed_config_tag` (`:109`), which stay `0` on a node that never led. And there was no responder timestamp to line a captured status up against `ctrl-N.log`. Req 10 fixes the off-primary committed sourcing; Req 11 adds a responder snapshot timestamp.
### Boundary Commitments (delta)
- **This spec owns (added):** the off-primary field-sourcing fix inside `ctrlstatus_fill` (read the *learned* committed values when not primary); a new read-only `CtrlStatus.responderTimeMs` field stamped at snapshot time; its render in `cmd_ctrlstatus`.
- **Out of Boundary (firm):** making the off-primary committed value **live-fresh** — i.e. consuming `HeartbeatMsg.committedLsn` / extending `OP_nop` to refresh `last_known_committed_lsn` on every heartbeat — is a heartbeat/replication-protocol change. It violates Req 10.4 and is routed to **ha-heartbeat-liveness** / **ha-repl-hardening**. controller-status reports the honest best-known value only.
- **Allowed Dependencies (added):** read access to `ha->last_known_committed_lsn` (`_Atomic uint64_t`, `ha.h:170`) and `ha->last_known_committed_config_tag` (`uint64_t`, `ha.h:172`); `clock_gettime(CLOCK_REALTIME)`; the existing `xcelog.h` wall-clock format for client rendering.
- **Revalidation Triggers (added):** any change to where/when `last_known_committed_lsn`/`last_known_committed_config_tag` are written (today: `cluster_apply_ctrlhello_reply` at join/RDHUP, and the election safe-commit path) → re-check the off-primary committed reporting and the staleness caveat.
### Locked design decisions (from research items)
1. **Req 10 sourcing → role-branch (not uniform `max()`).** When `responderRole == CTRL_PRIMARY`, keep the node's own authoritative live markers (`persist.committed_lsn`, `ha->committed_config_tag`). Otherwise report the learned pair (`ha->last_known_committed_lsn`, `ha->last_known_committed_config_tag`). Rationale: a standby's learned value is its only honest "committed" knowledge; a primary's own marker is authoritative and must not be shadowed by a possibly-stale learned value. (Mirrors the `last_known_*` read precedent at `ha.c:2570`, `ha.c:4126`.)
2. **Req 10 staleness caveat (documented behavior, not a bug).** `last_known_committed_lsn` is refreshed at **CtrlHello join / RDHUP-failover**, not per heartbeat, so an off-primary committed value MAY lag the live primary between joins. This is the in-boundary contract; live-fresh is the out-of-boundary item above. The `0`-inversion is eliminated regardless (a standby that has joined has a non-zero learned value; a never-joined node correctly reports `0` = unknown, per Req 1.3 / 10.2).
3. **Req 11 wire type → `UInt64` epoch-milliseconds.** Numeric, locale-safe; the client formats it. (Pre-formatted `Text` was the alternative; rejected to keep the wire value machine-comparable and avoid encoding a locale into the reply.)
4. **Req 11 clock → `CLOCK_REALTIME`, stamped in `ctrlstatus_fill` at snapshot time.** NOT `CLOCK_MONOTONIC` (which `ha_on_heartbeat_received` uses for liveness) — monotonic is not wall-clock and cannot be correlated to logs. The client renders the millis to local `YYYY-MM-DD HH:MM:SS,mmm` using the `xcelog.h` convention so it is directly comparable to a log line.
### Research item carried to implementation
- Confirm the exact wall-clock formatter/helper in `src/include/xcelog.h` and reuse it (or match its `YYYY-MM-DD HH:MM:SS,mmm` form) in `cmd_ctrlstatus` so the printed timestamp is byte-comparable to `ctrl-N.log`. `last_known_committed_config_tag` is non-atomic (`ha.h:172`, written under lock at `ha_convergence.c:377`); the status read is a best-effort aligned 8-byte load (acceptable for a reporting snapshot) — do not add locking on the read path.
### Schema delta (Cap'n Proto)
Append one field to `CtrlStatus` (next free ordinal `@13`, after `peerCount @12`; append-only, never reorder):
```capnp
responderTimeMs @13: UInt64 $C.mapname("responder_time_ms"); # CLOCK_REALTIME epoch millis at snapshot (Req 11.1)
```
No change to `committedLsn @8` / `committedConfigVersion @7` wire shape — Req 10 changes only the **value source**, not the field.
### `ctrlstatus_fill` delta — Service contract (`src/controller/ctrlstatus.c`)
- **Req 10 (committed sourcing):** replace the unconditional reads at `ctrlstatus.c:108–110,130` with the role-branch above:
- `responderRole == CTRL_PRIMARY` → `committedLsn = persist.committed_lsn`, `committedConfigVersion = ha->committed_config_tag` (atomic acquire loads, unchanged).
- else → `committedLsn = ha->last_known_committed_lsn` (atomic acquire), `committedConfigVersion = ha->last_known_committed_config_tag` (plain aligned load).
- `appliedLsn` (`ha->applied_lsn`) is unchanged — it is already correct in every role.
- **Req 11 (timestamp):** stamp `st->responder_time_ms` from `clock_gettime(CLOCK_REALTIME)` (`tv_sec*1000 + tv_nsec/1e6`) at the top of the snapshot, before/with the other field loads, so it marks the responder's sample moment.
- **Read-only invariant (10.4, 1.2) preserved:** all additions are atomic/aligned loads + a clock read; no HA/persist/config mutation.
### `cmd_ctrlstatus` delta — State contract (`src/controller/testsuite/test.c`)
- Print one added line rendering `responder_time_ms` as local `YYYY-MM-DD HH:MM:SS,mmm` (xcelog.h form) alongside the existing fields (Req 11.3). The committed lines need no rendering change — Req 10 only changes the values they already print.
### File Structure Plan (delta)
**Modified files:**
- `src/schema/controller.capnp` — add `responderTimeMs @13: UInt64` to `CtrlStatus` (regen `controller.capnp.h/.c` → `xce_ctrlstatus_t.responder_time_ms`).
- `src/controller/ctrlstatus.c` — `ctrlstatus_fill`: role-branch committed sourcing (Req 10) + `CLOCK_REALTIME` timestamp stamp (Req 11).
- `src/controller/testsuite/test.c` — `cmd_ctrlstatus`: render the responder timestamp line (Req 11.3).
- `src/controller/testsuite/test_ctrlstatus.c` — add/extend unit tests (below).
No new files (the capnp field is generated; all logic slots into existing TUs).
### Requirements Traceability (delta)
| Requirement | Summary | Components | Interfaces |
|-------------|---------|------------|------------|
| 10.1 | Off-primary reports *learned* committed LSN + config version | `ctrlstatus_fill` role-branch | `CtrlStatus.committedLsn`, `committedConfigVersion` (value source = `ha->last_known_committed_*`) |
| 10.2 | No learned value yet → 0/unknown | `ctrlstatus_fill` | `last_known_committed_lsn == 0` (correct unknown, per 1.3) |
| 10.3 | No spurious applied>committed inversion | `ctrlstatus_fill` | `appliedLsn` vs role-correct `committedLsn` |
| 10.4 | Read-only reporting; no production/replication change | `ctrlstatus_fill`; out-of-boundary routing | atomic/aligned loads only |
| 11.1 | Responder wall-clock at sample time | `ctrlstatus_fill` (`CLOCK_REALTIME`) | `CtrlStatus.responderTimeMs` |
| 11.2 | Same clock/format as logs | `cmd_ctrlstatus` render (xcelog.h form) | local `YYYY-MM-DD HH:MM:SS,mmm` |
| 11.3 | Test client prints the timestamp | `cmd_ctrlstatus` | added print line |
### Testing Strategy (delta)
**Unit (`test_ctrlstatus.c`, TDD red→green):**
- STANDBY with `last_known_committed_lsn = 36`, `persist.committed_lsn = 0`, `applied_lsn = 38` → reply reports `committedLsn == 36` (not 0) and `appliedLsn == 38` (no inversion below the learned value) (10.1, 10.3).
- STANDBY pre-join (`last_known_committed_lsn == 0`) → reply reports `committedLsn == 0` / `committedConfigVersion == 0` (correct unknown) (10.2).
- PRIMARY → reply reports `committedLsn == persist.committed_lsn` and `committedConfigVersion == ha->committed_config_tag` (authoritative own markers unchanged) (10.1 primary branch).
- `committedConfigVersion` off-primary equals `last_known_committed_config_tag` (symmetry with the LSN) (10.1).
- `responder_time_ms` is non-zero and within a tolerance of a test-captured `CLOCK_REALTIME` sample; the handler still mutates no state (11.1, 1.2/10.4).
**Integration / round-trip (libxce):** `xcecli_ctrlstatus` decodes the new `responderTimeMs` field end-to-end (no manual decode — codegen), proving wire compatibility.
**E2E (live `/xce-deploy-ha`):** reproduce the original finding — submit jobs so the primary commits, query a **standby**; assert the standby's `committedLsn` is non-zero and tracks (within the join-refresh caveat) rather than `0`, and that the printed `responder timestamp` matches the standby's own `ctrl-N.log` line for that query.