lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

checkpoint: bug-851 root cause + raft TLA+ conformance spec family

Browse source 
Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
This commit is contained in:
Rongsong Shen 2026-06-21 14:02:15 +00:00
parent 7d70820aac
commit 20e59d0b42
85 changed files with 10525 additions and 1993 deletions
Download patch file Download diff file Expand all files Collapse all files

60
.kiro/specs/controller-status/research.md
View file

@ -91,3 +91,63 @@ Two review issues resolved with the user; design.md updated:
2. **Config fields → loaded pair + committed tag (user decision).** Report `configVersion`+`configChecksum` from the LOADED config (self-consistent: version = first 8 bytes of the checksum) AND `committedConfigVersion` (`committed_config_tag`) separately — "what I run" vs "what is committed". Avoids the earlier mismatched committed-tag + loaded-checksum pairing.
Also fixed: mermaid render issues — removed `/`, parentheses, and quotes from diagram labels/messages (the LR architecture graph + the new redirect sequence diagram).
---
## Gap Analysis — Delta for Requirements 10 & 11 (2026-06-20)
Reopened after a live HA-deploy E2E finding: a standby answering `ctrlstatus` showed
`committedLsn: 0` / `committedConfigVersion: 0` while `appliedLsn: 38` (an apparent
applied-greater-than-committed inversion), and there was no way to correlate a captured
status against the per-node logs. Scope of this delta = Req 10 (off-primary committed
fidelity) + Req 11 (responder snapshot timestamp). Reqs 1–9 are implemented and unchanged.
### Requirement-to-Asset Map
| Need (Req) | Existing asset | Gap |
|---|---|---|
| 10.1 standby reports *learned* committed LSN | `ctrlstatus_fill` reads `c->persist.committed_lsn` (`ctrlstatus.c:130`) — a **primary-only** marker, 0 on a node that never led | **Constraint**: wrong source field; the learned value `ha->last_known_committed_lsn` (`ha.h:170`) exists but is unused here |
| 10.1 standby reports *learned* committed config tag | `ctrlstatus_fill` reads `ha->committed_config_tag` (`ctrlstatus.c:109`) — also primary-only | **Constraint**: learned `ha->last_known_committed_config_tag` (`ha.h:172`) exists but is unused here |
| 10.1 freshness of the learned value | `last_known_committed_lsn` written only at `cluster_apply_ctrlhello_reply` (`cluster.c:291`, `ha_convergence.c:375`) and an election safe-commit path (`ha.c:925`) | **Constraint (key)**: refreshed only at **CtrlHello join / RDHUP-failover handshake**, NOT per heartbeat. The `OP_nop` heartbeat (`cluster.c:300`) carries no committed payload; the richer `HeartbeatMsg{committedLsn@2, committedConfigTag@3}` (`controller.capnp:761`) is **defined but never consumed** on receive → the learned value can LAG between joins |
| 10.2 genuinely-unknown → 0/unknown | existing `NULLSTR`/0 conventions (Req 1.3) | none — `last_known_committed_lsn` is 0 before first learn, which is the correct "unknown" |
| 11.1 responder wall-clock at sample time | none on `CtrlStatus` (schema `controller.capnp:496`, fields @0–@12; next free ordinal **@13**) | **Missing**: new field + a `CLOCK_REALTIME` stamp in `ctrlstatus_fill`. Note `ha_on_heartbeat_received` uses `CLOCK_MONOTONIC` — **not** reusable for log correlation |
| 11.2 same clock/format as logs | logger header `src/include/xcelog.h` produces `[YYYY-MM-DD HH:MM:SS,mmm]` | **Research Needed**: confirm the exact formatter/helper in `xcelog.h` to reuse (or match) for byte-comparable output |
| 11.3 test client prints timestamp | `cmd_ctrlstatus` in `src/controller/testsuite/test.c` (C `xce_test` client; print block ~line 1297) — the "client" is xce_test, not Go | **Missing**: one print line; libxce decode is codegen from schema (auto after regen) |
### Implementation Approach Options
**Req 10 — Option A (Extend `ctrlstatus_fill`) — RECOMMENDED.** When `responderRole != CTRL_PRIMARY`,
source `committedLsn` from `ha->last_known_committed_lsn` and `committedConfigVersion` from
`ha->last_known_committed_config_tag`; keep the primary's own `persist.committed_lsn` /
`committed_config_tag` when it IS primary (those are the authoritative live values). Read-only,
satisfies Req 10.4, mirrors the existing `last_known_*` usage in the RDHUP/freshness paths
(`ha.c:2570`, `ha.c:4126`). A simpler variant — report `max(persist.committed_lsn,
last_known_committed_lsn)` uniformly (no role branch) — also kills the 0-inversion; design picks.
*Trade-off*: ✅ tiny, in-boundary, fixes the inversion. ❌ value can lag between joins (see below).
**Req 10 — out-of-boundary (do NOT pull into this spec).** Making the off-primary committed
value *live-fresh* requires consuming `HeartbeatMsg.committedLsn` (or extending `OP_nop`) and
storing it into `last_known_committed_lsn` on **every** heartbeat — a heartbeat/replication
protocol change. That violates Req 10.4 ("shall not change how committed positions are produced,
replicated, or persisted") and belongs to **ha-heartbeat-liveness** / **ha-repl-hardening**.
controller-status should report the honest best-known value and the design must state the lag caveat.
**Req 11 — Option A (Extend) — RECOMMENDED.** Add `responderTime @13: UInt64` (epoch millis) to
`CtrlStatus`; stamp it in `ctrlstatus_fill` from `CLOCK_REALTIME`; render in `cmd_ctrlstatus`
formatted via the `xcelog.h` convention so it lines up with `ctrl-N.log`. *Design decision*:
numeric epoch-millis (client formats — locale-safe, clean) vs pre-formatted `Text` (zero client
work, mirrors the `configChecksum`-as-Text precedent). Recommend numeric epoch-millis.
*Trade-off*: ✅ one field + one stamp + one print line. ❌ adds a schema ordinal (codegen regen).
### Effort & Risk
- **Req 10**: **S** (≤1 day) — one function's field sourcing + TDD (standby-known-primary asserts learned value; pre-learn asserts 0). **Risk Low** — read-only, no HA-state change.
- **Req 11**: **S** (1–2 days) — schema field + `ctrlstatus_fill` stamp + `cmd_ctrlstatus` print + codegen regen + round-trip decode test. **Risk Low**.
- **Combined**: **S**. The one **Medium-risk temptation to refuse**: expanding into heartbeat committed-propagation — keep it out of this spec.
### Research Items to Carry to Design
1. Confirm the exact wall-clock formatter/helper in `src/include/xcelog.h`; reuse it (or document the chosen format) so Req 11's value is byte-comparable to log lines.
2. Decide Req 11 wire type: `UInt64` epoch-millis (recommended) vs `Text`.
3. Decide Req 10 sourcing: role-branch (`persist.*` on primary, `last_known_*` off-primary) vs uniform `max()`; keep LSN and config-tag symmetric.
4. State the **staleness caveat** in design (learned value refreshes at join, not per heartbeat) and explicitly route live-fresh-committed to **ha-heartbeat-liveness / ha-repl-hardening** as out-of-boundary.