Commit graph

19 commits

Author SHA1 Message Date
20e59d0b42 checkpoint: bug-851 root cause + raft TLA+ conformance spec family
Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
2026-06-21 14:02:15 +00:00
9a0fec667b chore(controller-status): record delta impl status (6.1-6.3 done, 6.4 pending live)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-20 09:31:29 +00:00
4388d99306 docs(controller-status): mark 6.4 pending live-cluster validation (manual env)
6.1-6.3 committed + unit-green. 6.4 (off-primary committed + log correlation
E2E) needs a real-fs /xce-deploy-ha run; sandbox overlayfs fdatasync errno=5
(bug-795) breaks the commit path. Hand-off recipe recorded in Implementation Notes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-20 09:30:44 +00:00
ee30765d31 feat(controller-status): render responderTime in cmd_ctrlstatus, log-comparable format (task 6.3)
Add ctrlstatus_fmt.h (ctrlstatus_format_time_ms): renders responder_time_ms as
local YYYY-MM-DD HH:MM:SS,mmm, byte-equivalent to the controller log prefix
(tlog/my_tlog_format) so a captured ctrlstatus lines up against ctrl-N.log.
cmd_ctrlstatus prints a responderTime line; TZ-pinned exact-string unit test.
Req 11.2, 11.3. ctrlstatus group passes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-20 09:28:49 +00:00
3799bd6af4 feat(controller-status): off-primary learned committed + responder timestamp in ctrlstatus_fill (task 6.2)
Role-branch the committed sourcing: PRIMARY reports its own authoritative
persist.committed_lsn / committed_config_tag; off-primary reports the LEARNED
ha->last_known_committed_lsn / last_known_committed_config_tag instead of the
primary-only raw markers that read 0 on a standby (fixes the applied>committed
inversion). Stamp responder_time_ms from CLOCK_REALTIME for log correlation.
Read-only (Req 10.1-10.4, 11.1). 4 new TDD tests; controller suite 110/0/0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-20 09:17:07 +00:00
65ddafeb66 feat(controller-status): add responderTimeMs @13 to CtrlStatus schema + typemap (task 6.1)
Append-only UInt64 epoch-millis field for the responder snapshot timestamp
(Req 11.1, wire-format setup). Hand-authored xce_ctrlstatus_t typemap gains a
matching responder_time_ms member; codec round-trips it. Controller suite 1/1 OK.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-20 08:56:51 +00:00
cdc6e2377a spec(controller-status): mark implemented (E2E pass; primaryName bug fixed); log upstream dead-first-master failover finding 2026-06-20 06:04:41 +00:00
6747304184 feat(controller-status): wait-primary uses authoritative ctrlstatus_primary probe (task 4.2) 2026-06-20 05:46:24 +00:00
9e87d2c92e feat(controller-status): ctrlstatus client command (task 4.1) 2026-06-20 05:40:15 +00:00
03813bf560 feat(controller-status): xcecli_ctrlstatus_primary authoritative reach-the-primary read (task 3.2) 2026-06-20 05:32:18 +00:00
691cd9199e feat(controller-status): libxce xcecli_ctrlstatus any-node read + decode + free (task 3.1) 2026-06-20 05:18:04 +00:00
e56409cbd4 feat(controller-status): allow ctrlstatus through ha_dispatch_gate in all roles incl no-quorum (task 2.2) 2026-06-20 05:04:03 +00:00
a6e03d87ab feat(controller-status): handle_ctrlstatus read-only state snapshot + dispatch + configChecksum as Text hex (task 2.1) 2026-06-20 04:53:27 +00:00
97a501e96e feat(controller-status): add ctrlstatus opcode + CtrlStatus/CtrlPeer schema + typemap typedefs (task 1.1) 2026-06-20 04:32:42 +00:00
4b75bfa4db spec(controller-status): tasks — schema, handler, dispatch-gate, libxce any-node+authoritative, ctrlstatus cmd, wait-primary swap, E2E (sanity-review PASS) 2026-06-20 04:22:28 +00:00
ce8c333be0 spec(controller-status): design review fixes — authoritative-from-primary wait-primary (redirect), loaded+committed config fields, mermaid render fixes 2026-06-20 04:11:52 +00:00
423b560353 spec(controller-status): design — new ctrlstatus opcode, dispatch-gate all-roles, version+SHA256 checksum, wait-primary probe swap 2026-06-20 03:51:27 +00:00
2afaf1cbe5 spec(controller-status): gap analysis — new lean opcode (not extend clusterconf); config CRC already exists; dispatch-gate allow-list is the critical gap 2026-06-20 03:44:24 +00:00
5b14bcf832 spec(controller-status): requirements (rename from controller-lsid)
EARS requirements for a controller status query (XCE lsid analog): primary identity
(unknown when unelected), answered-by node+role (distinct from cluster view),
configured peers+count, committed config version + content checksum (catch same-version
divergence), term/commit-apply/fenced/cluster-name, failover-aware delivery, test-client
command, and wait-primary switching its probe to it. C tier only.
2026-06-20 03:34:14 +00:00