myxce

lei.chen/myxce

Fork 1

Commit graph

Author	SHA1	Message	Date
Rongsong Shen	6bb9d81225	docs(ha-commit-durability): sharpen bug-1306 root cause — xce_run_hooks clobbers commit -ETIMEDOUT 10-round soak waldump+log bisect: the chaotic committed_job_loss is NOT the catch-up/entity-snapshot prefix gap. All 3 WALs gapless/CRC-intact/full payloads; lost job only on ctrl-1@LSN30 term9 vs term-10 NOOP on the majority. The durable-majority gate worked (commit_submit -> -ETIMEDOUT), but xce_run_hooks (hook.c:134-157) returns only the LAST-registered hook's rc, so a later JOB AFTER_EVENT hook (sched/apiserver) returning 0 clobbered persist's -ETIMEDOUT -> handle_jobsubmit replied jobsubmit ok for an uncommitted entry -> term-10 leader legally overwrote it. Fix per raft-apply-decoupling item #1: move the WAL append + commit barrier out of the hook into an explicit handle_jobsubmit-driven raft step whose rc is read directly (down payment on the full apply-after-commit inversion). Entity-snapshot gap is a separate, still-real defect (not this soak's mechanism).	2026-07-01 00:21:23 +00:00
Rongsong Shen	20e59d0b42	checkpoint: bug-851 root cause + raft TLA+ conformance spec family Investigation & fixes (failover committed-job loss): - root-cause bug-851 on an owned 3-node cluster: chaotic failover causes silent committed-prefix divergence (WAL hdr.term always 0; two leaders per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION. - live-validated fixes 848/849/850 (recovery-on-promotion, advertise real applied_lsn, persist-layer contiguity guard); add waldump WAL inspector. Specs (strict raft.tla conformance — library owns the log): - TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6. - raft-log-replication, raft-commit-safety, raft-sim-conformance: extend src/lib/raft to own log/commit/verification via callbacks (specs generated, cross-spec review passed). - ha-term-safety: controller-side adapter + config marker (R1-R7); design done. - config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857). - roadmap + buglog (851/854/857) + .wolf knowledge updated.	2026-06-21 14:02:15 +00:00
Rongsong Shen	7d70820aac	docs(ha-commit-durability): root-cause findings for failover committed-job loss (bug-828) WAL-byte decode proves REAL data loss (not commit-index): committed prefix entries physically absent from live nodes (ctrl-2 WAL starts LSN 16, ctrl-3 LSN 13), present only on the down node. Mechanism: empty-rejoin catch-up streams [watermark..committed] not [1..], no snapshot to backfill the prefix -> permanent gap. Predates libxce-failover. Full findings + open forks + xce_test waldump tooling plan in INVESTIGATION-failover-job-loss.md. Not yet fixed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-20 13:51:40 +00:00

Author

SHA1

Message

Date

Rongsong Shen

6bb9d81225

docs(ha-commit-durability): sharpen bug-1306 root cause — xce_run_hooks clobbers commit -ETIMEDOUT

10-round soak waldump+log bisect: the chaotic committed_job_loss is NOT the
catch-up/entity-snapshot prefix gap. All 3 WALs gapless/CRC-intact/full
payloads; lost job only on ctrl-1@LSN30 term9 vs term-10 NOOP on the majority.
The durable-majority gate worked (commit_submit -> -ETIMEDOUT), but
xce_run_hooks (hook.c:134-157) returns only the LAST-registered hook's rc, so a
later JOB AFTER_EVENT hook (sched/apiserver) returning 0 clobbered persist's
-ETIMEDOUT -> handle_jobsubmit replied jobsubmit ok for an uncommitted entry ->
term-10 leader legally overwrote it. Fix per raft-apply-decoupling item #1:
move the WAL append + commit barrier out of the hook into an explicit
handle_jobsubmit-driven raft step whose rc is read directly (down payment on
the full apply-after-commit inversion). Entity-snapshot gap is a separate,
still-real defect (not this soak's mechanism).

2026-07-01 00:21:23 +00:00

Rongsong Shen

20e59d0b42

checkpoint: bug-851 root cause + raft TLA+ conformance spec family

Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.

2026-06-21 14:02:15 +00:00

Rongsong Shen

7d70820aac

docs(ha-commit-durability): root-cause findings for failover committed-job loss (bug-828)

WAL-byte decode proves REAL data loss (not commit-index): committed prefix
entries physically absent from live nodes (ctrl-2 WAL starts LSN 16, ctrl-3 LSN
13), present only on the down node. Mechanism: empty-rejoin catch-up streams
[watermark..committed] not [1..], no snapshot to backfill the prefix -> permanent
gap. Predates libxce-failover. Full findings + open forks + xce_test waldump
tooling plan in INVESTIGATION-failover-job-loss.md. Not yet fixed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-20 13:51:40 +00:00

3 commits