Commit graph

16 commits

Author SHA1 Message Date
40f510fb14 fix(controller): bug-1433 async-thread fdatasync block + bug-1434 committed_lsn CAS race
bug-1433: raft_cb_become_leader runs synchronously off peer_ctrl_dispatch_to_pool,
which hardcodes VTASK_LIVENESS for every inbound peer-control packet including
vote replies. The bug-1431 round-2 fix (flusher_flush_once force-flush in
persist_promotion_advance_committed) was reachable from that never-blocks task
class -- same bug family as bug-1198. Removed the force-flush; round-3
(durable-capped only) plus the already-shipped bug-1432 immediate-first-heartbeat
fix close the visibility-latency gap non-blockingly. TRACKED RESIDUAL:
persist_apply_pump_run_sync is still reachable on the same VTASK_LIVENESS path
and does bounded pread() -- lower risk (page-cache-hot) but strictly still
blocking I/O; not safe to remove outright (reopens bug-1043-family job loss),
needs its own follow-up (reclassify peer_ctrl_classify's winning-vote-reply case
or dispatch promotion tail onto a fresh VTASK_LIVE_BLOCK vtask).

bug-1434: persist_promotion_advance_committed and ha_standby_advance_committed_and_apply
raced on persist_t.committed_lsn via an unlocked load-then-conditional-store during
the promotion window (before ha->role flips to CTRL_PRIMARY) -- a losing racer could
regress committed_lsn. Added persist_committed_lsn_advance_atomic (CAS retry loop,
lock-free) and routed both call sites through it.

Validated: 19/19 unit test suites, 6-round chaotic soak (0 loss/divergence) + 6-round
sequential+leadership-coverage soak, both PASS. Also folds in raft-apply-decoupling
and raft-ha-core-safety validate-impl phase bumps to "implemented" and clears
config-upgrade-snapshot's now-satisfied entity-bearing-snapshots block.
2026-07-11 12:16:28 +00:00
20e59d0b42 checkpoint: bug-851 root cause + raft TLA+ conformance spec family
Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
2026-06-21 14:02:15 +00:00
eda1f4c9d3 fix(xce-test-e2e-primitives): shell-style -e/+e errexit with mode-aware default — interactive REPL no longer exits on a bad command 2026-06-20 06:16:43 +00:00
d82cc78284 test(xce-test-e2e-primitives): harden durability soak — separate liveness from durability, no false bug-661 cascades
Conflation fix: gate every kill + durability assertion on cluster health (3 nodes
up + serving + rejoin settle); assert_no_loss probes serveability first and reports
INCONCLUSIVE (not DATA LOSS) when unserveable; confirm-settle re-check distinguishes
catch-up lag from persistent loss; post-kill election SLA configurable (60s default).
DATA LOSS now reported ONLY for a job persistently absent from a SERVING cluster.
2026-06-20 01:27:38 +00:00
156ddfb6bb spec(xce-test-e2e-primitives): mark implemented — 7.2 reqs 8.2/8.3 met live; cold-restart-all gap routed to ha-commit-durability 2026-06-20 00:42:44 +00:00
c70fb52f03 docs(xce-test-e2e-primitives): record 7.2 live-soak findings (bug-786 strengthened; cold-restart-all durability gap routed upstream) 2026-06-19 17:02:31 +00:00
6315368b01 test(xce-test-e2e-primitives): convert durability soak to exit-code primitives (task 7.1) 2026-06-19 16:41:30 +00:00
9466b71691 feat(xce-test-e2e-primitives): wait-primary [timeout_ms] (task 6.1) 2026-06-19 16:29:26 +00:00
f3334c0fb6 feat(xce-test-e2e-primitives): expect-job <jobid> assertion (task 5.3) 2026-06-19 16:22:22 +00:00
b0734c87c9 feat(xce-test-e2e-primitives): expect-jobs N assertion (task 5.2) 2026-06-19 16:16:43 +00:00
171c88fcec feat(xce-test-e2e-primitives): joblist_query helper + $jobcount capture (task 5.1) 2026-06-19 16:10:44 +00:00
fae0bc0ff4 fix(xce-test-e2e-primitives): jobsub clean-ack gating + $jobid capture, closes bug-786 (task 4.1) 2026-06-19 16:04:13 +00:00
5331832565 feat(xce-test-e2e-primitives): auto-var substitution $?/$jobid/$jobcount (task 3.1) 2026-06-19 15:54:34 +00:00
0584a6229a feat(xce-test-e2e-primitives): !cmd shell-escape via system(), feeds $? (task 2.1) 2026-06-19 15:45:09 +00:00
7ad63e7403 feat(xce-test-e2e-primitives): add echo command (task 1.2) 2026-06-19 15:38:38 +00:00
481e197928 feat(xce-test-e2e-primitives): session verdict + exit code + abort/continue + comment handling (task 1.1) 2026-06-19 15:32:11 +00:00