Commit graph

20 commits

Author SHA1 Message Date
295769d8ae add wolf state 2026-06-25 14:30:46 +08:00
61e1391cd0 fix bug-1060 bug-1067 2026-06-24 17:05:43 +08:00
0eafed3999 save wolf status 2026-06-23 07:08:43 +00:00
20e59d0b42 checkpoint: bug-851 root cause + raft TLA+ conformance spec family
Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
2026-06-21 14:02:15 +00:00
16f4ba67c7 docs(libxce-failover): log bug-609 (known-dead EINVAL) + record 6.2 live-E2E partial status 2026-06-17 04:46:45 +00:00
d11c6e1d89 feat(libxce-failover): reusable failover test harness — fake masters, status taxonomy, concurrency runner (task 1.2) 2026-06-17 02:32:19 +00:00
387ccfa8dc feat(libxce-failover): foundation — per-ctx failover state, redirect-hint fields, remove orphaned retry primitive (task 1.1) 2026-06-17 00:05:41 +00:00
c45f80ab2b commit missed 2026-06-16 15:42:43 +08:00
732e836868 test(ha-commit-barrier-rollout): fact-retention-across-quorum-loss proof (task 4.3)
Green integration proof: 4 facts driven during total quorum loss append
RETAIN with rc 0 in <1000ms (vs 5000ms timeout — wrongly-applied WAIT
tripwire), pending with retained-depth=4 logged, present exactly once
on disk (real WAL header scan); reconnect + one covering ACK commits
the suffix with current_lsn unchanged (no redelivery). Completes the
bug-396 fix evidence (umbrella Req 2.8). Census 495/495.
2026-06-12 15:29:47 +00:00
7103fa643d test(ha-commit-barrier-rollout): receipt-ACK verdict pins (task 3.3)
4 green pins: handle_jobdispatch replies receipt-only with zero WAL
appends through a real 2-alloc dispatch cycle (notify outcomes intact,
reply-precedes-processing); set/clr_preason and do_update_op rusage
stay receipt-style with RETAIN-pending fact appends ACK-committed.
Umbrella Reply-Ordering Contract cited per pin. DISCOVERY bug-451
(reviewer-verified): RUSAGE hook va_list type confusion — 4-byte OOB
read + garbage state in WAL_JOB_RUSAGE payloads; fix deferred to 7.1.
Census 488; zero failures.
2026-06-12 11:16:51 +00:00
e1c3c173f4 test(ha-commit-barrier-rollout): RED commit-first-then-reply tests per opcode (task 3.1)
New test_commit_barrier_reply.c drives do_jobcontrol_op /
do_queuecontrol_op / do_hostcontrol_op / handle_worker_join directly
(pthread harness, ep->outq reply capture, wired engine n_peers=3).
12 cases: 8 RED (reply-before-commit + bug-447 pins), 4 validation
pins green. DISCOVERY bug-447 (reviewer-confirmed): key_find found-rc
leak means queue/host control never append WAL in production — 3.2's
rc normalization is the fix point. Census 484 = 472 + 12; zero
baseline regressions.
2026-06-12 10:00:38 +00:00
fec9519960 test(ha-commit-barrier-rollout): RED tests for classification totality (task 1.1)
New test_commit_barrier_rollout.c: per-type rows matching the VERIFIED
design table (STOP/RESUME/UPDATE/FAIL RETAIN per supersession), 13-value
totality sweep, fork/SIGABRT unknown-type fail-fast, fact-hook retention
contract against real persist+engine. wal_commit_mode_t + wal_classify
declared in persist.h; inert sentinel stub. 8/8 RED; suite baseline
zero preserved. Finding: RETAIN is registration-free (commit.c:233-239)
— observable contract documented in Implementation Notes.
2026-06-12 04:13:22 +00:00
d91f446e9e test(ha-commit-engine): RED tests for WAIT mode, admission cap, own-term gate (task 2.1)
9 new cases (8-16): block-until-majority, deadline timeout, standalone
fast path, zero-ACK regression, n_vcpus-relative cap with floor, ACK
progress at saturation, gate as blocking predicate (NOT_READY is
admission-cap-only per corrected contract). Main-thread-only assertions
after joins; 16 tests / 9 RED failures; baseline unchanged.
2026-06-11 14:56:31 +00:00
eae4767d7d feat(ha-commit-engine): per-LSN majority commit engine core (task 1.2)
commit_on_ack: active-standby (non-catch-up, connected) watermark scan,
descending sort, candidate arr[n_peers/2-1], clamped to appended log,
monotonic single-site committed_lsn store, condvar broadcast, never
blocks beyond engine mutex. RETAIN returns immediately. Self-quorum for
n_peers<=1; zero-fill makes zero-standby commit impossible. 7/7 task-1.1
tests green unmodified; full suite at known 3-failure baseline.
2026-06-11 14:37:01 +00:00
d5a0f9c64d test(ha-commit-engine): RED tests for per-LSN majority evaluation (task 1.1)
commit.h umbrella-verbatim interface + inert stub commit.c (test target
only); 7 Unity tests covering per-LSN vs acked_lsn>0 counting, majority
median, monotonicity, append clamp, lagging standby, RETAIN immediacy.
All 7 fail RED as intended. Documented 3 pre-existing masked suite
failures (bug-411) and the per-group exit-code masking root cause.
2026-06-11 14:17:09 +00:00
06135954ff chore: baseline snapshot — prior-session work + ha-consistency-invariants spec family
Pre-existing uncommitted work: ha-repl-hardening payload data plane (7.x),
heartbeat liveness fix (Req 1.3), connection-reuse ASAN fixes, doc cleanup.
New: ha-consistency-invariants umbrella (requirements/design/research) and
its four implementation sub-specs (specs generated via batch, cross-spec
review applied), steering roadmap + product.md Raft+CP decision.
2026-06-11 13:21:50 +00:00
18e351ee1b feat(ha-wal-replication): repl_broadcast TCP send (task 1.1)
Replace no-send stub with real WAL entry transmission via
xce_wal_batch_req_raw. For each non-catch-up registered standby,
serialize the entry header+payload into a wal_catchup_entry_t
and enqueue. Keep the MAX_PENDING_ACKS eviction guard.

5 new tests: no-standbys, one-standby-send, skip-catch-up,
evict-overloaded, send-failure-continues. Controller suite green.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-06-10 08:11:05 +00:00
cdd446fc17 docs(specs): gap-analysis validation pass across all 9 HA-family specs
Appended "Gap Analysis — Validation Pass (2026-06-09)" to each spec's
research.md with requirement-to-asset maps, status tags, and effort/risk.

Key outcomes:
- ha-repl-hardening: REOPENED with task series 7.1-7.7 — replication
  payload data plane is missing (snapshot chunks and WAL catch-up ship
  header-only; standbys converge on LSN without holding entity state)
- worker-join-refactor, resval-id-decoupling: marked implemented
  (all criteria re-verified, full test suite green)
- buglog: 335/336 (scheduler dequeue, open), 337/338/339 (fixed in
  previous commit); cerebrum Decision Log corrected (term-only vote
  deny was recorded as intentional but was a regression)
2026-06-10 01:26:33 +00:00
fa66d1ba9c finalize configuration replication 2026-06-09 16:40:13 +08:00
e31673dd88 chore: checkpoint all prior work for clean state
Captures all uncommitted work from ha-config-bootstrap,
ha-heartbeat-liveness, and associated spec/tooling updates:

- src: ha.c, ha.h, cluster.c, config_repl.c, myapp.c, raft.c —
  heartbeat liveness, RDHUP integration, committing phase fixes
- tests: test_ha_role.c, test_config_repl.c, test_ha_rdhup_integration.c,
  test_p1_integration.c, test_p3_hardening.c, scheduler test
- specs: election-iomgr-refactor, ha-heartbeat-liveness,
  ha-peer-connection-reuse, raft-library (new); ha-config-bootstrap
  and election-iomgr-refactor task/research updates
- tooling: .wolf/ state, ha/data cluster fixtures, raft/raft.tla,
  src/build symlink, xce-build skill update
2026-06-04 13:04:16 +00:00