Rongsong Shen
a5e29a6ffa
Adds --factstream HA soak mode driving real jstart/jfinish/jfail worker-fact traffic against a live 3-node cluster, killing the primary mid-stream and asserting zero committed-state divergence and zero phantom fact reactions. Building this scenario surfaced bug-1416: persist_recover()'s boot-time apply-correctness verification can race the live apply pump and the standby live-replication apply path, both of which mutate the same last_applied_lsn/ job-table state the verification re-reads live -- producing a false APPLY_DIVERGENT that permanently fences an otherwise-healthy node. Fixed with a new persist_t.recovery_complete gate that defers only the in-memory apply (never the durability write or replication ACK) while a recovery pass is in flight, closed from the earliest safe point in myapp_run() so an election- retry candidate can't apply live traffic before persist_recover() ever runs. Round-2 adversarial review independently verified the ACK/durability paths stay ungated, reproduced the myapp.c fix as load-bearing via revert-and- reproduce, and confirmed real-divergence detection still fires. Live 3-node factstream soak: 2/2 rounds PASS, zero APPLY_DIVERGENT across 5 recover calls. controller_unit_test: zero new failures across 23 groups.
3.4 KiB
3.4 KiB
Work Summary: June 30 – July 4, 2026
Monday June 30 — Raft Election + Bug Fixes
- Phase 5 RED shipped with heartbeat-freshness guard
- 3 concurrency bugs fixed: bug-1282 (ABBA deadlock netio.c), bug-1286 (NULL ep→owner), bug-1298 (endpoint UAF refcount)
- Tasks 5.5–5.8 complete: election timeout resonance fix (per-campaign jitter [T,2T], bounded terms), ASAN soak 6/6 rounds green (182 elections, zero runaway)
- Task 5.9 backfill identified blocker for 7.2 acceptance soak
Tuesday July 1 — Spec Completion + Gap Audit
- Raft_tla: 12/13 tasks done (task 2 checkpoint refined via RED-reject-remediate, tasks 5-6 restore pipeline, task 7 ASAN regress shipped)
- Ha-entity-snapshots: 13/13 complete (checkpoint write, rotation guard, restore-before-advance)
- Gap audit: 3 blockers found blocking raft-ha-core-safety → raft-drive-async 7.2 acceptance gate (2 unscaffolded specs, 1 design-unapproved)
Wednesday July 2 — Soak Validation + More Bug Fixes
- Raft-ha-core-safety phase 7.2 soak green; raft-drive-async task 3.3 unblocked
- 3 more bugs fixed: bug-1386 (silent-send), bug-1389 (65KB chunk corruption), bug-1381 (endpoint UAF)
- Diagnostic tools added: snapverify/snapdiff to xce_test
- Ha-commit-durability phases 6.1–6.4 ext4-validated (5/5 rolling+cold-restart, zero committed-job loss)
- Enabled
g_drive_async_configpush_offload
Thursday July 3 — Raft-Apply-Decoupling Tasks 6.1–6.3
- Task 6.1 (ordered apply pump): committed; discovered bug-1409 (apply_pump_drain unbounded busy-loop, missing return-code check)
- Task 6.2 (fire continuation post-apply): rejected by adversarial review for bug-1415 (multi-vcpu race: apply-pump orphans pending-apply registrations → hang); healed at
persist_register_pending_apply()with self-heal check; committed - Task 6.3 (apply/notify timing integration): approved, committed
Friday July 4 — Tasks 6.4 + 7.1 + Bug Cleanup
- Task 6.4 (quorum-loss catch-up + 4-vcpu burst tests): committed (
2de410c), 30/30 consecutive runs pass - Task 7.1 (conformance harness extension): apply-after-commit + idempotent-reapply assertions in raft_sim.c; RED phase verified bidirectionally; 287/287 tests pass; adversarial review launched
- Bug triage: bug-1400 (submit.start_info leak), bug-1401 (xce_free_ctrlstatus macro freed inner fields not struct), bug-1396 (harness instrumentation)
- buglog pruned: 1406→721 entries (659 junk removed)
- ha-consistency-invariants spec closed
Key Metrics
| Metric | Count |
|---|---|
| Commits | 7+ (across raft-apply-decoupling chain) |
| Bugs fixed | 9 (1282, 1286, 1298, 1381, 1386, 1389, 1400, 1401, 1415) |
| Bugs found+logged | 2 (1409, 1415) |
| Specs closed | 1 (ha-consistency-invariants) |
| Tests passing | 287/287 (xcelib_test) |
| ASAN soaks | 6-round election + 30-round apply-pump, all green |
Insights
- Adversarial review caught bug-1415: a self-disclosed race condition the implementer flagged during task 6.2 — the review process worked as designed (RED→reject→remediate→GREEN)
- buglog hygiene matters: 659 junk entries (~47% of total) were removed — stale/duplicate observations degrade memory search quality and waste context tokens
- The task chain dependency is tight: 6.1→6.2→6.3→6.4 each depend on the prior's API surface (e.g., 6.2's
persist_take_pending_apply()is 6.2's hand-off point to 6.3)