Rongsong Shen
a5e29a6ffa
Adds --factstream HA soak mode driving real jstart/jfinish/jfail worker-fact traffic against a live 3-node cluster, killing the primary mid-stream and asserting zero committed-state divergence and zero phantom fact reactions. Building this scenario surfaced bug-1416: persist_recover()'s boot-time apply-correctness verification can race the live apply pump and the standby live-replication apply path, both of which mutate the same last_applied_lsn/ job-table state the verification re-reads live -- producing a false APPLY_DIVERGENT that permanently fences an otherwise-healthy node. Fixed with a new persist_t.recovery_complete gate that defers only the in-memory apply (never the durability write or replication ACK) while a recovery pass is in flight, closed from the earliest safe point in myapp_run() so an election- retry candidate can't apply live traffic before persist_recover() ever runs. Round-2 adversarial review independently verified the ACK/durability paths stay ungated, reproduced the myapp.c fix as load-bearing via revert-and- reproduce, and confirmed real-divergence detection still fires. Live 3-node factstream soak: 2/2 rounds PASS, zero APPLY_DIVERGENT across 5 recover calls. controller_unit_test: zero new failures across 23 groups.
48 lines
3.4 KiB
Markdown
48 lines
3.4 KiB
Markdown
# Work Summary: June 30 – July 4, 2026
|
||
|
||
## Monday June 30 — Raft Election + Bug Fixes
|
||
- **Phase 5 RED** shipped with heartbeat-freshness guard
|
||
- **3 concurrency bugs fixed**: bug-1282 (ABBA deadlock netio.c), bug-1286 (NULL ep→owner), bug-1298 (endpoint UAF refcount)
|
||
- **Tasks 5.5–5.8 complete**: election timeout resonance fix (per-campaign jitter [T,2T], bounded terms), ASAN soak 6/6 rounds green (182 elections, zero runaway)
|
||
- Task 5.9 backfill identified blocker for 7.2 acceptance soak
|
||
|
||
## Tuesday July 1 — Spec Completion + Gap Audit
|
||
- **Raft_tla**: 12/13 tasks done (task 2 checkpoint refined via RED-reject-remediate, tasks 5-6 restore pipeline, task 7 ASAN regress shipped)
|
||
- **Ha-entity-snapshots**: 13/13 complete (checkpoint write, rotation guard, restore-before-advance)
|
||
- **Gap audit**: 3 blockers found blocking raft-ha-core-safety → raft-drive-async 7.2 acceptance gate (2 unscaffolded specs, 1 design-unapproved)
|
||
|
||
## Wednesday July 2 — Soak Validation + More Bug Fixes
|
||
- **Raft-ha-core-safety phase 7.2** soak green; raft-drive-async task 3.3 unblocked
|
||
- **3 more bugs fixed**: bug-1386 (silent-send), bug-1389 (65KB chunk corruption), bug-1381 (endpoint UAF)
|
||
- **Diagnostic tools** added: snapverify/snapdiff to xce_test
|
||
- **Ha-commit-durability phases 6.1–6.4** ext4-validated (5/5 rolling+cold-restart, zero committed-job loss)
|
||
- Enabled `g_drive_async_configpush_offload`
|
||
|
||
## Thursday July 3 — Raft-Apply-Decoupling Tasks 6.1–6.3
|
||
- **Task 6.1** (ordered apply pump): committed; discovered **bug-1409** (apply_pump_drain unbounded busy-loop, missing return-code check)
|
||
- **Task 6.2** (fire continuation post-apply): rejected by adversarial review for **bug-1415** (multi-vcpu race: apply-pump orphans pending-apply registrations → hang); healed at `persist_register_pending_apply()` with self-heal check; committed
|
||
- **Task 6.3** (apply/notify timing integration): approved, committed
|
||
|
||
## Friday July 4 — Tasks 6.4 + 7.1 + Bug Cleanup
|
||
- **Task 6.4** (quorum-loss catch-up + 4-vcpu burst tests): committed (2de410c), 30/30 consecutive runs pass
|
||
- **Task 7.1** (conformance harness extension): apply-after-commit + idempotent-reapply assertions in raft_sim.c; RED phase verified bidirectionally; 287/287 tests pass; adversarial review launched
|
||
- **Bug triage**: bug-1400 (submit.start_info leak), bug-1401 (xce_free_ctrlstatus macro freed inner fields not struct), bug-1396 (harness instrumentation)
|
||
- **buglog pruned**: 1406→721 entries (659 junk removed)
|
||
- **ha-consistency-invariants** spec closed
|
||
|
||
## Key Metrics
|
||
|
||
| Metric | Count |
|
||
|---|---|
|
||
| Commits | 7+ (across raft-apply-decoupling chain) |
|
||
| Bugs fixed | 9 (1282, 1286, 1298, 1381, 1386, 1389, 1400, 1401, 1415) |
|
||
| Bugs found+logged | 2 (1409, 1415) |
|
||
| Specs closed | 1 (ha-consistency-invariants) |
|
||
| Tests passing | 287/287 (xcelib_test) |
|
||
| ASAN soaks | 6-round election + 30-round apply-pump, all green |
|
||
|
||
## Insights
|
||
|
||
- **Adversarial review caught bug-1415**: a self-disclosed race condition the implementer flagged during task 6.2 — the review process worked as designed (RED→reject→remediate→GREEN)
|
||
- **buglog hygiene matters**: 659 junk entries (~47% of total) were removed — stale/duplicate observations degrade memory search quality and waste context tokens
|
||
- **The task chain dependency is tight**: 6.1→6.2→6.3→6.4 each depend on the prior's API surface (e.g., 6.2's `persist_take_pending_apply()` is 6.2's hand-off point to 6.3)
|