9.1 KiB
9.1 KiB
Jul 6 → Jul 11 Progress Report
Overview
Five days of HA stability hardening, chaotic soak debugging, spec work, and controller invariant audit. raft-apply-decoupling completed. Two controller invariant bugs fixed (1433/1434) bringing HA soaks to clean PASS. Three specs advanced (scheduler-framework design, admin-config-tool finalized, ha-elect-ctx-lifecycle initiated).
Jul 6-7: raft-apply-decoupling — COMPLETE
Final tasks closed out the raft-apply-decoupling spec:
| Task | Detail | Commit |
|---|---|---|
| 7.2 | kill-mid-fact-stream soak, fix bug-1416 boot-time TOCTOU | a5e29a6 |
| 7.3 | Stale apply-before-commit assertions updated | 16b890b |
| 7.4 | O(n²) apply-pump WAL rescan → O(log n) throughput benchmarked | 40196be |
Bugs fixed during close-out:
- bug-1416: persist recover TOCTOU — boot-time applied_lsn > committed_lsn gap
- bug-1419 (×3 layers): resubmit-dedup gap — layer 1 (
fe4352f), layer 2 (0eb1054), layer 3 (f28ce57) - bug-1420: xce_test CLI harness jobid collision within same wall-clock second
Spec marked complete: 903b284.
Jul 7-9: HA Election Liveness — Cascade of Bugs
Six bugs discovered and fixed in rapid succession, all around election/term/vote correctness:
| Bug | Manifestation | Root cause | Fix |
|---|---|---|---|
| 1417 | Election stall, votes silently dropped | Vote reply index mapping: compressed local index → absolute peer index | ha_repl_ack_abs_peer_idx resolver (d512275) |
| 1418 | Term-desync + duplicate-node guard | Two factors: (1) ha->term/raft.term desync on standby, (2) start_node re-launch on already-running process | Multi-pass fix (9b173c9, 48b4dba) |
| 1421 | AE-reject blindspot, term desync cycle | raft_on_higher_term_observed ignored stale-term AE when term was already advanced | Close blindspot (327dbb9) |
| 1422 | verify_no_loss false-positive after election | Race: job presence check ran before post-election convergence completed | Retry with backoff (016c598) |
| 1423 | vote-req failures silently dropped (production path) | Production vote-reply dispatch never called; only PreVote path ran | Route prod path through dispatch |
| 1424 | PreVote twin of 1423 | PreVote connect/monitor failure branches never prod | Same pattern, PreVote path |
Validation: 40-round sequential PASS (111 jobs, 0 loss), 30-round chaotic PASS (93 jobs, 0 loss).
Jul 9-10: Chaotic Soak Debugging — Major Effort
~6 chaotic soak rounds chasing persistent committed_job_loss=5-8:
Round 1: fdatasync investigation
- Initial hypothesis: bug-924 (overlayfs fdatasync EIO)
- Disproved: errors were
errno=22 EINVALon ext4, not overlayfs - Stale attribution in buglog corrected
Round 2: Disk cleanup
- 11,062 stale
xce_waldumptmp files accumulated across rounds - Freed 156GB (86% → 68% disk), eliminated 2 false-positive losses
Round 3: persist_apply_pump_run_sync race
- CAS-coalescing pattern caused apply-pump to bail out before catching up
- Fixed: replaced with guaranteed loop-until-convergence (
last_applied_lsn >= committed_lsn) - Build: 920 tests PASS, 5/5 meson suites OK
Round 4: PreVote UAF (bug-1430)
- ASan caught heap-use-after-free in
ha_prevote_round_timeout_fn - Fixed: unconditional CAS before free in cancel path
- 19 PreVote unit tests pass, 0 regressions
Round 5: Force-flush (bug-1431)
- Added force-flush for commit durability path
- Reduced loss from 8→5 but didn't eliminate
Round 6: Residual loss (bug-1432)
- Remaining 8 losses identified as pre-convergence false positives
- New primary starts with
committed_lsn=0during truncation, verify runs before convergence - Not real data loss — timing artifact in soak harness
Jul 10: Controller Invariant Audit
Created xce-audit-invariants skill. Ran full audit across 4 dimensions:
| Dimension | Finding | Severity |
|---|---|---|
| Raft TLA+ checklist | bug-1434: committed_lsn CAS race in persist_promotion_advance_committed |
CRITICAL |
| async-I/O thread role | bug-1433: fdatasync blocks on delaymgr thread (blocking-I/O violation) |
CRITICAL |
| dispatch gate | Clean — all gates verified | — |
| join/startup race | Clean — no exploit path | — |
bug-1433: async-thread fdatasync block
flusher_flush_oncecalledfdatasync()directly, blocking the delaymgr worker- Violation of async-I/O invariant (blocking ops must go through slowlane or dedicated thread)
- Fix: moved fdatasync to slowlane
bug-1434: committed_lsn CAS race
persist_promotion_advance_committedused CAS without lock, racing with concurrent writers- Violation of Raft TLA+ invariant (committed_lsn must be monotonic and consistent)
- Fix: replaced CAS with atomic store under existing lock
Validation: 20-round chaotic + 20-round sequential both PASS (63 jobs, 0 loss, 0 divergence, all 3 nodes led).
Committed: 40f510f.
Jul 10-11: Spec Work
scheduler-framework (design discussion)
Four-level picker architecture designed:
| Level | Picker | Plugin slot | Default (v1) |
|---|---|---|---|
| 0 | Project picker | sched->project_pick_plugin |
builtin_flat |
| 1 | Inter-queue picker | sched->queue_pick_plugin |
builtin_strict_priority |
| 2 | Intra-queue picker | queue→project→sched chain |
builtin_fcfs |
| 3 | Preemption policy | queue->preempt_policy |
lowest_priority_first |
Key design decisions:
- Unified picker loop: always
project → queue → job, identical code for v1+v2 - Queue = scheduling mechanism (HOW), Project = resource ownership (WHO)
- Preemption: SIGSTP-based v1, LSF-style cross-queue victim selection
- New
XCE_DECISION_PREEMPTdecision type flowing scheduler→controller→worker requirements.mdrewritten: 6→9 requirements, architecture diagrams, data structure changesbrief.mdupdated: four-level architecture, queue vs project roles, preemption flow
admin-config-tool
Spec finalized: 10 EARS requirements, 3-skill harness (xce-build/deploy-ha/test-ha), Nix 7 packages, controller upgrade pipeline verified. Roadmap: rollback + entity-snapshot phases deferred.
ha-elect-ctx-lifecycle
Requirements generated (58e047f). New spec addressing election context lifecycle management.
Build & Test Infrastructure
| Change | Detail |
|---|---|
| xce-build/xce-test-ha/xce-deploy-ha | Subagent dispatch fixed (5e0579d) |
| xce-test-ha skill | Updated docs: 5-50 round range (was stale 5-10) |
| xce-audit-invariants skill | New: audit+fix+regression loop workflow |
Bug Tally: Jul 6-11
Fixed (17)
1416, 1417, 1418, 1419(×3 layers), 1420, 1421, 1422, 1423, 1424, 1426, 1427, 1428, 1430, 1431, 1433, 1434
Known residual
- bug-1432: Pre-convergence false-positive committed_job_loss in soak harness (not real data loss)
- bug-1425: Intermittent client hang after failover in synchronous jobsub path (~2/9 attempts)
Commits (Jul 7-11)
40f510f fix(controller): bug-1433 async-thread fdatasync block + bug-1434 committed_lsn CAS race
58e047f spec(ha-elect-ctx-lifecycle): generate requirements
ea0ac81 save wolf state and document
c689351 chore(wolf): restore buglog.json, log bug-1421/bug-1422, memory trail
5e0579d feat(skills): fix xce-build/xce-test-ha/xce-deploy-ha subagent dispatch
016c598 fix(xce-test-ha): bug-1422 verify_no_loss false-positive after election
327dbb9 fix(ha): bug-1421 close term-desync AE-reject blindspot in HA election
0941cf9 fix(ha): close two independent chaotic-failover bugs — term-desync liveness stall and bug-847 WAL batch payload loss
11244dd fix(raft-apply-decoupling): bug-1419 real fix — replay tally, not dedup-guard
f28ce57 fix(raft-apply-decoupling): bug-1419 layer 3 — close cross-primary client-retry dedup gap
1b6041a fix(xce-test-ha): bug-1420 harness jobid collision + bug-1419 regression coverage
0eb1054 fix(raft-apply-decoupling): bug-1419 layer 2 — checkpoint_lsn mislabeled from WAL tip instead of applied cursor
fe4352f fix(raft-apply-decoupling): bug-1419 layer 1 — close resubmit-dedup gap left by apply-after-commit deferral
9b173c9 fix(xce-test-ha): guard start_node against duplicate launches (bug-1418)
d512275 fix(ha): correct real-vote reply index mapping for middle-priority nodes (bug-1417)
903b284 chore(raft-apply-decoupling): mark task group 7 complete, close out spec
40196be perf(raft-apply-decoupling): fix O(n^2) apply-pump WAL rescan, measure throughput (task 7.4)
Current State
| Area | Status |
|---|---|
| HA sequential soak | 40-round PASS (zero loss) |
| HA chaotic soak | 20-round PASS after bug-1433/1434 fixes |
| raft-apply-decoupling | COMPLETE (all 7 task groups) |
| ha-term-safety | Phases 1-5 complete, tasks 1-5 done |
| ha-elect-ctx-lifecycle | Requirements generated, pending design |
| ha-commit-durability | Phases 6.1-6.4 ext4-validated |
| scheduler-framework | Requirements updated (design discussion), pending approval |
| admin-config-tool | Spec finalized, pending implementation |
| Overlayfs bug-924 | fdatasync wedge on overlayfs — known, workaround: use ~/tmp (ext4) |