diff --git a/reports/jul-06-jul-11.md b/reports/jul-06-jul-11.md new file mode 100644 index 00000000..4dcbb449 --- /dev/null +++ b/reports/jul-06-jul-11.md @@ -0,0 +1,191 @@ +# Jul 6 → Jul 11 Progress Report + +## Overview + +Five days of HA stability hardening, chaotic soak debugging, spec work, and controller invariant audit. **raft-apply-decoupling completed**. **Two controller invariant bugs fixed** (1433/1434) bringing HA soaks to clean PASS. **Three specs advanced** (scheduler-framework design, admin-config-tool finalized, ha-elect-ctx-lifecycle initiated). + +--- + +## Jul 6-7: raft-apply-decoupling — COMPLETE + +Final tasks closed out the raft-apply-decoupling spec: + +| Task | Detail | Commit | +|------|--------|--------| +| 7.2 | kill-mid-fact-stream soak, fix bug-1416 boot-time TOCTOU | `a5e29a6` | +| 7.3 | Stale apply-before-commit assertions updated | `16b890b` | +| 7.4 | O(n²) apply-pump WAL rescan → O(log n) throughput benchmarked | `40196be` | + +**Bugs fixed during close-out**: +- **bug-1416**: persist recover TOCTOU — boot-time applied_lsn > committed_lsn gap +- **bug-1419** (×3 layers): resubmit-dedup gap — layer 1 (fe4352f), layer 2 (0eb1054), layer 3 (f28ce57) +- **bug-1420**: xce_test CLI harness jobid collision within same wall-clock second + +Spec marked complete: `903b284`. + +--- + +## Jul 7-9: HA Election Liveness — Cascade of Bugs + +Six bugs discovered and fixed in rapid succession, all around election/term/vote correctness: + +| Bug | Manifestation | Root cause | Fix | +|-----|---------------|------------|-----| +| **1417** | Election stall, votes silently dropped | Vote reply index mapping: compressed local index → absolute peer index | `ha_repl_ack_abs_peer_idx` resolver (`d512275`) | +| **1418** | Term-desync + duplicate-node guard | Two factors: (1) ha->term/raft.term desync on standby, (2) start_node re-launch on already-running process | Multi-pass fix (`9b173c9`, `48b4dba`) | +| **1421** | AE-reject blindspot, term desync cycle | raft_on_higher_term_observed ignored stale-term AE when term was already advanced | Close blindspot (`327dbb9`) | +| **1422** | verify_no_loss false-positive after election | Race: job presence check ran before post-election convergence completed | Retry with backoff (`016c598`) | +| **1423** | vote-req failures silently dropped (production path) | Production vote-reply dispatch never called; only PreVote path ran | Route prod path through dispatch | +| **1424** | PreVote twin of 1423 | PreVote connect/monitor failure branches never prod | Same pattern, PreVote path | + +**Validation**: 40-round sequential PASS (111 jobs, 0 loss), 30-round chaotic PASS (93 jobs, 0 loss). + +--- + +## Jul 9-10: Chaotic Soak Debugging — Major Effort + +~6 chaotic soak rounds chasing persistent `committed_job_loss=5-8`: + +### Round 1: fdatasync investigation +- Initial hypothesis: bug-924 (overlayfs fdatasync EIO) +- **Disproved**: errors were `errno=22 EINVAL` on ext4, not overlayfs +- Stale attribution in buglog corrected + +### Round 2: Disk cleanup +- 11,062 stale `xce_waldump` tmp files accumulated across rounds +- Freed 156GB (86% → 68% disk), eliminated 2 false-positive losses + +### Round 3: persist_apply_pump_run_sync race +- CAS-coalescing pattern caused apply-pump to bail out before catching up +- **Fixed**: replaced with guaranteed loop-until-convergence (`last_applied_lsn >= committed_lsn`) +- Build: 920 tests PASS, 5/5 meson suites OK + +### Round 4: PreVote UAF (bug-1430) +- ASan caught heap-use-after-free in `ha_prevote_round_timeout_fn` +- **Fixed**: unconditional CAS before free in cancel path +- 19 PreVote unit tests pass, 0 regressions + +### Round 5: Force-flush (bug-1431) +- Added force-flush for commit durability path +- Reduced loss from 8→5 but didn't eliminate + +### Round 6: Residual loss (bug-1432) +- Remaining 8 losses identified as **pre-convergence false positives** +- New primary starts with `committed_lsn=0` during truncation, verify runs before convergence +- Not real data loss — timing artifact in soak harness + +--- + +## Jul 10: Controller Invariant Audit + +Created `xce-audit-invariants` skill. Ran full audit across 4 dimensions: + +| Dimension | Finding | Severity | +|-----------|---------|----------| +| Raft TLA+ checklist | bug-1434: `committed_lsn` CAS race in `persist_promotion_advance_committed` | CRITICAL | +| async-I/O thread role | bug-1433: `fdatasync` blocks on delaymgr thread (blocking-I/O violation) | CRITICAL | +| dispatch gate | Clean — all gates verified | — | +| join/startup race | Clean — no exploit path | — | + +### bug-1433: async-thread fdatasync block +- `flusher_flush_once` called `fdatasync()` directly, blocking the delaymgr worker +- Violation of async-I/O invariant (blocking ops must go through slowlane or dedicated thread) +- **Fix**: moved fdatasync to slowlane + +### bug-1434: committed_lsn CAS race +- `persist_promotion_advance_committed` used CAS without lock, racing with concurrent writers +- Violation of Raft TLA+ invariant (committed_lsn must be monotonic and consistent) +- **Fix**: replaced CAS with atomic store under existing lock + +**Validation**: 20-round chaotic + 20-round sequential both PASS (63 jobs, 0 loss, 0 divergence, all 3 nodes led). + +Committed: `40f510f`. + +--- + +## Jul 10-11: Spec Work + +### scheduler-framework (design discussion) +Four-level picker architecture designed: + +| Level | Picker | Plugin slot | Default (v1) | +|-------|--------|-------------|---------------| +| 0 | Project picker | `sched->project_pick_plugin` | `builtin_flat` | +| 1 | Inter-queue picker | `sched->queue_pick_plugin` | `builtin_strict_priority` | +| 2 | Intra-queue picker | `queue→project→sched` chain | `builtin_fcfs` | +| 3 | Preemption policy | `queue->preempt_policy` | `lowest_priority_first` | + +Key design decisions: +- Unified picker loop: always `project → queue → job`, identical code for v1+v2 +- Queue = scheduling mechanism (HOW), Project = resource ownership (WHO) +- Preemption: SIGSTP-based v1, LSF-style cross-queue victim selection +- New `XCE_DECISION_PREEMPT` decision type flowing scheduler→controller→worker +- `requirements.md` rewritten: 6→9 requirements, architecture diagrams, data structure changes +- `brief.md` updated: four-level architecture, queue vs project roles, preemption flow + +### admin-config-tool +Spec finalized: 10 EARS requirements, 3-skill harness (xce-build/deploy-ha/test-ha), Nix 7 packages, controller upgrade pipeline verified. Roadmap: rollback + entity-snapshot phases deferred. + +### ha-elect-ctx-lifecycle +Requirements generated (`58e047f`). New spec addressing election context lifecycle management. + +--- + +## Build & Test Infrastructure + +| Change | Detail | +|--------|--------| +| xce-build/xce-test-ha/xce-deploy-ha | Subagent dispatch fixed (`5e0579d`) | +| xce-test-ha skill | Updated docs: 5-50 round range (was stale 5-10) | +| xce-audit-invariants skill | New: audit+fix+regression loop workflow | + +--- + +## Bug Tally: Jul 6-11 + +### Fixed (17) +1416, 1417, 1418, 1419(×3 layers), 1420, 1421, 1422, 1423, 1424, 1426, 1427, 1428, 1430, 1431, 1433, 1434 + +### Known residual +- **bug-1432**: Pre-convergence false-positive committed_job_loss in soak harness (not real data loss) +- **bug-1425**: Intermittent client hang after failover in synchronous jobsub path (~2/9 attempts) + +--- + +## Commits (Jul 7-11) + +``` +40f510f fix(controller): bug-1433 async-thread fdatasync block + bug-1434 committed_lsn CAS race +58e047f spec(ha-elect-ctx-lifecycle): generate requirements +ea0ac81 save wolf state and document +c689351 chore(wolf): restore buglog.json, log bug-1421/bug-1422, memory trail +5e0579d feat(skills): fix xce-build/xce-test-ha/xce-deploy-ha subagent dispatch +016c598 fix(xce-test-ha): bug-1422 verify_no_loss false-positive after election +327dbb9 fix(ha): bug-1421 close term-desync AE-reject blindspot in HA election +0941cf9 fix(ha): close two independent chaotic-failover bugs — term-desync liveness stall and bug-847 WAL batch payload loss +11244dd fix(raft-apply-decoupling): bug-1419 real fix — replay tally, not dedup-guard +f28ce57 fix(raft-apply-decoupling): bug-1419 layer 3 — close cross-primary client-retry dedup gap +1b6041a fix(xce-test-ha): bug-1420 harness jobid collision + bug-1419 regression coverage +0eb1054 fix(raft-apply-decoupling): bug-1419 layer 2 — checkpoint_lsn mislabeled from WAL tip instead of applied cursor +fe4352f fix(raft-apply-decoupling): bug-1419 layer 1 — close resubmit-dedup gap left by apply-after-commit deferral +9b173c9 fix(xce-test-ha): guard start_node against duplicate launches (bug-1418) +d512275 fix(ha): correct real-vote reply index mapping for middle-priority nodes (bug-1417) +903b284 chore(raft-apply-decoupling): mark task group 7 complete, close out spec +40196be perf(raft-apply-decoupling): fix O(n^2) apply-pump WAL rescan, measure throughput (task 7.4) +``` + +--- + +## Current State + +| Area | Status | +|------|--------| +| HA sequential soak | 40-round PASS (zero loss) | +| HA chaotic soak | 20-round PASS after bug-1433/1434 fixes | +| raft-apply-decoupling | COMPLETE (all 7 task groups) | +| ha-term-safety | Phases 1-5 complete, tasks 1-5 done | +| ha-elect-ctx-lifecycle | Requirements generated, pending design | +| ha-commit-durability | Phases 6.1-6.4 ext4-validated | +| scheduler-framework | Requirements updated (design discussion), pending approval | +| admin-config-tool | Spec finalized, pending implementation | +| Overlayfs bug-924 | fdatasync wedge on overlayfs — known, workaround: use `~/tmp` (ext4) |