forked from lei.chen/myxce
191 lines
9.1 KiB
Markdown
191 lines
9.1 KiB
Markdown
# Jul 6 → Jul 11 Progress Report
|
||
|
||
## Overview
|
||
|
||
Five days of HA stability hardening, chaotic soak debugging, spec work, and controller invariant audit. **raft-apply-decoupling completed**. **Two controller invariant bugs fixed** (1433/1434) bringing HA soaks to clean PASS. **Three specs advanced** (scheduler-framework design, admin-config-tool finalized, ha-elect-ctx-lifecycle initiated).
|
||
|
||
---
|
||
|
||
## Jul 6-7: raft-apply-decoupling — COMPLETE
|
||
|
||
Final tasks closed out the raft-apply-decoupling spec:
|
||
|
||
| Task | Detail | Commit |
|
||
|------|--------|--------|
|
||
| 7.2 | kill-mid-fact-stream soak, fix bug-1416 boot-time TOCTOU | `a5e29a6` |
|
||
| 7.3 | Stale apply-before-commit assertions updated | `16b890b` |
|
||
| 7.4 | O(n²) apply-pump WAL rescan → O(log n) throughput benchmarked | `40196be` |
|
||
|
||
**Bugs fixed during close-out**:
|
||
- **bug-1416**: persist recover TOCTOU — boot-time applied_lsn > committed_lsn gap
|
||
- **bug-1419** (×3 layers): resubmit-dedup gap — layer 1 (fe4352f), layer 2 (0eb1054), layer 3 (f28ce57)
|
||
- **bug-1420**: xce_test CLI harness jobid collision within same wall-clock second
|
||
|
||
Spec marked complete: `903b284`.
|
||
|
||
---
|
||
|
||
## Jul 7-9: HA Election Liveness — Cascade of Bugs
|
||
|
||
Six bugs discovered and fixed in rapid succession, all around election/term/vote correctness:
|
||
|
||
| Bug | Manifestation | Root cause | Fix |
|
||
|-----|---------------|------------|-----|
|
||
| **1417** | Election stall, votes silently dropped | Vote reply index mapping: compressed local index → absolute peer index | `ha_repl_ack_abs_peer_idx` resolver (`d512275`) |
|
||
| **1418** | Term-desync + duplicate-node guard | Two factors: (1) ha->term/raft.term desync on standby, (2) start_node re-launch on already-running process | Multi-pass fix (`9b173c9`, `48b4dba`) |
|
||
| **1421** | AE-reject blindspot, term desync cycle | raft_on_higher_term_observed ignored stale-term AE when term was already advanced | Close blindspot (`327dbb9`) |
|
||
| **1422** | verify_no_loss false-positive after election | Race: job presence check ran before post-election convergence completed | Retry with backoff (`016c598`) |
|
||
| **1423** | vote-req failures silently dropped (production path) | Production vote-reply dispatch never called; only PreVote path ran | Route prod path through dispatch |
|
||
| **1424** | PreVote twin of 1423 | PreVote connect/monitor failure branches never prod | Same pattern, PreVote path |
|
||
|
||
**Validation**: 40-round sequential PASS (111 jobs, 0 loss), 30-round chaotic PASS (93 jobs, 0 loss).
|
||
|
||
---
|
||
|
||
## Jul 9-10: Chaotic Soak Debugging — Major Effort
|
||
|
||
~6 chaotic soak rounds chasing persistent `committed_job_loss=5-8`:
|
||
|
||
### Round 1: fdatasync investigation
|
||
- Initial hypothesis: bug-924 (overlayfs fdatasync EIO)
|
||
- **Disproved**: errors were `errno=22 EINVAL` on ext4, not overlayfs
|
||
- Stale attribution in buglog corrected
|
||
|
||
### Round 2: Disk cleanup
|
||
- 11,062 stale `xce_waldump` tmp files accumulated across rounds
|
||
- Freed 156GB (86% → 68% disk), eliminated 2 false-positive losses
|
||
|
||
### Round 3: persist_apply_pump_run_sync race
|
||
- CAS-coalescing pattern caused apply-pump to bail out before catching up
|
||
- **Fixed**: replaced with guaranteed loop-until-convergence (`last_applied_lsn >= committed_lsn`)
|
||
- Build: 920 tests PASS, 5/5 meson suites OK
|
||
|
||
### Round 4: PreVote UAF (bug-1430)
|
||
- ASan caught heap-use-after-free in `ha_prevote_round_timeout_fn`
|
||
- **Fixed**: unconditional CAS before free in cancel path
|
||
- 19 PreVote unit tests pass, 0 regressions
|
||
|
||
### Round 5: Force-flush (bug-1431)
|
||
- Added force-flush for commit durability path
|
||
- Reduced loss from 8→5 but didn't eliminate
|
||
|
||
### Round 6: Residual loss (bug-1432)
|
||
- Remaining 8 losses identified as **pre-convergence false positives**
|
||
- New primary starts with `committed_lsn=0` during truncation, verify runs before convergence
|
||
- Not real data loss — timing artifact in soak harness
|
||
|
||
---
|
||
|
||
## Jul 10: Controller Invariant Audit
|
||
|
||
Created `xce-audit-invariants` skill. Ran full audit across 4 dimensions:
|
||
|
||
| Dimension | Finding | Severity |
|
||
|-----------|---------|----------|
|
||
| Raft TLA+ checklist | bug-1434: `committed_lsn` CAS race in `persist_promotion_advance_committed` | CRITICAL |
|
||
| async-I/O thread role | bug-1433: `fdatasync` blocks on delaymgr thread (blocking-I/O violation) | CRITICAL |
|
||
| dispatch gate | Clean — all gates verified | — |
|
||
| join/startup race | Clean — no exploit path | — |
|
||
|
||
### bug-1433: async-thread fdatasync block
|
||
- `flusher_flush_once` called `fdatasync()` directly, blocking the delaymgr worker
|
||
- Violation of async-I/O invariant (blocking ops must go through slowlane or dedicated thread)
|
||
- **Fix**: moved fdatasync to slowlane
|
||
|
||
### bug-1434: committed_lsn CAS race
|
||
- `persist_promotion_advance_committed` used CAS without lock, racing with concurrent writers
|
||
- Violation of Raft TLA+ invariant (committed_lsn must be monotonic and consistent)
|
||
- **Fix**: replaced CAS with atomic store under existing lock
|
||
|
||
**Validation**: 20-round chaotic + 20-round sequential both PASS (63 jobs, 0 loss, 0 divergence, all 3 nodes led).
|
||
|
||
Committed: `40f510f`.
|
||
|
||
---
|
||
|
||
## Jul 10-11: Spec Work
|
||
|
||
### scheduler-framework (design discussion)
|
||
Four-level picker architecture designed:
|
||
|
||
| Level | Picker | Plugin slot | Default (v1) |
|
||
|-------|--------|-------------|---------------|
|
||
| 0 | Project picker | `sched->project_pick_plugin` | `builtin_flat` |
|
||
| 1 | Inter-queue picker | `sched->queue_pick_plugin` | `builtin_strict_priority` |
|
||
| 2 | Intra-queue picker | `queue→project→sched` chain | `builtin_fcfs` |
|
||
| 3 | Preemption policy | `queue->preempt_policy` | `lowest_priority_first` |
|
||
|
||
Key design decisions:
|
||
- Unified picker loop: always `project → queue → job`, identical code for v1+v2
|
||
- Queue = scheduling mechanism (HOW), Project = resource ownership (WHO)
|
||
- Preemption: SIGSTP-based v1, LSF-style cross-queue victim selection
|
||
- New `XCE_DECISION_PREEMPT` decision type flowing scheduler→controller→worker
|
||
- `requirements.md` rewritten: 6→9 requirements, architecture diagrams, data structure changes
|
||
- `brief.md` updated: four-level architecture, queue vs project roles, preemption flow
|
||
|
||
### admin-config-tool
|
||
Spec finalized: 10 EARS requirements, 3-skill harness (xce-build/deploy-ha/test-ha), Nix 7 packages, controller upgrade pipeline verified. Roadmap: rollback + entity-snapshot phases deferred.
|
||
|
||
### ha-elect-ctx-lifecycle
|
||
Requirements generated (`58e047f`). New spec addressing election context lifecycle management.
|
||
|
||
---
|
||
|
||
## Build & Test Infrastructure
|
||
|
||
| Change | Detail |
|
||
|--------|--------|
|
||
| xce-build/xce-test-ha/xce-deploy-ha | Subagent dispatch fixed (`5e0579d`) |
|
||
| xce-test-ha skill | Updated docs: 5-50 round range (was stale 5-10) |
|
||
| xce-audit-invariants skill | New: audit+fix+regression loop workflow |
|
||
|
||
---
|
||
|
||
## Bug Tally: Jul 6-11
|
||
|
||
### Fixed (17)
|
||
1416, 1417, 1418, 1419(×3 layers), 1420, 1421, 1422, 1423, 1424, 1426, 1427, 1428, 1430, 1431, 1433, 1434
|
||
|
||
### Known residual
|
||
- **bug-1432**: Pre-convergence false-positive committed_job_loss in soak harness (not real data loss)
|
||
- **bug-1425**: Intermittent client hang after failover in synchronous jobsub path (~2/9 attempts)
|
||
|
||
---
|
||
|
||
## Commits (Jul 7-11)
|
||
|
||
```
|
||
40f510f fix(controller): bug-1433 async-thread fdatasync block + bug-1434 committed_lsn CAS race
|
||
58e047f spec(ha-elect-ctx-lifecycle): generate requirements
|
||
ea0ac81 save wolf state and document
|
||
c689351 chore(wolf): restore buglog.json, log bug-1421/bug-1422, memory trail
|
||
5e0579d feat(skills): fix xce-build/xce-test-ha/xce-deploy-ha subagent dispatch
|
||
016c598 fix(xce-test-ha): bug-1422 verify_no_loss false-positive after election
|
||
327dbb9 fix(ha): bug-1421 close term-desync AE-reject blindspot in HA election
|
||
0941cf9 fix(ha): close two independent chaotic-failover bugs — term-desync liveness stall and bug-847 WAL batch payload loss
|
||
11244dd fix(raft-apply-decoupling): bug-1419 real fix — replay tally, not dedup-guard
|
||
f28ce57 fix(raft-apply-decoupling): bug-1419 layer 3 — close cross-primary client-retry dedup gap
|
||
1b6041a fix(xce-test-ha): bug-1420 harness jobid collision + bug-1419 regression coverage
|
||
0eb1054 fix(raft-apply-decoupling): bug-1419 layer 2 — checkpoint_lsn mislabeled from WAL tip instead of applied cursor
|
||
fe4352f fix(raft-apply-decoupling): bug-1419 layer 1 — close resubmit-dedup gap left by apply-after-commit deferral
|
||
9b173c9 fix(xce-test-ha): guard start_node against duplicate launches (bug-1418)
|
||
d512275 fix(ha): correct real-vote reply index mapping for middle-priority nodes (bug-1417)
|
||
903b284 chore(raft-apply-decoupling): mark task group 7 complete, close out spec
|
||
40196be perf(raft-apply-decoupling): fix O(n^2) apply-pump WAL rescan, measure throughput (task 7.4)
|
||
```
|
||
|
||
---
|
||
|
||
## Current State
|
||
|
||
| Area | Status |
|
||
|------|--------|
|
||
| HA sequential soak | 40-round PASS (zero loss) |
|
||
| HA chaotic soak | 20-round PASS after bug-1433/1434 fixes |
|
||
| raft-apply-decoupling | COMPLETE (all 7 task groups) |
|
||
| ha-term-safety | Phases 1-5 complete, tasks 1-5 done |
|
||
| ha-elect-ctx-lifecycle | Requirements generated, pending design |
|
||
| ha-commit-durability | Phases 6.1-6.4 ext4-validated |
|
||
| scheduler-framework | Requirements updated (design discussion), pending approval |
|
||
| admin-config-tool | Spec finalized, pending implementation |
|
||
| Overlayfs bug-924 | fdatasync wedge on overlayfs — known, workaround: use `~/tmp` (ext4) |
|