# Jul 6 → Jul 11 Progress Report

## Overview

Five days of HA stability hardening, chaotic soak debugging, spec work, and controller invariant audit. **raft-apply-decoupling completed**. **Two controller invariant bugs fixed** (1433/1434) bringing HA soaks to clean PASS. **Three specs advanced** (scheduler-framework design, admin-config-tool finalized, ha-elect-ctx-lifecycle initiated).

---

## Jul 6-7: raft-apply-decoupling — COMPLETE

Final tasks closed out the raft-apply-decoupling spec:

| Task | Detail | Commit |
|------|--------|--------|
| 7.2 | kill-mid-fact-stream soak, fix bug-1416 boot-time TOCTOU | `a5e29a6` |
| 7.3 | Stale apply-before-commit assertions updated | `16b890b` |
| 7.4 | O(n²) apply-pump WAL rescan → O(log n) throughput benchmarked | `40196be` |

**Bugs fixed during close-out**:
- **bug-1416**: persist recover TOCTOU — boot-time applied_lsn > committed_lsn gap
- **bug-1419** (×3 layers): resubmit-dedup gap — layer 1 (fe4352f), layer 2 (0eb1054), layer 3 (f28ce57)
- **bug-1420**: xce_test CLI harness jobid collision within same wall-clock second

Spec marked complete: `903b284`.

---

## Jul 7-9: HA Election Liveness — Cascade of Bugs

Six bugs discovered and fixed in rapid succession, all around election/term/vote correctness:

| Bug | Manifestation | Root cause | Fix |
|-----|---------------|------------|-----|
| **1417** | Election stall, votes silently dropped | Vote reply index mapping: compressed local index → absolute peer index | `ha_repl_ack_abs_peer_idx` resolver (`d512275`) |
| **1418** | Term-desync + duplicate-node guard | Two factors: (1) ha->term/raft.term desync on standby, (2) start_node re-launch on already-running process | Multi-pass fix (`9b173c9`, `48b4dba`) |
| **1421** | AE-reject blindspot, term desync cycle | raft_on_higher_term_observed ignored stale-term AE when term was already advanced | Close blindspot (`327dbb9`) |
| **1422** | verify_no_loss false-positive after election | Race: job presence check ran before post-election convergence completed | Retry with backoff (`016c598`) |
| **1423** | vote-req failures silently dropped (production path) | Production vote-reply dispatch never called; only PreVote path ran | Route prod path through dispatch |
| **1424** | PreVote twin of 1423 | PreVote connect/monitor failure branches never prod | Same pattern, PreVote path |

**Validation**: 40-round sequential PASS (111 jobs, 0 loss), 30-round chaotic PASS (93 jobs, 0 loss).

---

## Jul 9-10: Chaotic Soak Debugging — Major Effort

~6 chaotic soak rounds chasing persistent `committed_job_loss=5-8`:

### Round 1: fdatasync investigation
- Initial hypothesis: bug-924 (overlayfs fdatasync EIO)
- **Disproved**: errors were `errno=22 EINVAL` on ext4, not overlayfs
- Stale attribution in buglog corrected

### Round 2: Disk cleanup
- 11,062 stale `xce_waldump` tmp files accumulated across rounds
- Freed 156GB (86% → 68% disk), eliminated 2 false-positive losses

### Round 3: persist_apply_pump_run_sync race
- CAS-coalescing pattern caused apply-pump to bail out before catching up
- **Fixed**: replaced with guaranteed loop-until-convergence (`last_applied_lsn >= committed_lsn`)
- Build: 920 tests PASS, 5/5 meson suites OK

### Round 4: PreVote UAF (bug-1430)
- ASan caught heap-use-after-free in `ha_prevote_round_timeout_fn`
- **Fixed**: unconditional CAS before free in cancel path
- 19 PreVote unit tests pass, 0 regressions

### Round 5: Force-flush (bug-1431)
- Added force-flush for commit durability path
- Reduced loss from 8→5 but didn't eliminate

### Round 6: Residual loss (bug-1432)
- Remaining 8 losses identified as **pre-convergence false positives**
- New primary starts with `committed_lsn=0` during truncation, verify runs before convergence
- Not real data loss — timing artifact in soak harness

---

## Jul 10: Controller Invariant Audit

Created `xce-audit-invariants` skill. Ran full audit across 4 dimensions:

| Dimension | Finding | Severity |
|-----------|---------|----------|
| Raft TLA+ checklist | bug-1434: `committed_lsn` CAS race in `persist_promotion_advance_committed` | CRITICAL |
| async-I/O thread role | bug-1433: `fdatasync` blocks on delaymgr thread (blocking-I/O violation) | CRITICAL |
| dispatch gate | Clean — all gates verified | — |
| join/startup race | Clean — no exploit path | — |

### bug-1433: async-thread fdatasync block
- `flusher_flush_once` called `fdatasync()` directly, blocking the delaymgr worker
- Violation of async-I/O invariant (blocking ops must go through slowlane or dedicated thread)
- **Fix**: moved fdatasync to slowlane

### bug-1434: committed_lsn CAS race
- `persist_promotion_advance_committed` used CAS without lock, racing with concurrent writers
- Violation of Raft TLA+ invariant (committed_lsn must be monotonic and consistent)
- **Fix**: replaced CAS with atomic store under existing lock

**Validation**: 20-round chaotic + 20-round sequential both PASS (63 jobs, 0 loss, 0 divergence, all 3 nodes led).

Committed: `40f510f`.

---

## Jul 10-11: Spec Work

### scheduler-framework (design discussion)
Four-level picker architecture designed:

| Level | Picker | Plugin slot | Default (v1) |
|-------|--------|-------------|---------------|
| 0 | Project picker | `sched->project_pick_plugin` | `builtin_flat` |
| 1 | Inter-queue picker | `sched->queue_pick_plugin` | `builtin_strict_priority` |
| 2 | Intra-queue picker | `queue→project→sched` chain | `builtin_fcfs` |
| 3 | Preemption policy | `queue->preempt_policy` | `lowest_priority_first` |

Key design decisions:
- Unified picker loop: always `project → queue → job`, identical code for v1+v2
- Queue = scheduling mechanism (HOW), Project = resource ownership (WHO)
- Preemption: SIGSTP-based v1, LSF-style cross-queue victim selection
- New `XCE_DECISION_PREEMPT` decision type flowing scheduler→controller→worker
- `requirements.md` rewritten: 6→9 requirements, architecture diagrams, data structure changes
- `brief.md` updated: four-level architecture, queue vs project roles, preemption flow

### admin-config-tool
Spec finalized: 10 EARS requirements, 3-skill harness (xce-build/deploy-ha/test-ha), Nix 7 packages, controller upgrade pipeline verified. Roadmap: rollback + entity-snapshot phases deferred.

### ha-elect-ctx-lifecycle
Requirements generated (`58e047f`). New spec addressing election context lifecycle management.

---

## Build & Test Infrastructure

| Change | Detail |
|--------|--------|
| xce-build/xce-test-ha/xce-deploy-ha | Subagent dispatch fixed (`5e0579d`) |
| xce-test-ha skill | Updated docs: 5-50 round range (was stale 5-10) |
| xce-audit-invariants skill | New: audit+fix+regression loop workflow |

---

## Bug Tally: Jul 6-11

### Fixed (17)
1416, 1417, 1418, 1419(×3 layers), 1420, 1421, 1422, 1423, 1424, 1426, 1427, 1428, 1430, 1431, 1433, 1434

### Known residual
- **bug-1432**: Pre-convergence false-positive committed_job_loss in soak harness (not real data loss)
- **bug-1425**: Intermittent client hang after failover in synchronous jobsub path (~2/9 attempts)

---

## Commits (Jul 7-11)

```
40f510f fix(controller): bug-1433 async-thread fdatasync block + bug-1434 committed_lsn CAS race
58e047f spec(ha-elect-ctx-lifecycle): generate requirements
ea0ac81 save wolf state and document
c689351 chore(wolf): restore buglog.json, log bug-1421/bug-1422, memory trail
5e0579d feat(skills): fix xce-build/xce-test-ha/xce-deploy-ha subagent dispatch
016c598 fix(xce-test-ha): bug-1422 verify_no_loss false-positive after election
327dbb9 fix(ha): bug-1421 close term-desync AE-reject blindspot in HA election
0941cf9 fix(ha): close two independent chaotic-failover bugs — term-desync liveness stall and bug-847 WAL batch payload loss
11244dd fix(raft-apply-decoupling): bug-1419 real fix — replay tally, not dedup-guard
f28ce57 fix(raft-apply-decoupling): bug-1419 layer 3 — close cross-primary client-retry dedup gap
1b6041a fix(xce-test-ha): bug-1420 harness jobid collision + bug-1419 regression coverage
0eb1054 fix(raft-apply-decoupling): bug-1419 layer 2 — checkpoint_lsn mislabeled from WAL tip instead of applied cursor
fe4352f fix(raft-apply-decoupling): bug-1419 layer 1 — close resubmit-dedup gap left by apply-after-commit deferral
9b173c9 fix(xce-test-ha): guard start_node against duplicate launches (bug-1418)
d512275 fix(ha): correct real-vote reply index mapping for middle-priority nodes (bug-1417)
903b284 chore(raft-apply-decoupling): mark task group 7 complete, close out spec
40196be perf(raft-apply-decoupling): fix O(n^2) apply-pump WAL rescan, measure throughput (task 7.4)
```

---

## Current State

| Area | Status |
|------|--------|
| HA sequential soak | 40-round PASS (zero loss) |
| HA chaotic soak | 20-round PASS after bug-1433/1434 fixes |
| raft-apply-decoupling | COMPLETE (all 7 task groups) |
| ha-term-safety | Phases 1-5 complete, tasks 1-5 done |
| ha-elect-ctx-lifecycle | Requirements generated, pending design |
| ha-commit-durability | Phases 6.1-6.4 ext4-validated |
| scheduler-framework | Requirements updated (design discussion), pending approval |
| admin-config-tool | Spec finalized, pending implementation |
| Overlayfs bug-924 | fdatasync wedge on overlayfs — known, workaround: use `~/tmp` (ext4) |