myxce/reports/jul-06-jul-11.md

Jul 6 → Jul 11 Progress Report

Overview

Five days of HA stability hardening, chaotic soak debugging, spec work, and controller invariant audit. raft-apply-decoupling completed. Two controller invariant bugs fixed (1433/1434) bringing HA soaks to clean PASS. Three specs advanced (scheduler-framework design, admin-config-tool finalized, ha-elect-ctx-lifecycle initiated).

---

Jul 6-7: raft-apply-decoupling — COMPLETE

Final tasks closed out the raft-apply-decoupling spec:

Task	Detail	Commit
7.2	kill-mid-fact-stream soak, fix bug-1416 boot-time TOCTOU	a5e29a6
7.3	Stale apply-before-commit assertions updated	16b890b
7.4	O(n²) apply-pump WAL rescan → O(log n) throughput benchmarked	40196be

Bugs fixed during close-out:

• bug-1416: persist recover TOCTOU — boot-time applied_lsn > committed_lsn gap
• bug-1419 (×3 layers): resubmit-dedup gap — layer 1 (fe4352f), layer 2 (0eb1054), layer 3 (f28ce57)
• bug-1420: xce_test CLI harness jobid collision within same wall-clock second

Spec marked complete: 903b284.

---

Jul 7-9: HA Election Liveness — Cascade of Bugs

Six bugs discovered and fixed in rapid succession, all around election/term/vote correctness:

Bug	Manifestation	Root cause	Fix
1417	Election stall, votes silently dropped	Vote reply index mapping: compressed local index → absolute peer index	ha_repl_ack_abs_peer_idx resolver (d512275)
1418	Term-desync + duplicate-node guard	Two factors: (1) ha->term/raft.term desync on standby, (2) start_node re-launch on already-running process	Multi-pass fix (9b173c9, 48b4dba)
1421	AE-reject blindspot, term desync cycle	raft_on_higher_term_observed ignored stale-term AE when term was already advanced	Close blindspot (327dbb9)
1422	verify_no_loss false-positive after election	Race: job presence check ran before post-election convergence completed	Retry with backoff (016c598)
1423	vote-req failures silently dropped (production path)	Production vote-reply dispatch never called; only PreVote path ran	Route prod path through dispatch
1424	PreVote twin of 1423	PreVote connect/monitor failure branches never prod	Same pattern, PreVote path

Validation: 40-round sequential PASS (111 jobs, 0 loss), 30-round chaotic PASS (93 jobs, 0 loss).

---

Jul 9-10: Chaotic Soak Debugging — Major Effort

~6 chaotic soak rounds chasing persistent committed_job_loss=5-8:

Round 1: fdatasync investigation

• Initial hypothesis: bug-924 (overlayfs fdatasync EIO)
• Disproved: errors were errno=22 EINVAL on ext4, not overlayfs
• Stale attribution in buglog corrected

Round 2: Disk cleanup

• 11,062 stale xce_waldump tmp files accumulated across rounds
• Freed 156GB (86% → 68% disk), eliminated 2 false-positive losses

Round 3: persist_apply_pump_run_sync race

• CAS-coalescing pattern caused apply-pump to bail out before catching up
• Fixed: replaced with guaranteed loop-until-convergence (last_applied_lsn >= committed_lsn)
• Build: 920 tests PASS, 5/5 meson suites OK

Round 4: PreVote UAF (bug-1430)

• ASan caught heap-use-after-free in ha_prevote_round_timeout_fn
• Fixed: unconditional CAS before free in cancel path
• 19 PreVote unit tests pass, 0 regressions

Round 5: Force-flush (bug-1431)

• Added force-flush for commit durability path
• Reduced loss from 8→5 but didn't eliminate

Round 6: Residual loss (bug-1432)

• Remaining 8 losses identified as pre-convergence false positives
• New primary starts with committed_lsn=0 during truncation, verify runs before convergence
• Not real data loss — timing artifact in soak harness

---

Jul 10: Controller Invariant Audit

Created xce-audit-invariants skill. Ran full audit across 4 dimensions:

Dimension	Finding	Severity
Raft TLA+ checklist	bug-1434: committed_lsn CAS race in persist_promotion_advance_committed	CRITICAL
async-I/O thread role	bug-1433: fdatasync blocks on delaymgr thread (blocking-I/O violation)	CRITICAL
dispatch gate	Clean — all gates verified	—
join/startup race	Clean — no exploit path	—

bug-1433: async-thread fdatasync block

• flusher_flush_once called fdatasync() directly, blocking the delaymgr worker
• Violation of async-I/O invariant (blocking ops must go through slowlane or dedicated thread)
• Fix: moved fdatasync to slowlane

bug-1434: committed_lsn CAS race

• persist_promotion_advance_committed used CAS without lock, racing with concurrent writers
• Violation of Raft TLA+ invariant (committed_lsn must be monotonic and consistent)
• Fix: replaced CAS with atomic store under existing lock

Validation: 20-round chaotic + 20-round sequential both PASS (63 jobs, 0 loss, 0 divergence, all 3 nodes led).

Committed: 40f510f.

---

Jul 10-11: Spec Work

scheduler-framework (design discussion)

Four-level picker architecture designed:

Level	Picker	Plugin slot	Default (v1)
0	Project picker	sched->project_pick_plugin	builtin_flat
1	Inter-queue picker	sched->queue_pick_plugin	builtin_strict_priority
2	Intra-queue picker	queue→project→sched chain	builtin_fcfs
3	Preemption policy	queue->preempt_policy	lowest_priority_first

Key design decisions:

• Unified picker loop: always project → queue → job, identical code for v1+v2
• Queue = scheduling mechanism (HOW), Project = resource ownership (WHO)
• Preemption: SIGSTP-based v1, LSF-style cross-queue victim selection
• New XCE_DECISION_PREEMPT decision type flowing scheduler→controller→worker
• requirements.md rewritten: 6→9 requirements, architecture diagrams, data structure changes
• brief.md updated: four-level architecture, queue vs project roles, preemption flow

admin-config-tool

Spec finalized: 10 EARS requirements, 3-skill harness (xce-build/deploy-ha/test-ha), Nix 7 packages, controller upgrade pipeline verified. Roadmap: rollback + entity-snapshot phases deferred.

ha-elect-ctx-lifecycle

Requirements generated (58e047f). New spec addressing election context lifecycle management.

---

Build & Test Infrastructure

Change	Detail
xce-build/xce-test-ha/xce-deploy-ha	Subagent dispatch fixed (5e0579d)
xce-test-ha skill	Updated docs: 5-50 round range (was stale 5-10)
xce-audit-invariants skill	New: audit+fix+regression loop workflow

---

Bug Tally: Jul 6-11

Fixed (17)

1416, 1417, 1418, 1419(×3 layers), 1420, 1421, 1422, 1423, 1424, 1426, 1427, 1428, 1430, 1431, 1433, 1434

Known residual

• bug-1432: Pre-convergence false-positive committed_job_loss in soak harness (not real data loss)
• bug-1425: Intermittent client hang after failover in synchronous jobsub path (~2/9 attempts)

---

Commits (Jul 7-11)

40f510f fix(controller): bug-1433 async-thread fdatasync block + bug-1434 committed_lsn CAS race
58e047f spec(ha-elect-ctx-lifecycle): generate requirements
ea0ac81 save wolf state and document
c689351 chore(wolf): restore buglog.json, log bug-1421/bug-1422, memory trail
5e0579d feat(skills): fix xce-build/xce-test-ha/xce-deploy-ha subagent dispatch
016c598 fix(xce-test-ha): bug-1422 verify_no_loss false-positive after election
327dbb9 fix(ha): bug-1421 close term-desync AE-reject blindspot in HA election
0941cf9 fix(ha): close two independent chaotic-failover bugs — term-desync liveness stall and bug-847 WAL batch payload loss
11244dd fix(raft-apply-decoupling): bug-1419 real fix — replay tally, not dedup-guard
f28ce57 fix(raft-apply-decoupling): bug-1419 layer 3 — close cross-primary client-retry dedup gap
1b6041a fix(xce-test-ha): bug-1420 harness jobid collision + bug-1419 regression coverage
0eb1054 fix(raft-apply-decoupling): bug-1419 layer 2 — checkpoint_lsn mislabeled from WAL tip instead of applied cursor
fe4352f fix(raft-apply-decoupling): bug-1419 layer 1 — close resubmit-dedup gap left by apply-after-commit deferral
9b173c9 fix(xce-test-ha): guard start_node against duplicate launches (bug-1418)
d512275 fix(ha): correct real-vote reply index mapping for middle-priority nodes (bug-1417)
903b284 chore(raft-apply-decoupling): mark task group 7 complete, close out spec
40196be perf(raft-apply-decoupling): fix O(n^2) apply-pump WAL rescan, measure throughput (task 7.4)

---

Current State

Area	Status
HA sequential soak	40-round PASS (zero loss)
HA chaotic soak	20-round PASS after bug-1433/1434 fixes
raft-apply-decoupling	COMPLETE (all 7 task groups)
ha-term-safety	Phases 1-5 complete, tasks 1-5 done
ha-elect-ctx-lifecycle	Requirements generated, pending design
ha-commit-durability	Phases 6.1-6.4 ext4-validated
scheduler-framework	Requirements updated (design discussion), pending approval
admin-config-tool	Spec finalized, pending implementation
Overlayfs bug-924	fdatasync wedge on overlayfs — known, workaround: use ~/tmp (ext4)