myxce/reports/sep20-sep26-2026.md

XCE Project Status — Sep 20 – Sep 26, 2026

Note: the calendar-default window (Sep 14–Sep 19) is already covered by sep13-sep19-2026.md; this report covers the actual gap — Sun Sep 20 through Sat Sep 26, the last completed day before this report was generated.

TL;DR

A bug-burn-down week driven by live evidence rather than spec work. Sunday closed out scheduler-sim-diagnostics Requirement 6 (human-readable limit units) and fixed a cluster of percent-limit bugs it surfaced. Monday–Thursday were a run of user-driven live scheduler replays and HA soaks: the replays found 11 scheduler/sim bugs (over-dispatch past live capacity, --sim-verify false positives, a recorder that silently truncated decisions, mid-round rejoin truncation), and the soaks found three controller SIGSEGVs (HA election, standby-endpoint handler, rejoin use-after-free). Thursday also rebuilt the bug log's bookkeeping: an explicit status field backfilled on all 1,193 entries, a buglog skill, and a status-driven gitea-sync. That resolves last report's buglog-integrity and Gitea-backlog threads. Friday cleared a backlog of scheduler COMPOUND/ALTER allocation bugs and closed the scheduler-allocation spec; Saturday made WAL failure facts and checkpoints field-complete (exit code and finish time) and hardened the worker's root-owned /tmp opens against planted symlinks. 41 new bugs logged (bug-1830 – bug-1870): 37 fixed, 2 cannot-reproduce, 2 still open, plus 8 older bugs fixed.

Commits by day

Sun 09-20 (Sunday) — scheduler-sim-diagnostics Req 6 to done, percent-limit fallout

• eb23c158..bf61f595 (6 commits) — tasks 7–10: promoted byte-unit formatting to the shared lib, added fmt_limit_units() and wired info limit to it, corrected the usage display's resource classification (task 9), then strengthened coincidentally-passing assertions and closed cross-cutting test gaps (10.1/10.2).
• 0ce71254, a8fb7685, 7982f285, ea76f477 — the user's own retest of Req 6 found bug-1831 (percent limits displayed as a raw fraction), bug-1832 (night_mem_pct in limits.yaml was 100× too large), bug-1833 (xceadmin now validates percent values are in [0.0, 1.0]) and bug-1834 (percent-limit capacity read mem/swap instead of maxmem/maxswap). bug-1830 was logged then corrected same-day (07b5c989) after a retest showed mem-limit usage does populate.
• 0a983a74 — last week's report and change log.

Mon 09-21 (Monday) — HA election crash + sim over-dispatch

• cd9ecb41 — bug-1835: a live 3-node HA cluster SIGSEGV in leader election, caused by CtrlHelloRequest.key's Data field.
• 94115b00, 4f94638a — bug-1836/1838: sim mode force-dispatched recorded decisions past live capacity (user-reported live over-dispatch during xce_scheduler -i -s <replay>); bug-1837: sim reconciliation's on_dispatch() now routes through the allocator thread (found by /xce-audit-invariants scheduler).

Tue 09-22 (Tuesday) — HA soak crash + RECONCILING never cleared

• e1da9f1e — bug-1839: the standby-endpoint error handler freed ep->data under a live vtask (controller SIGSEGV in the same soak run that produced bug-1840).
• 1f6f219d, e592e921 — bug-1841: resync_resolve_round never told the scheduler that RECONCILING had cleared (user-reported from a live replay). Added a --worker-capacity-soak mode to xce-test-ha as its regression gate.

Wed 09-23 (Wednesday) — --sim-verify correctness

• 7901db57, 5ee329b1, fcf119b4 — replaying real captures turned up bug-1842 (every DISPATCH decision false-flagged on wall-clock drift), bug-1843 (the recorder silently truncated a round's decisions on realloc failure), bug-1844/1845 (three more --sim-verify false positives) and bug-1772 (snapshot resreq_fp gap).
• 668e3f68 — new spec controller-peers-env-fallback (design generated) and scheduler backlog notes.

Thu 09-24 (Thursday) — replay/soak fixes + buglog status overhaul

• 77f047f2, a9feded8, ac033f47 — bug-1846 (host-scoped limit staged N times per dispatch; a mid-round rejoin config push truncated live rounds) and its follow-ups bug-1847/1848/1849 (config pushes now recorded, an hlimits leak, a benign picker WARN quieted).
• ca4db13a — bug-1851: string resvals now own their bytes; a rejoin use-after-free was killing a newly promoted primary during a sequential HA soak.
• 9c02184f, d9ff02cd, 5b891b50 — bug-1852 (scheduler records events before the core applies them), a scheduler replay-parity gate added to the sequential/chaotic soaks, and bug-1853 (a chaotic-soak walcompare false divergence — a harness bug, not a safety bug).
• 19104f19, e915f8de, b0f2aea4 — buglog status field: backfilled on all 1,193 entries, with the 85 whose text said not-fixed/pending checked against HEAD. Also a buglog skill (scripts/buglog.py), the .claude/rules/buglog-status.md convention, and a status-driven gitea-sync that fixes bug-1855 (gitea-sync closed any issue with a non-empty fix text, including "NOT FIXED" ones). The first sync wrote Gitea milestone/issue IDs into 65 spec.json files.
• 3cc2f6e3, 9e9f2798 — bug-1857 (xce-test-multihost now deploys a fixture config that clears limits) and bug-1858 (ASAN invalid free: release an interned queue resreq with dereference, not free).
• 6e0463e7 — new spec libxce-alloc-preason-equality (tasks generated, 0/8).

Fri 09-25 (Friday) — allocation backlog cleared, scheduler-allocation complete

• 1ecd348f, b4e9a099, aea34738, 601b245e — bug-1769/1771 (ALTER allocation type confusion; NULL queue in the legacy fcfs hooks), bug-1768 (slots/rusage checks for primaries nested in COMPOUND/ALTER), bug-1863 (a COMPOUND with 2+ primaries kept only the last one's allocation) and bug-1766 (a failed COMPOUND left a partial allocation behind). 4ba0f327 then marked scheduler-allocation implementation-complete (22/22).
• b5d59253, 7c98c056, c29518b2 — bug-1429 (a failed test hung the whole unit suite), bug-1861 (mkstemp for the config-upgrade temp file), bug-1859 (a failover outage no longer expires a finished job's retained outcome) and bug-1862 (the worker list is locked against vtask-thread readers; ASAN UAF in xcecd_virtual).
• 0d3bb965, 910b43cd, 6b1862f8, 663ea63c — bug-1864 (the vtask dispatcher now keeps per-endpoint message order; standbys were NACKing in-order catch-up), bug-1865 (never close() the primary channel's raw fd), bug-1850 (libxce retries a NOT_READY controller in place instead of failing over — the root trigger of bug-1846's mid-round rejoins) and bug-1854 (re-pushed resource definitions are not errors). All were soak-verified (d87b78dc).
• 1ba643b8 — bug-1714 and bug-1840 set to cannot-reproduce on current builds; bug-1866 logged (LeakSanitizer: remove_worker never frees launch_type).

Sat 09-26 (Saturday) — persistence field-completeness + worker /tmp hardening

• 6876e4f6, 1d6c04c2, e4bb492a — WAL_JOB_FAIL now carries the exit code (bug-1754) and the worker-reported finish time (bug-1747 FAIL half, via record_job_fail_fact_at()), and checkpoints keep Job.exitInfo in the capnp schema (bug-1867). These close the save/restore field-completeness gaps persistence.md calls out.
• 63fd3d2a — bug-1870: checkpoints no longer punch a hole in the replicated log (the checkpoint marker was never replicated). Found because the force-snapshot soak could not submit enough jobs to trigger a checkpoint.
• 7723a4f4, 7864a3da, 5df33e69 — worker hardening: fetchfile only reads the job's own output file (bug-1508), and neither a job's output file (bug-1868) nor any other fixed root-owned /tmp name (bug-1869) follows a planted symlink.
• ff4e2cb8, 21e72726 — gitea-sync --ids; bugs synced to Gitea #264–#289.

Bugs opened/fixed this window

• 41 new bugs (bug-1830 – bug-1870; no ID collisions — the duplicate-ID census is now 0): 37 fixed, 2 cannot-reproduce (bug-1830 — framing corrected by the user's retest; bug-1840 — scheduler SIGABRT, no ASAN repro, strongest candidate fixed via bug-1769), 2 open (bug-1856 — an upstream OpenWolf duplicate-ID hook defect, mitigated locally by openwolf.buglog.auto_detect=false; bug-1866 — xcecd_virtual leak at exit).
• 8 older bugs fixed in the window: bug-1429, bug-1508, bug-1747, bug-1754, bug-1766, bug-1768, bug-1769, bug-1771. bug-1714 moved to cannot-reproduce.
• Carry-over status from last week's report:
  • Buglog integrity (the Sep 19 mass retag) — resolved structurally: status is now the single source of truth, and tags/fix text are explicitly not status signals. One loose end: bug-1811 reads fixed with no status_note, although last week's report recorded its fix as reverted and reclassified as a design decision.
  • Gitea sync backlog (32 bugs) — resolved: status-driven gitea-sync pushed #264–#289. (The gitea MCP server still fails to connect; the sync script works without it.)
  • bug-1714 → cannot-reproduce (Sep 25). bug-1726 → cannot-reproduce (Sep 24 triage). bug-1632 — still open, 4th window. bug-1728 — still open at window close; fixed Sep 27 (33c4f28c, next window).
  • bug-1777/1793/1796/1797 — still open, now explicitly tracked as such; 4th window unowned.
  • bug-1824 — still open (deferred).
• Highest-severity fixes: bug-1835/1839/1851 — three live HA-cluster controller SIGSEGVs, all fixed within a day of being found. bug-1864/1865 — standby catch-up NACKs and a WAL fd closed out from under a standby at failover. bug-1870 — a hole in the replicated log at every checkpoint. bug-1868/1869/1508 — root-owned worker file opens following planted symlinks or reading arbitrary paths (security-relevant).

Spec status changes

• scheduler-sim-diagnostics — Requirement 6 (human-readable units) was built and validated end to end on Sunday (tasks 7–10), so the whole spec is now functionally complete. tasks.md shows 17/19 because the parent headers for tasks 8 and 10 are unchecked while their subtasks are done (the same convention gap as last week). spec.json still reads tasks-generated.
• scheduler-allocation — marked implementation-complete, 22/22 (4ba0f327, Fri), after the COMPOUND/ALTER bug sweep.
• controller-peers-env-fallback — new; design-generated, not ready for implementation (Wed).
• libxce-alloc-preason-equality — new; tasks-generated, 0/8, ready_for_implementation: false (Thu).
• Metadata lag — not addressed, now 4 specs: controller-status (29/30, both deltas GO), scheduler-sim-live-allocation (14/19, ready: false though the spec shipped), xce-test-cli-ux (23/27, 3rd window) and now scheduler-sim-diagnostics all still read tasks-generated.
• 65 spec.json files gained a gitea block (milestone and issue IDs) from the first status-driven sync. This is bookkeeping, not phase changes.

Stats

• 82 commits — 56 touching at least one non-.wolf file, 26 .wolf-only (mostly chore(buglog) status updates plus OpenWolf syncs).
• 347 non-.wolf file changes, +14,670 / −1,200 (non-wolf headline; raw incl. .wolf: +20,285 / −5,774).
• Busiest areas by distinct files touched: .kiro (80 — 65 of them from the one Gitea-ID commit), src/scheduler (67), src/controller (29), src/lib (17), src/worker (16), src/include (11), .claude (10).
• By commit scope: buglog (24), scheduler (17), controller (10), scheduler-sim-diagnostics (6), worker (5), xce-test-ha (3).
• 1 spec completed (scheduler-allocation), 1 requirement delta finished (scheduler-sim-diagnostics Req 6), 2 new specs started.
• 11 sessions logged in .wolf/memory.md (Sep 21 ×2, Sep 22 ×2, Sep 23 ×1, Sep 24 ×5, Sep 26 ×1). The busiest logged day was Sep 21 (156 actions). Sep 20 and Sep 25 have commits but no session headers.
• Commits landed on all 7 days. The peak day was Fri 09-25 (25 commits, 13 of them fixes), followed by Thu 09-24 (18) and Sat 09-26 (17).
• Everything is on scheduler-framework / origin/scheduler-framework; 55 of the 82 are also on for_test.

Open threads / to-do heading into next session

1. HA / async-I/O group landed just after the window closed (Sep 27 UTC, 7aac018c..c21ca0b7, pushed): bug-1450/1451/1728/1433/1474/1469/1741, a bug-1870 follow-up (load_snapshot resumed at the checkpoint instead of checkpoint+1, cb1415d7) and opcode names for tracing. Next week's report covers these. The one new thread is bug-1871 (investigating): an HA snapshot transfer stalls 3+ s mid-stream on a rejoining standby, seen in 2 of 4 force-snapshot runs; suspected edge-triggered epoll not re-signalling data buffered while the endpoint was blocked.
2. Spec metadata sync, now 4 specs — controller-status, scheduler-sim-live-allocation, xce-test-cli-ux, scheduler-sim-diagnostics all read tasks-generated despite being functionally complete. One batch phase-sync pass (plus flipping the unchecked parent headers) clears them all.
3. Gitea is behind the buglog again: bug-1450, 1451, 1744, 1860 and 1867 are closed locally but open/partial on Gitea, and bug-1847–1849, 1851–1853, 1870 and 1871 have no Gitea issue yet. Run gitea-sync.
4. xce_test harness batch: bug-1866 (remove_worker leaks launch_type) plus bug-1777/1793/1796/1797 (stale join maxswap/maxtmp defaults, workerlist ignoring the hosts.yaml override, stale perror errno, cmd_qctrl leak) — all small, all in src/harness/worker, 4th window unowned.
5. bug-1632 (ASAN stack-use-after-return in test_sim_verify.c) — open for a 4th window. bug-1824 (repl_info.c bounds check) is still deferred. Both need a fix-or-wontfix decision.
6. bug-1811 — set its status to wontfix (or add a status_note) so it matches last week's record of a reverted fix reclassified as a design decision.
7. New specs awaiting approval: controller-peers-env-fallback (design review) and libxce-alloc-preason-equality (tasks review, ready_for_implementation: false).