myxce/reports/sep13-sep19-2026.md

# XCE Project Status — Sep 13 – Sep 19, 2026

> Note: the calendar-default window (Sep 07–Sep 12) is already covered by
> `sep06-sep12-2026.md`; this report covers the actual gap — the last
> completed day before this report was generated.

## TL;DR

A scheduler-architecture week, one spec per day or two: Monday relocated `xce_test` out of `controller` into its own standalone `harness` component; Tuesday was a live-replay-driven resource-accounting firefight (7 bugs fixed same-day — stale `JOBS_PUSH` resync races, a systemd rusage sentinel bug, a reconciliation-freeze gap); Wednesday shipped `controller-status`'s Req 12 (election term/time) and Req 13 (per-controller connection status) deltas to feature-level GO; Thursday landed `scheduler-sim-diagnostics`' human-readable limit-inspection feature (Req 1–5) end-to-end, plus a run of `deploy-ha` worker-cleanup fixes; Friday–Saturday built and shipped `scheduler-sim-live-allocation` — a genuinely large unification merging the normal-mode driver's and sim-mode driver's job-election/allocation logic into one shared implementation (`sched_election.c`), closing bug-1825/1826 (sim-mode limit-plugin bootstrap and matching) along the way. 25 new bugs were opened, 22 fixed, 2 deliberately deferred, and 1 fix attempt was reverted after it broke the recorded-decision-reconciliation live-verify test. ⚠️ A mass bug-log retag on Saturday touched 1,164 historical entries with an identical timestamp — at least one now carries a "fixed" tag that contradicts its own fix-field prose (see Bugs section).

## Commits by day

### Sun 09-13 (Sunday) — bookkeeping only
- **`04807985`..`b8ef82c1`** (2 commits) — added last week's report (`sep06-sep12-2026.md`) and `CHANGES.md`, then a routine `.wolf` sync. No code changes.

### Mon 09-14 (Monday) — xce-harness-component: full spec in one day
- **`cf55d691`..`868c6427`** (19 commits) — relocated `xce_test` and its four standalone test binaries out of `controller` into a new standalone `harness` component (own `/xce-build harness` target, decoupled from `controller`'s build), promoted `ctrlstatus_fmt.h` to the shared `xce/` include path, and wired `harness` into `xce-test-ha`/`xce-test-multihost`/`xce-test-resources`'s explicit build/check steps. Verified build isolation, `xce_test`'s post-relocation behavioral parity, and every downstream workflow end-to-end (tasks 1–4, all in one day). Also fixed **bug-1807** (`getopt_long`'s `optind=0` reset, cross-command state corruption) and **bug-1808** (a longjmp'd Unity assertion leaking a spawned `xce_test` child process).

### Tue 09-15 (Tuesday) — live-replay resource-accounting firefight
- **`b7250c2e`..`7f3237ae`** (8 commits) — a live-replay debugging session found and fixed, same-day: **bug-1809** (routine `EEXIST` resync log flooding operational output), **bug-1810** (`cbuf_put`/`cbuf_get` orphaning their mutex on cancellation, hanging `vtaskmgr_fini`), **bug-1812/1813** (a stale `JOBS_PUSH` resync claim could destroy a real allocation; job-array expansion moved off the vtask thread pool as the underlying architectural fix), **bug-1814** (worker rusage collection was treating systemd's `UINT64_MAX` sentinel as real memory data), **bug-1815** (mem-usage reconciliation now freezes for a RECONCILING/UNAVAIL worker too, closing a gap next to bug-1801), and **bug-1816** (`xce-deploy-ha`: stop a stale deploy's processes surviving a hand-typed cleanup mistake). Also added a live-computed default for `xce-build`'s `--jobs` flag.

### Wed 09-16 (Wednesday) — controller-status: two requirement deltas to GO
- **`63de9473`..`d19e9f27`** (20 commits) — **Req 12** (primary election term/time): new schema fields + `ha_state_t`, stamped on both promotion roads, carried on the heartbeat, sourced in `ctrlstatus_fill`, rendered in the `ctrlstatus` CLI — reached feature-level GO. **Req 13** (per-controller connection status): same shape (schema field → `ctrlstatus_fill` → CLI render) — reached feature-level GO, but its own live E2E validation (task 8.6) found **bug-1819**: `ctrlstatus_fill` was preferring a stale disconnected registry slot over a live one (slot-reuse, first-match-wins bug), fixed same-day. Also fixed **bug-1818** (a dead-code gate had `cmd_ctrlstatus` never actually reaching its own display-layer election check) and a `zarray_fini`/`xlist_fini` bug that left stale `caps`/`count` behind instead of resetting them.

### Thu 09-17 (Thursday) — scheduler-sim-diagnostics limit inspection, plus deploy-ha/xceadmin fixes
- **`baa07d78`..`77924f78`** (18 commits, 2 `.wolf`-only) — built `info limit`'s full feature set end-to-end: name/`all` resolution, configured-setting display via `fmt_limit()`, effective host-scope notes (Req 1.6), live usage for both cluster-wide and host-scoped limit instances, and REPL dispatch wiring — with a read-only/deterministic proof as the closing task. Found and logged **bug-1824** (a latent, pre-existing `zarray` bounds imprecision in `repl_info_resource_one()`, left unfixed as out-of-scope) along the way. Same day: **bug-1820/1823** (`stop-all.sh`/`cleanup.sh` couldn't find a real, sudo-launched worker regardless of process-tree depth — two rounds, the second a regression on the first's own fix) and **bug-1821/1822** (`xceadmin` now accepts a comma-separated `--controller` failover list; found a `NoPrimaryError` retry gap live while verifying it).

### Fri 09-18 (Friday) — scheduler-sim-live-allocation: foundation
- **`fc23ea8d`..`2f1ffd1e`** (8 commits) — started the driver-unification effort: threaded core state through allocation context and host search, made plugin-hook dispatch, capacity checks, allocation-dispatch/decision-commit, and limit matching all "core-safe" (usable from either driver). Same day, fixed **bug-1825**: the limit plugin was never registered on any sim-mode driver path at all — a `limit_new_plugin()` factory plus a new `sim_bootstrap_limit_plugin()` (mirroring `sim_picker.c`'s own bootstrap shape) closed the gap.

### Sat 09-19 (Saturday) — scheduler-sim-live-allocation: unification to done
- **`4195331b`..`8c363de7`** (13 commits, 1 `.wolf`-only) — extracted the shared picker/allocator into a new `sched_election.c`, wired simulation mode onto it, narrowed reconciliation to a timing-gap safety net, closed a host-scoped limit fan-out seam gap, then proved contention parity across drivers and replay-driven limit visibility (closing **bug-1826**, the sim-mode limit-*matching* follow-up to Friday's bootstrap fix) and reconciled the equivalence test suites. Also fixed **bug-1682/1492** (`plugin_end_session()` guarding on the wrong field) and **bug-1827** (a `core_clock.c` coverage gap in the core-isolation test found during an `/xce-audit-invariants scheduler` run), and corrected a stale bug-1611 status claim in the driver-equivalence steering doc.

## Bugs opened/fixed this window

- **25 genuinely new bugs** (bug-1805 – bug-1829): **22 fixed**, **2 deliberately deferred** (bug-1806 — accepted as a documented Meson platform limitation, user decision 2026-09-14; bug-1824 — latent pre-existing bug, out of its discovering task's scope), **1 reverted** (bug-1811 — an initial guard-based fix broke `test_e2e_original_motivation_long_replay`, the live-verify for the entire Recorded-Decision Reconciliation mechanism, and was rolled back; reclassified as a design decision, not a defect).
- **Carry-over status from last week's report**: `xce-test-cli-ux`'s `spec.json` metadata lag — **still unsynced**, now 2 windows running. bug-1796/1797/1793/1777 — **still open**, deliberately deferred, unowned. bug-1714 — **still zero movement**, now a 3rd consecutive window (last real progress Aug 26). bug-1632/1726/1728 — **still zero movement**, also a 3rd consecutive window. `gitea` MCP connectivity — **still failing to connect** (confirmed again this session).
- Highest-severity fixes: **bug-1817** (a SIGSEGV core dump in a live, HA-connected `xce_scheduler` — a "previously-unroot-caused, recurring bug class" per its own tags, fixed at the `zarray`/`xlist` library-primitive level), **bug-1814/1815** (a live-user-traced memory-accounting chain: a systemd rusage sentinel bug plus the reconciliation-freeze gap it exposed), **bug-1819** (a display-layer stale-slot bug found during `controller-status`'s own live E2E validation).
- ⚠️ **Integrity concern**: a mass edit on Sat, `last_seen=2026-09-19T09:00:00Z`, added the tag `closed-by-scheduler-sim-live-allocation` (plus, on many entries, a bare `fixed` tag) to **1,164** historical bug entries — including clearly unrelated ones from as far back as May (`bug-002`). At least one (**bug-1811**) now carries a `fixed` tag that directly contradicts its own fix-field text ("REVERTED"). Any future tooling that trusts `.tags` or `.fix`-field-non-empty as a status oracle will misclassify entries like this — the fix field's actual prose (read fresh) remains the only reliable signal. Worth either a stale-tag cleanup pass or documenting this caveat permanently in the buglog's own schema notes.

## Spec status changes

- **`xce-harness-component`** — `implementation-complete` (Mon), 16/20 tasks checked (the 4 unchecked are top-level group headers whose own subtasks are all done — a checkbox-convention gap, not real incompleteness). Metadata is correctly synced this time.
- **`controller-status`** — Req 12 and Req 13 deltas both reached feature-level GO this window (Wed); 29/30 tasks done, the 1 remaining explicitly marked optional (`8.5`, live-cluster observation). `spec.json` still reads phase `tasks-generated` / `ready_for_implementation: true` rather than `implementation-complete` — a metadata-sync candidate, same class of gap flagged for `xce-test-cli-ux`.
- **`scheduler-sim-diagnostics`** — Requirements 1–5 (limit inspection) shipped end-to-end (Thu), 17/19 tasks checked (tasks 8 and 10's own subtasks are all done; only their parent headers are unflipped, same convention gap as above). Requirement 6 (human-readable units) had not started as of this window's close.
- **`scheduler-sim-live-allocation`** — a new spec, taken from Task 1.1 to Task 5.4 (all subtasks complete per commit history) in two days (Fri–Sat). `spec.json` still reads `tasks-generated` / `ready_for_implementation: false` — the THIRD spec this window showing the same phase/checkbox metadata lag (alongside `xce-test-cli-ux` and `controller-status`), suggesting a quick batch phase-sync pass would clear all three at once.

## Stats

- **89 commits** — **83 touching at least one non-`.wolf` file**, **6 pure-`.wolf` bookkeeping** (4 routine `chore(openwolf)` syncs, 2 buglog-only edits).
- **454 non-`.wolf` file changes**, **+21,856 / −2,047** (non-wolf headline; raw incl. `.wolf`: +30,969 / −5,452).
- Busiest areas by files touched: `src/scheduler` (78), `src/harness` (50), `.kiro` (25), `src/lib` (12), `src/controller` (12), `.claude` (8), `src/xceadmin` (7), `.wolf` (6), `src/include` (4).
- By spec scope: `controller-status` (19 commits), `xce-harness-component` (18), `scheduler-sim-live-allocation` (14), `scheduler-sim-diagnostics` (13), `scheduler` (7, unscoped fixes).
- **1 spec shipped and correctly synced** (`xce-harness-component`); **2 requirement deltas reached feature-level GO** (`controller-status` Req 12/13); **2 specs functionally complete pending metadata sync** (`scheduler-sim-diagnostics` Req 1–5, `scheduler-sim-live-allocation`).
- **10 sessions** logged in `.wolf/memory.md` touching this window (Sep 14 ×3, Sep 15 ×3, Sep 16 ×2, Sep 17 ×1, Sep 18 ×1); busiest logged day **Sep 18** (383 actions — the `scheduler-sim-live-allocation` foundation day).
- Commits landed on all 7 days; peak day **Wed 09-16** (20 commits, `controller-status` Req 12+13), close behind by **Sat 09-19** (13 commits, closing `scheduler-sim-live-allocation`).

## Open threads / to-do heading into next session

1. **Metadata-lag pattern now spans three specs**: `xce-test-cli-ux` (2 windows unaddressed), and newly `controller-status` (29/30 done, both deltas GO) and `scheduler-sim-live-allocation` (all subtasks done) — all three still read a stale phase/`ready_for_implementation` in `spec.json`. A single batch phase-sync pass would clear all three at once.
2. **Buglog integrity**: the Sat mass retag (1,164 entries, `closed-by-scheduler-sim-live-allocation`, identical `last_seen`) leaves `.tags` and `.fix`-non-empty unreliable as status signals — confirmed via `bug-1811`'s contradicting `fixed` tag. Needs a cleanup pass or a documented caveat before any tooling trusts those fields directly again.
3. **`gitea` sync backlog now 32 bugs deep** (bug-1798 through bug-1829) — the MCP server is still failing to connect (`CONNECT_TIMEOUT`, reconfirmed this session). Retry `gitea-sync` once reachable.
4. **`bug-1714`/`bug-1632`/`bug-1726`/`bug-1728`** — zero movement for a 3rd consecutive window each. Worth an explicit triage-or-accept decision rather than letting them keep aging silently.
5. **`bug-1796`/`bug-1797`/`bug-1793`/`bug-1777`** — still deliberately deferred from `xce-test-cli-ux`'s validation tasks, still unowned.
6. **`scheduler-sim-eager-reconciliation`'s live-replay smoke pass** (flagged last week) — still not directly done; `scheduler-sim-live-allocation`'s tasks 5.1/5.2/5.4 delivered adjacent live-replay proof, but for a different spec — the original open item remains untouched.
7. **`scheduler-sim-live-allocation` shipped a full driver-unification architecture in two days** — before treating it as fully done, it warrants a live multi-round replay or HA-adjacent smoke pass beyond the unit suite, the same caution this project's own `/xce-audit-invariants` methodology applies to architectural/consistency changes of this size.