90 lines
14 KiB
Markdown
90 lines
14 KiB
Markdown
# XCE Project Status — Sep 20 – Sep 26, 2026
|
||
|
||
> Note: the calendar-default window (Sep 14–Sep 19) is already covered by
|
||
> `sep13-sep19-2026.md`; this report covers the actual gap — Sun Sep 20
|
||
> through Sat Sep 26, the last completed day before this report was generated.
|
||
|
||
## TL;DR
|
||
|
||
A bug-burn-down week driven by live evidence rather than spec work. Sunday closed out `scheduler-sim-diagnostics` Requirement 6 (human-readable limit units) and fixed a cluster of percent-limit bugs it surfaced. Monday–Thursday were a run of user-driven live scheduler replays and HA soaks: the replays found 11 scheduler/sim bugs (over-dispatch past live capacity, `--sim-verify` false positives, a recorder that silently truncated decisions, mid-round rejoin truncation), and the soaks found three controller SIGSEGVs (HA election, standby-endpoint handler, rejoin use-after-free). Thursday also rebuilt the bug log's bookkeeping: an explicit `status` field backfilled on all 1,193 entries, a `buglog` skill, and a status-driven `gitea-sync`. That resolves last report's buglog-integrity and Gitea-backlog threads. Friday cleared a backlog of scheduler COMPOUND/ALTER allocation bugs and closed the `scheduler-allocation` spec; Saturday made WAL failure facts and checkpoints field-complete (exit code and finish time) and hardened the worker's root-owned `/tmp` opens against planted symlinks. **41 new bugs logged (bug-1830 – bug-1870): 37 fixed, 2 cannot-reproduce, 2 still open**, plus 8 older bugs fixed.
|
||
|
||
## Commits by day
|
||
|
||
### Sun 09-20 (Sunday) — scheduler-sim-diagnostics Req 6 to done, percent-limit fallout
|
||
- **`eb23c158`..`bf61f595`** (6 commits) — tasks 7–10: promoted byte-unit formatting to the shared lib, added `fmt_limit_units()` and wired `info limit` to it, corrected the usage display's resource classification (task 9), then strengthened coincidentally-passing assertions and closed cross-cutting test gaps (10.1/10.2).
|
||
- **`0ce71254`, `a8fb7685`, `7982f285`, `ea76f477`** — the user's own retest of Req 6 found **bug-1831** (percent limits displayed as a raw fraction), **bug-1832** (`night_mem_pct` in `limits.yaml` was 100× too large), **bug-1833** (xceadmin now validates percent values are in [0.0, 1.0]) and **bug-1834** (percent-limit capacity read `mem`/`swap` instead of `maxmem`/`maxswap`). **bug-1830** was logged then corrected same-day (`07b5c989`) after a retest showed mem-limit usage does populate.
|
||
- **`0a983a74`** — last week's report and change log.
|
||
|
||
### Mon 09-21 (Monday) — HA election crash + sim over-dispatch
|
||
- **`cd9ecb41`** — **bug-1835**: a live 3-node HA cluster SIGSEGV in leader election, caused by `CtrlHelloRequest.key`'s Data field.
|
||
- **`94115b00`, `4f94638a`** — **bug-1836/1838**: sim mode force-dispatched recorded decisions past live capacity (user-reported live over-dispatch during `xce_scheduler -i -s <replay>`); **bug-1837**: sim reconciliation's `on_dispatch()` now routes through the allocator thread (found by `/xce-audit-invariants scheduler`).
|
||
|
||
### Tue 09-22 (Tuesday) — HA soak crash + RECONCILING never cleared
|
||
- **`e1da9f1e`** — **bug-1839**: the standby-endpoint error handler freed `ep->data` under a live vtask (controller SIGSEGV in the same soak run that produced bug-1840).
|
||
- **`1f6f219d`, `e592e921`** — **bug-1841**: `resync_resolve_round` never told the scheduler that RECONCILING had cleared (user-reported from a live replay). Added a `--worker-capacity-soak` mode to `xce-test-ha` as its regression gate.
|
||
|
||
### Wed 09-23 (Wednesday) — `--sim-verify` correctness
|
||
- **`7901db57`, `5ee329b1`, `fcf119b4`** — replaying real captures turned up **bug-1842** (every DISPATCH decision false-flagged on wall-clock drift), **bug-1843** (the recorder silently truncated a round's decisions on realloc failure), **bug-1844/1845** (three more `--sim-verify` false positives) and **bug-1772** (snapshot `resreq_fp` gap).
|
||
- **`668e3f68`** — new spec `controller-peers-env-fallback` (design generated) and scheduler backlog notes.
|
||
|
||
### Thu 09-24 (Thursday) — replay/soak fixes + buglog status overhaul
|
||
- **`77f047f2`, `a9feded8`, `ac033f47`** — **bug-1846** (host-scoped limit staged N times per dispatch; a mid-round rejoin config push truncated live rounds) and its follow-ups **bug-1847/1848/1849** (config pushes now recorded, an `hlimits` leak, a benign picker WARN quieted).
|
||
- **`ca4db13a`** — **bug-1851**: string resvals now own their bytes; a rejoin use-after-free was killing a newly promoted primary during a sequential HA soak.
|
||
- **`9c02184f`, `d9ff02cd`, `5b891b50`** — **bug-1852** (scheduler records events before the core applies them), a scheduler replay-parity gate added to the sequential/chaotic soaks, and **bug-1853** (a chaotic-soak `walcompare` false divergence — a harness bug, not a safety bug).
|
||
- **`19104f19`, `e915f8de`, `b0f2aea4`** — **buglog `status` field**: backfilled on all 1,193 entries, with the 85 whose text said not-fixed/pending checked against HEAD. Also a `buglog` skill (`scripts/buglog.py`), the `.claude/rules/buglog-status.md` convention, and a status-driven `gitea-sync` that fixes **bug-1855** (gitea-sync closed any issue with a non-empty `fix` text, including "NOT FIXED" ones). The first sync wrote Gitea milestone/issue IDs into 65 `spec.json` files.
|
||
- **`3cc2f6e3`, `9e9f2798`** — **bug-1857** (`xce-test-multihost` now deploys a fixture config that clears limits) and **bug-1858** (ASAN invalid free: release an interned queue resreq with dereference, not `free`).
|
||
- **`6e0463e7`** — new spec `libxce-alloc-preason-equality` (tasks generated, 0/8).
|
||
|
||
### Fri 09-25 (Friday) — allocation backlog cleared, scheduler-allocation complete
|
||
- **`1ecd348f`, `b4e9a099`, `aea34738`, `601b245e`** — **bug-1769/1771** (ALTER allocation type confusion; NULL queue in the legacy fcfs hooks), **bug-1768** (slots/rusage checks for primaries nested in COMPOUND/ALTER), **bug-1863** (a COMPOUND with 2+ primaries kept only the last one's allocation) and **bug-1766** (a failed COMPOUND left a partial allocation behind). **`4ba0f327`** then marked `scheduler-allocation` implementation-complete (22/22).
|
||
- **`b5d59253`, `7c98c056`, `c29518b2`** — **bug-1429** (a failed test hung the whole unit suite), **bug-1861** (`mkstemp` for the config-upgrade temp file), **bug-1859** (a failover outage no longer expires a finished job's retained outcome) and **bug-1862** (the worker list is locked against vtask-thread readers; ASAN UAF in `xcecd_virtual`).
|
||
- **`0d3bb965`, `910b43cd`, `6b1862f8`, `663ea63c`** — **bug-1864** (the vtask dispatcher now keeps per-endpoint message order; standbys were NACKing in-order catch-up), **bug-1865** (never `close()` the primary channel's raw fd), **bug-1850** (libxce retries a NOT_READY controller in place instead of failing over — the root trigger of bug-1846's mid-round rejoins) and **bug-1854** (re-pushed resource definitions are not errors). All were soak-verified (`d87b78dc`).
|
||
- **`1ba643b8`** — **bug-1714** and **bug-1840** set to cannot-reproduce on current builds; **bug-1866** logged (LeakSanitizer: `remove_worker` never frees `launch_type`).
|
||
|
||
### Sat 09-26 (Saturday) — persistence field-completeness + worker `/tmp` hardening
|
||
- **`6876e4f6`, `1d6c04c2`, `e4bb492a`** — `WAL_JOB_FAIL` now carries the exit code (**bug-1754**) and the worker-reported finish time (**bug-1747** FAIL half, via `record_job_fail_fact_at()`), and checkpoints keep `Job.exitInfo` in the capnp schema (**bug-1867**). These close the save/restore field-completeness gaps `persistence.md` calls out.
|
||
- **`63fd3d2a`** — **bug-1870**: checkpoints no longer punch a hole in the replicated log (the checkpoint marker was never replicated). Found because the force-snapshot soak could not submit enough jobs to trigger a checkpoint.
|
||
- **`7723a4f4`, `7864a3da`, `5df33e69`** — worker hardening: `fetchfile` only reads the job's own output file (**bug-1508**), and neither a job's output file (**bug-1868**) nor any other fixed root-owned `/tmp` name (**bug-1869**) follows a planted symlink.
|
||
- **`ff4e2cb8`, `21e72726`** — `gitea-sync --ids`; bugs synced to Gitea #264–#289.
|
||
|
||
## Bugs opened/fixed this window
|
||
|
||
- **41 new bugs** (bug-1830 – bug-1870; no ID collisions — the duplicate-ID census is now **0**): **37 fixed**, **2 cannot-reproduce** (bug-1830 — framing corrected by the user's retest; bug-1840 — scheduler SIGABRT, no ASAN repro, strongest candidate fixed via bug-1769), **2 open** (bug-1856 — an upstream OpenWolf duplicate-ID hook defect, mitigated locally by `openwolf.buglog.auto_detect=false`; bug-1866 — `xcecd_virtual` leak at exit).
|
||
- **8 older bugs fixed** in the window: bug-1429, bug-1508, bug-1747, bug-1754, bug-1766, bug-1768, bug-1769, bug-1771. bug-1714 moved to cannot-reproduce.
|
||
- **Carry-over status from last week's report**:
|
||
- Buglog integrity (the Sep 19 mass retag) — **resolved structurally**: `status` is now the single source of truth, and tags/`fix` text are explicitly not status signals. One loose end: **bug-1811** reads `fixed` with no `status_note`, although last week's report recorded its fix as reverted and reclassified as a design decision.
|
||
- Gitea sync backlog (32 bugs) — **resolved**: status-driven `gitea-sync` pushed #264–#289. (The `gitea` MCP server still fails to connect; the sync script works without it.)
|
||
- **bug-1714** → cannot-reproduce (Sep 25). **bug-1726** → cannot-reproduce (Sep 24 triage). **bug-1632** — still **open**, 4th window. **bug-1728** — still open at window close; fixed Sep 27 (`33c4f28c`, next window).
|
||
- **bug-1777/1793/1796/1797** — still **open**, now explicitly tracked as such; 4th window unowned.
|
||
- **bug-1824** — still open (deferred).
|
||
- Highest-severity fixes: **bug-1835/1839/1851** — three live HA-cluster controller SIGSEGVs, all fixed within a day of being found. **bug-1864/1865** — standby catch-up NACKs and a WAL fd closed out from under a standby at failover. **bug-1870** — a hole in the replicated log at every checkpoint. **bug-1868/1869/1508** — root-owned worker file opens following planted symlinks or reading arbitrary paths (security-relevant).
|
||
|
||
## Spec status changes
|
||
|
||
- **`scheduler-sim-diagnostics`** — Requirement 6 (human-readable units) was built and validated end to end on Sunday (tasks 7–10), so the whole spec is now functionally complete. `tasks.md` shows 17/19 because the parent headers for tasks 8 and 10 are unchecked while their subtasks are done (the same convention gap as last week). `spec.json` still reads `tasks-generated`.
|
||
- **`scheduler-allocation`** — marked `implementation-complete`, 22/22 (`4ba0f327`, Fri), after the COMPOUND/ALTER bug sweep.
|
||
- **`controller-peers-env-fallback`** — new; `design-generated`, not ready for implementation (Wed).
|
||
- **`libxce-alloc-preason-equality`** — new; `tasks-generated`, 0/8, `ready_for_implementation: false` (Thu).
|
||
- **Metadata lag — not addressed, now 4 specs**: `controller-status` (29/30, both deltas GO), `scheduler-sim-live-allocation` (14/19, `ready: false` though the spec shipped), `xce-test-cli-ux` (23/27, 3rd window) and now `scheduler-sim-diagnostics` all still read `tasks-generated`.
|
||
- 65 `spec.json` files gained a `gitea` block (milestone and issue IDs) from the first status-driven sync. This is bookkeeping, not phase changes.
|
||
|
||
## Stats
|
||
|
||
- **82 commits** — **56 touching at least one non-`.wolf` file**, **26 `.wolf`-only** (mostly `chore(buglog)` status updates plus OpenWolf syncs).
|
||
- **347 non-`.wolf` file changes**, **+14,670 / −1,200** (non-wolf headline; raw incl. `.wolf`: +20,285 / −5,774).
|
||
- Busiest areas by distinct files touched: `.kiro` (80 — 65 of them from the one Gitea-ID commit), `src/scheduler` (67), `src/controller` (29), `src/lib` (17), `src/worker` (16), `src/include` (11), `.claude` (10).
|
||
- By commit scope: `buglog` (24), `scheduler` (17), `controller` (10), `scheduler-sim-diagnostics` (6), `worker` (5), `xce-test-ha` (3).
|
||
- **1 spec completed** (`scheduler-allocation`), **1 requirement delta finished** (`scheduler-sim-diagnostics` Req 6), **2 new specs** started.
|
||
- **11 sessions** logged in `.wolf/memory.md` (Sep 21 ×2, Sep 22 ×2, Sep 23 ×1, Sep 24 ×5, Sep 26 ×1). The busiest logged day was **Sep 21** (156 actions). Sep 20 and Sep 25 have commits but no session headers.
|
||
- Commits landed on all 7 days. The peak day was **Fri 09-25** (25 commits, 13 of them fixes), followed by **Thu 09-24** (18) and **Sat 09-26** (17).
|
||
- Everything is on `scheduler-framework` / `origin/scheduler-framework`; 55 of the 82 are also on `for_test`.
|
||
|
||
## Open threads / to-do heading into next session
|
||
|
||
1. **HA / async-I/O group landed just after the window closed** (Sep 27 UTC, `7aac018c`..`c21ca0b7`, pushed): bug-1450/1451/1728/1433/1474/1469/1741, a bug-1870 follow-up (`load_snapshot` resumed at the checkpoint instead of checkpoint+1, `cb1415d7`) and opcode names for tracing. Next week's report covers these. The one new thread is **bug-1871** (investigating): an HA snapshot transfer stalls 3+ s mid-stream on a rejoining standby, seen in 2 of 4 force-snapshot runs; suspected edge-triggered epoll not re-signalling data buffered while the endpoint was blocked.
|
||
2. **Spec metadata sync, now 4 specs** — `controller-status`, `scheduler-sim-live-allocation`, `xce-test-cli-ux`, `scheduler-sim-diagnostics` all read `tasks-generated` despite being functionally complete. One batch phase-sync pass (plus flipping the unchecked parent headers) clears them all.
|
||
3. **Gitea is behind the buglog again**: bug-1450, 1451, 1744, 1860 and 1867 are closed locally but open/partial on Gitea, and bug-1847–1849, 1851–1853, 1870 and 1871 have no Gitea issue yet. Run `gitea-sync`.
|
||
4. **`xce_test` harness batch**: bug-1866 (`remove_worker` leaks `launch_type`) plus bug-1777/1793/1796/1797 (stale join maxswap/maxtmp defaults, workerlist ignoring the `hosts.yaml` override, stale `perror` errno, `cmd_qctrl` leak) — all small, all in `src/harness`/worker, 4th window unowned.
|
||
5. **bug-1632** (ASAN stack-use-after-return in `test_sim_verify.c`) — open for a 4th window. **bug-1824** (`repl_info.c` bounds check) is still deferred. Both need a fix-or-wontfix decision.
|
||
6. **bug-1811** — set its status to `wontfix` (or add a `status_note`) so it matches last week's record of a reverted fix reclassified as a design decision.
|
||
7. **New specs awaiting approval**: `controller-peers-env-fallback` (design review) and `libxce-alloc-preason-equality` (tasks review, `ready_for_implementation: false`).
|