111 lines
14 KiB
Markdown
111 lines
14 KiB
Markdown
# XCE Project Status — Aug 23 – Aug 29, 2026
|
||
|
||
> Note: the calendar-default window (Aug 17–22) is already covered by
|
||
> `aug16-aug22-2026.md`; this report covers the actual gap — the last
|
||
> complete day before this report was generated (Aug 30 is in progress).
|
||
|
||
## TL;DR
|
||
|
||
An unusually dense week: 97 real commits shipped two specs end-to-end
|
||
(`scheduler-dispatch-commit`, `xce-test-rc-file`), drove
|
||
`scheduler-sim-eager-reconciliation` and `scheduler-snapshot-job-allocation`
|
||
to substantial completion, and closed out a systematic five-bug sweep of
|
||
WAL-replay field-completeness gaps alongside several safety-critical HA
|
||
correctness fixes (vote-denial on persist failure, stale-term
|
||
re-election blocking, fail-loud disconnect handling). Last week's #1
|
||
open thread — 39 duplicate buglog IDs — was fully repaired on Monday.
|
||
The week closed with a deep multi-session live-debugging investigation
|
||
that fixed three confirmed production bugs (a scheduler resource leak,
|
||
a controller heap-use-after-free that crashed 2 of 3 controllers under
|
||
load, and a set of permanently-dead job-notification hooks that could
|
||
leave a job's state stuck as RUN forever in the scheduler and apiserver)
|
||
and made major forensic progress on a fourth, still-open scheduler crash
|
||
(bug-1714) — proved it's a genuine cross-thread use-after-free, not the
|
||
overflow first suspected, but the exact stale-pointer source remains
|
||
unisolated.
|
||
|
||
## Commits by day
|
||
|
||
### Sun 08-23 — `scheduler-dispatch-commit` spec, start to finish in one day
|
||
|
||
- **`5724d1e9`→`a9cfee5f`** (13 commits) — dispatch-failure classification, a durable `WAL_JOB_DISPATCH` fact type, scheduler transitions to RUN at commit time, controller-side retryable/terminal dispatch-failure resolution with bounded escalation, and full multi-round + terminal + retryable e2e coverage. Spec marked implementation-complete (`f784e2bd`).
|
||
|
||
### Mon 08-24 — Gitea issue triage and buglog integrity repair
|
||
|
||
- **`1c9cfc1f`, `e83d207a`** — reproduced and logged 4 Gitea issues (#40–42, #44) plus a `gitea-sync` bug found while doing it; corrected an earlier false-negative on #43 (bug-1678 — turned out reproducible).
|
||
- **`db142d41`→`88c7c05d`** — fixed the 4 reproduced issues: dispatch-time resource allocation not persisted into `WAL_JOB_DISPATCH` (bug-1678), live job-status pushes to the apiserver never firing (bug-1679), `jobs list -l/-o` validating against the wrong struct (bug-1674), unmatched workers not falling back to a default `HostConfig` (bug-1673) — plus `from_host` threaded end-to-end through the wire format, CGO layer, and job-submit interning object (bug-1676).
|
||
- **`755976fc`** — **repaired last week's #1 open thread**: renumbered 40 buglog entries that had collided with `bug-875`–`bug-949` after the Aug 13–17 counter restart.
|
||
- **`5c82ad0b`** — **repaired last week's #2 open thread**: synced `spec.json` phase metadata for `build-time-versioning`/`dirty-content-fingerprint` (both were 21/21 done but stuck on `tasks-generated`) and flipped 23 stale section-header checkboxes across 3 more specs whose numbered subtasks were already complete.
|
||
|
||
### Tue 08-25 — peak day (46 commits): WAL-replay field-completeness sweep + HA correctness
|
||
|
||
- **`955efecd`→`d0e5f85a`** — five-bug systematic sweep: `WAL_JOB_START` losing pidinfo/start_time (bug-1687), `WAL_JOB_FINISH` losing exit_info/finish_time — and exit_info was never set in production at all (bug-1689), `WAL_JOB_RUSAGE` losing run_usage (bug-1690), `WAL_JOB_CKPT` losing ckptinfo (bug-1691), `WAL_JOB_PREASON` losing pend_reasons (bug-1692) — all on restart replay.
|
||
- **`e5e6cd18`** — **`raft_cb_persist` must deny the vote and roll back state on a persist failure** (bug-1684): `ha_persist_term()`'s every open/write/fsync/rename failure was silently swallowed; the raft library always granted the vote regardless. Safety-critical.
|
||
- **`3eba7c16`** — fail loud on scheduler/worker disconnect instead of relying on the reply-wait timeout (bug-1693).
|
||
- **`b4eb9ec8`** — offload `checkpoint_task_fn` off the delaymgr thread (bug-1694, async-I/O blocking-violation).
|
||
- **`ac46bfa6`** — restored-but-stale raft term permanently blocked re-election after restart (bug-1695), found via the mandatory live HA chaotic soak for this session's controller audit campaign.
|
||
- **`b040dfc0`, `bc3f76b4`, `6400f96d`** — removed a double-free in the limit plugin's config update (bug-1685), invalidated stale `builtin_fcfs` picker cursors across calls (bug-1686), mirrored `plugin_begin_session`/`plugin_end_session` into the sim-mode driver (bug-1681).
|
||
- **`9a230f08`** — apiserver `FilterRows` routing non-bitmask operators through bitmask matching (bug-1680, found while verifying bug-1679).
|
||
|
||
### Wed 08-26 — `xce-test-rc-file` spec
|
||
|
||
- **`9de4db95`→`8fc86ef4`** (7 commits) — rc-file path resolution with XDG precedence, `--norc` flag, pluggable rc-line-loading runner wired into `xce_test` startup, and full e2e coverage for discovery/loading/`--norc`/failure-handling, confirmed not to disturb existing automation.
|
||
|
||
### Thu 08-27 — `scheduler-sim-eager-reconciliation` spec + a fix batch + `scheduler-snapshot-job-allocation` kickoff
|
||
|
||
- **`e2bb98c8`→`446437bf`** (6 commits) — state-independent dispatch capture, resInUse deferral proof, capture-safety unit coverage, resource-contention/sim-verify/full regression proofs. 6/9 tasks done.
|
||
- **`f886c809`** — standby dispatch-gate rejections now redirect to a known primary (bug-1700, bug-1701) — closes a gap `admin-config-tool`'s own design doc had flagged as a documented blocker.
|
||
- **`ac1e312e`, `0ebf3a81`, `47532f0a`, `cfd7db8c`** — `job.start_time` never set due to a dispatch/`WAL_JOB_START` RUN-state collision (bug-1697); repeated `WORKERS_PUSH` with a NULL resval clobbering good values (bug-1688); first-boot promotion evaluating eligibility before its own WAL recovery ran (bug-1696); xceadmin retrying transient scheduler-admin failures after a redirect (bug-1713, bug-1715).
|
||
- **`097e33ae`→`22353849`** — `scheduler-snapshot-job-allocation` begins: `dup_job_alloc` deep-copy primitive, schedobj construction for restored jobs.
|
||
|
||
### Fri 08-28 — `scheduler-snapshot-job-allocation` completion + a heap-overflow fix
|
||
|
||
- **`6fa2d970`→`5948e079`** — allocs populated on snapshot save, restored on load, full cross-instance round-trip proof, backward-compat for allocation-less snapshots, allocator-crash regression test, and the final invariant-audit follow-up closed. Spec docs mark it feature-complete.
|
||
- **`00207503`** — **heap buffer overflow in `make_job_account()` for multi-host jobs** (bug-1722): the function `calloc`'d room for exactly 1 host account pointer but its own loop wrote up to `nhosts-1` — any job spanning more than one host overflowed the allocation.
|
||
|
||
### Sat 08-29 — live-debugging investigation: 4 bugs found, 3 fixed, 1 deeply diagnosed
|
||
|
||
- **`b1323f8d`** — scheduler leaked `job->accounts`/`job->allocs` on a `JOB_UPDATE` terminal transition (bug-1730).
|
||
- **`42e1eeaa`** — **heap-use-after-free in `vtask_enq()`** (bug-1733): a task was published to the dispatch queue before its own `owner` field finished initializing; a concurrent worker could dequeue, run, and free it before the producer's own write executed. ASAN-confirmed, crashed 2 of 3 controllers under a live burst-array-job HA workload.
|
||
- **`ebe691c3`, `67009815`** — `apply_committed_entry()` only ever fired the generic `HOOK_DETAIL_JOB_UPDATE` for job-state changes; `HOOK_DETAIL_JOB_START`/`JOB_EXIT`/`JOB_FAIL` were permanently dead code, leaving the Go apiserver's already-wired `OnJobFinished` handler starved of data — a finished (or failed) job's state could stay stuck as RUN in the scheduler's own view forever (bug-1731). Live-verified end to end on both the scheduler and apiserver sides.
|
||
- **`83cb660b`** — `xce-build.sh`'s `worker` target never built `xcecd_virtual`, so deploy tooling silently skipped generating the worker-start script in sandboxes with no real systemd manager (bug-1732).
|
||
- **`347e996e`, `14f95b06`** — worker now distinguishes signal-killed job exits from genuine crashes in its logs; scheduler snapshot load re-derives worker/shared resource "used" totals instead of leaving them at zero until the first session.
|
||
- **`6193fa7e`** — `test_e2e_live_ha_push.py`'s standby-redirect test rewritten to assert the new success behavior (was asserting the old, now-fixed rejection behavior).
|
||
- **Not landed as a commit, but the week's deepest investigation**: `bug-1714` (a scheduler `SIGSEGV` in `sched_get_project`, first reported live days earlier) was reproduced twice more this session — once under ASAN (no shadow-memory violation fired), once via a scripted gdb watchpoint that caught the actual corrupting write: freed `job_submit_data_t` memory being recycled by malloc for an unrelated Cap'n Proto decode buffer on a different thread. The free itself was verified legitimate by static reading — the bug is an unidentified second/stale reader elsewhere. Two related side-bugs found during the chase (an orphaned duplicate job silently handed to the picker, and the dedup-race that made it reachable) were fixed and confirmed independent of bug-1714's own root cause, which remains open.
|
||
|
||
## Bugs opened/fixed this window
|
||
|
||
- **69 genuinely new bugs** (bug-1625 – bug-1733), **66 fixed** across the window's 75 touched entries, **9 still open** at window end. Zero ID collisions — the buglog counter stayed clean all week.
|
||
- **Both of last week's open threads are resolved**: the 39 duplicate IDs were repaired Monday (`755976fc`); `build-time-versioning`/`dirty-content-fingerprint`'s phase-metadata lag was fixed the same day (`5c82ad0b`).
|
||
- Highest-severity fixes: **bug-1722** (heap buffer overflow in `make_job_account()` for any multi-host job — memory corruption, not just a crash), **bug-1733** (controller UAF crashing 2/3 nodes under live HA load), **bug-1684** (raft would grant a vote even when its own persist call failed — HA safety-critical), **bug-1695** (a restored-but-stale raft term could permanently block re-election after restart).
|
||
- Still open at window end (carry-over risk into next week):
|
||
- **bug-1714** — scheduler UAF crash, mechanism proven, stale-reader source not yet isolated (highest priority — this is the week's one unresolved thread with active investigation momentum).
|
||
- **bug-1429** — `controller_unit_test` 300s TIMEOUT, deadlock-suspected, open since Jul 10, no owner.
|
||
- **bug-1560** — `test_recordstart_symlink_escape_replies_error`, a known sandbox filesystem-permissions flake, open since Aug 15.
|
||
- **bug-1630, bug-1632, bug-1723, bug-1726, bug-1727, bug-1728** — six newer, lower-priority findings (a pre-existing ASAN flake in the sim driver, a test-fixture memory leak, a flaky HA e2e test distinct from bug-1485, a REPL resource-display bug, and a false-ERROR log severity on worker join) — none blocking, none investigated further this week.
|
||
- **Note on bug-1630 specifically**: its own `fix` field is still empty in the buglog, but the underlying problem it describes (the 39 duplicate IDs) is verifiably resolved — `755976fc`'s renumbering left **zero duplicate IDs** in the current file (re-verified directly). The entry itself just never got marked fixed; worth a one-line buglog update, not further investigation.
|
||
|
||
## Spec status changes
|
||
|
||
- **`scheduler-dispatch-commit`** — went 0→implementation-complete in one day (Sun); `spec.json` still reads 13/18 tasks despite the phase — metadata lag, not work lag (same pattern as last week's build-time-versioning).
|
||
- **`xce-test-rc-file`** — 7/10 tasks, phase `tasks-generated`.
|
||
- **`scheduler-sim-eager-reconciliation`** — 6/9 tasks, phase `tasks-generated`.
|
||
- **`scheduler-snapshot-job-allocation`** — driven to feature-complete per its own closing commit message, but `spec.json` phase still reads `tasks-generated` at 11/15 tasks — another phase/reality mismatch to sync.
|
||
- **`scheduler-config-version-tag`** — new spec created (requirements/design/research/tasks all approved), 0/8 tasks — not started.
|
||
- **`admin-config-tool`** — no new work, but its own design doc's last documented blocker (the standby-redirect gap) was closed this week by `f886c809`.
|
||
|
||
## Stats
|
||
|
||
- **164 commits** in the window — **97 real** (feature/fix/docs/test), **67 `.wolf` bookkeeping/checkpoint commits**.
|
||
- **341 non-`.wolf` files changed**, **+26,205 / −1,002** lines (excluding `.wolf/`; including it the raw figure is +220,722 / −1,049,394, dominated by buglog renumbering and token-ledger churn).
|
||
- Busiest areas: `src/controller` (46 files), `.kiro` (42), `src/scheduler` (35), `.wolf` (25), `src/frontend` (16).
|
||
- **2 specs shipped end-to-end** (`scheduler-dispatch-commit`, `xce-test-rc-file`), **2 more substantially advanced** (`scheduler-sim-eager-reconciliation`, `scheduler-snapshot-job-allocation`), **1 new spec created**.
|
||
- **18 sessions** logged in `.wolf/memory.md`, single busiest day Aug 23 (383 logged actions).
|
||
- Commits landed on **all 7 days**; peak day **Tue 08-25** (46 commits — the WAL-replay + HA sweep).
|
||
|
||
## Open threads / to-do heading into next session
|
||
|
||
1. **Isolate bug-1714's stale-reader.** The UAF mechanism is proven (freed `job_submit_data_t` recycled across threads); the actual second reader that still holds a pointer past the job's legitimate free is not yet found. Two leads already ruled unlikely or weak by this week's own static reading — needs either deeper tracing of the pending-update queue lifecycle or another live session with pointer-tracking instrumentation.
|
||
2. **Sync `scheduler-dispatch-commit` and `scheduler-snapshot-job-allocation`'s `spec.json` phase metadata** — both report a phase behind their actual completion state, the same lag class fixed for two other specs this Monday.
|
||
3. **Five lower-priority open bugs** (1632, 1723, 1726, 1727, 1728) have no assigned follow-up yet — worth a triage pass to confirm none has quietly become higher-priority. (bug-1630 is housekeeping only — see the note above; just needs its buglog entry marked fixed.)
|
||
4. **bug-1429** (controller test-suite deadlock-suspected timeout) has been open since Jul 10 with no owner — the oldest unresolved item in the log.
|