docs: add weekly status report (Aug 8-15, 2026)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

reports/aug08-aug15-2026.md

@@ -0,0 +1,111 @@
+# XCE Project Status — Aug 08 – Aug 15, 2026
+
+## TL;DR
+
+**component-failover-adoption: implementation-complete and GO-validated — but not yet committed.** The week opened with scheduler-core documentation cleanup (carried over from last week), then a worker CLI/config cleanup pass, then the bulk of the week (Aug 11–13) was the 21-task component-failover-adoption spec: unifying worker/scheduler/apiserver's dual connect paths into one candidate-list pipeline, adding rejoin-on-failover for all three components, and an apiserver backend-health circuit breaker. Live E2E validation during that work found two real production bugs (bug-1548/bug-1549: apiserver never rejoined a new primary during an idle/heartbeat-only failover, so job pushes silently stopped reaching it). `/kiro-validate-impl` caught this as an in-scope gap and returned **NO-GO** on Aug 15 at 13:10 UTC; a same-day Phase 6 remediation (a monotonic `reconnect_gen` counter) fixed it, live-reverified, and the decision flipped to **GO** at 14:20 UTC. That fix, plus a new `xceadmin` cluster-config dump feature and a batch of build-tooling fixes, are sitting validated-but-uncommitted in the working tree as of end of week — 57 modified/untracked files, no commits since Aug 13.
+
+## Commits by day
+
+### Aug 08 (Sat) — scheduler-core documentation cleanup (carried over from prior week)
+
+5 commits, docs-only:
+- `docs:` add scheduler recording/simulation/REPL to deployment guide
+- `docs(scheduler-core):` sync CLAUDE.md for core/driver/picker/sim/REPL architecture
+- `docs(scheduler-core):` commit accumulated brief/requirements/roadmap edits
+- `docs(scheduler-core):` backfill `_Verified` notes for tasks 2.1–2.6
+- `docs(scheduler-core):` commit design.md/research.md, sync design.md with as-built code
+
+### Aug 09 (Sun) — no commits
+
+### Aug 10 (Mon) — worker CLI/config cleanup + probeConfigTag fix
+
+11 commits:
+- **Worker cleanup:** removed the config-file mechanism and deprecated CLI flags entirely (`5e4baa27`, `f0f46f87`, `b3124767`), fixed `xcecd -h` exiting before root/config checks (`9ffb1d51`, **bug-1509** class), default logdir to `/tmp` when `-l` omitted (`e959ee61`), added `-v` verbose flag (`0df1cf5f`)
+- **fix(xceadmin)/feat(controller):** `OPCode_probeConfigTag` handler with Request-envelope reply, and `probe_config` updated to match (`90d7dd82`, `b00e97ff` — **bug-1506, bug-1507**)
+- `feat(scheduler):` working default record-dir, improved REPL output, job ID formatting (`c705cada`)
+
+### Aug 11 (Tue) — component-failover-adoption: foundation + worker + scheduler connect unification
+
+11 commits — Task groups 1–3 begin:
+- **Group 1 (Foundation):** reconciled `N_MAX_MASTERS`/`MAX_HA_CANDIDATES` (`fe8b4b86`), added the candidate-list→client-config conversion helper (`210faa2b`)
+- **Group 2 (Worker):** unified worker's dual connect paths into the single pipeline (`ce9f734a`), fixed startup ordering + added rejoin retry-on-failure (`331fa024`), validated thread isolation and fixed rejoin bugs found by live E2E (`619b119e`)
+- **Group 3 (Scheduler):** unified scheduler's dual connect paths (`a9f283dc`), added decision-publish retry-until-success-or-shutdown (`7c260e0c`), validated backpressure + fixed candidate-name diagnostics (`b7df8d60`), added decision-redelivery queue coverage (`d383d043`)
+
+### Aug 12 (Wed) — component-failover-adoption: apiserver connect unification + rejoin + circuit breaker
+
+7 commits — Task group 4:
+- Routed apiserver's candidate parsing through the real C parser (`8b1e13a5`)
+- Unified apiserver's dual connect paths — found and fixed a genuine use-after-free on `ctx->conf->masters[i]` during failover rotation (`e6359039`, **bug-1532**)
+- Added apiserver's rejoin logic with correctly-placed cgo registration (`2a205291`)
+- Wired rejoin to trigger from the idle-period heartbeat path (`ecf5eddc`)
+- Added the backend-health circuit breaker (`c8325e75`)
+- Closed an apiserver concurrency coverage gap (`79273a7f`)
+- Group 5 begins: full-cluster E2E reconnect-after-failover test passes (`6cb8f22b`) — **this run first surfaced bug-1548/bug-1549** (apiserver silently fails to rejoin on an idle/heartbeat-only failover)
+
+### Aug 13 (Thu) — component-failover-adoption closes; resource/config bugfixes; buglog integrity hardening
+
+15 commits (6 feature/fix, 9 wolf-buglog checkpoints):
+- Task 5.2 doubly-confirms bug-1548 in an isolated heartbeat-only scenario (`04d41d29`); task 5.3 validates the circuit breaker live (`51589b68`); task 5.4 closes the spec on a full regression pass (`d170623b`) — bug-1548/1549 were (incorrectly, see below) treated as out-of-scope follow-up items at this point
+- `chore(wolf):` buglog.json integrity guard, dedupe rule, auto-checkpoint hook (`ba4e5fd2`)
+- `fix(controller,xceadmin):` zero-slot scheduling block, misleading resval logs, byte-unit display (`e3b3ffaa` — **bug-1555, bug-1556**)
+- `refactor(controller,xceadmin):` dropped generic `Resource.unit`, hardcoded bytes for maxmem (`5bf123cc`)
+
+### Aug 14–15 (Fri–Sat) — validation reopened, Phase 6 remediation, tooling fixes — all uncommitted
+
+No commits landed, but substantial validated work sits in the working tree:
+- `/kiro-validate-impl` reopened component-failover-adoption: **NO-GO** (Aug 15, 13:10 UTC) — bug-1548 lives inside task 4.4's own `ApiserverBackendBridge` boundary against an already-existing acceptance criterion, so it doesn't qualify as legitimately out-of-scope
+- **Phase 6 remediation** (tasks 6.1/6.2): added a monotonic `reconnect_gen` counter to `endpoint_t` (`src/include/netio.h`, `src/lib/xcectx.c`), bumped on every successful transport reconnect; `heartbeat.go`'s `runHeartbeat()` replaced its failure-to-success transition detector with a per-tick generation comparison. New regression tests in `heartbeat_test.go`/`heartbeat_rejoin_test.go`. Live E2E re-run of the exact task-5.2 scenario passed: rejoin fired 22s after a primary kill (12s after election) with zero write traffic in the preceding 5.5 minutes; a post-recovery job submit correctly appeared via `GET /xce/v1/jobs`
+- Decision flipped to **GO** (14:20 UTC)
+- **bug-1558** fixed: `make test-backend` now self-derives `CGO_CFLAGS`/`CGO_LDFLAGS` via `pkg-config`
+- **bug-1559** fixed: `xce-build.sh`'s scheduler case now runs `meson test --suite scheduler` (all 41 registered targets, not just one binary by path)
+- **New, separate uncommitted feature:** `xceadmin` cluster-config dump/pretty-printer (`build.py`: `dump_cluster_config` + formatters, ~236 new lines) and a `parse_size_or_number` helper in `mapping.py` for size-string threshold parsing, with new tests (`test_cli.py`, `test_mapping.py`, untracked `test_config_dump.py`)
+
+## Bugs opened/fixed this window
+
+Named/significant bugs (55 total opened this window, `bug-1506`–`bug-1560`; most of the unlisted ones are terse review-catches — "significant refactor of X", "incorrect value in code" — logged during component-failover-adoption's two-round review process):
+
+| Bug | Status | Summary |
+|---|---|---|
+| bug-1506, bug-1507 | Fixed (`90d7dd82`, `b00e97ff`) | `probeConfigTag` missing Request-envelope handling; SIGABRT in REPL breakpoint inspection |
+| bug-1508 | **Open** | `handle_fetchfile()` — possible job-ownership/path-traversal gap found during doc audit; flagged, needs a security-reviewer pass |
+| bug-1509 | Fixed | `extralib` flake dependency removed; ported trivial linked-list code local to autoscaler |
+| bug-1514, bug-1515 | Fixed | design.md/requirements.md defects in component-failover-adoption's scheduler-backpressure description, found by `/kiro-validate-design` and a deep re-review |
+| bug-1519 | **Open** | AddressSanitizer "joining already joined thread" abort under `unshare -r`; worked around, not root-caused |
+| bug-1532 | Fixed (`e6359039`) | apiserver `tryConnect()` use-after-free on `ctx->conf->masters[i]` during failover rotation |
+| bug-1542 | Fixed | Illegal `import "C"` in a `_test.go` file; exposed the needed constant via a plain-Go package var instead |
+| **bug-1548** | Fixed (uncommitted, Phase 6) | apiserver never rejoined a new primary on an idle/heartbeat-only failover — root cause: `XCECLI_FF_CALL`'s silent inline reconnect was invisible to the old failure→success transition detector |
+| **bug-1549** | Fixed (uncommitted, Phase 6) | Downstream symptom of bug-1548 — job pushes never reached apiserver's local store after failover |
+| bug-1553 | **Open** | `controller_unit_test` segfaults intermittently (~1 in 3 runs); crash site not yet isolated |
+| bug-1554 | **Open, deferred by user** | `maxswap`/`maxtmp` report hardcoded stub values instead of real hardware detection |
+| bug-1555, bug-1556 | Fixed (`e3b3ffaa`) | Misleading `update_resval` error logs; zero-slot scheduling block on the documented `hosts.yaml` auto-detect default |
+| bug-1557 | **Open** | `xce_test worker join` defaults `maxmem=64` (bytes, not MB) |
+| bug-1558, bug-1559 | Fixed (uncommitted) | `make test-backend` CGO flags now self-derived via `pkg-config`; `xce-build.sh` scheduler suite now runs all 41 test targets |
+| bug-1560 | **Open, low priority** | `test_recordstart_symlink_escape_replies_error` fails under sandbox root — confirmed pre-existing, environment-specific, not a regression |
+
+## Spec status changes
+
+- **component-failover-adoption** — tasks 1.1–5.4 (21 tasks) implemented and initially marked complete (Aug 13). Reopened same week by its own `/kiro-validate-impl` run: **NO-GO** (Aug 15, 13:10 UTC) → Phase 6 remediation (2 tasks) → **GO** (Aug 15, 14:20 UTC). **Implementation and validation are done; the fix itself is not yet committed to git.**
+- **scheduler-core** — carried into this window only for Aug 08's documentation cleanup; no further status change.
+- Three new specs created this week, all still at brief/pre-requirements stage, not yet started: **component-rejoin-resync** and **worker-rejoin-resync** (both born from this week's bug-1548/1549 investigation — full-state resend cost on rejoin), and **scheduler-sim-diagnostics** (limit/pend-reason visibility in the REPL; not yet even `spec.json`-initialized).
+
+## Stats
+
+- **49 commits** in the window (39 feature/fix/docs, 10 wolf-infra/checkpoint)
+- **~124 files changed** across the week's commits (109,317 insertions / 18,170 deletions, dominated by component-failover-adoption)
+- **55 bugs opened** (bug-1506–bug-1560), **48 fixed**, **7 still open** (bug-1508, 1519, 1553, 1554, 1557, 1560, plus bug-1548/1549 fixed-but-uncommitted)
+- **1 spec closed and reopened same week**, remediated, now GO
+- **3 new specs** opened at brief stage
+- **0 commits on Aug 14–15** despite validated GO work landing — 57 files (modified + untracked) sitting uncommitted at week's end
+
+## Open threads / to-do heading into next session
+
+1. ~~**Commit the Phase 6 failover fix**~~ — **Done (2026-08-15, `d479b70`).** Verified via a full `lib asan` rebuild (all libxce tests incl. the new `test_netio_reconnect_gen` pass) and `make test-backend` (apiserver backend + auth suites pass) before committing. Scoped to `netio.h`, `xcectx.c`, `lib/meson.build`, `lib/testsuite/test.c`, `lib/testsuite/test_netio_reconnect_gen.c`, `heartbeat.go`, `clib.go`, `heartbeat_test.go`, `heartbeat_rejoin_test.go`, `spec.json`, `tasks.md`. Note: `design.md`/`requirements.md`/`research.md` were deliberately left out — they've been stale since the spec's very first commit (`4ad31d77`, never updated across all 21+2 tasks), so syncing them is a separate, broader doc-catch-up task, not part of the Phase 6 diff.
+2. ~~**Commit or explicitly shelve the xceadmin config-dump feature**~~ — **Done (2026-08-15, `3afe3d5`).** Verified via `xce-build.sh xceadmin`'s full pinned out-of-tree install + pytest run (256 passed, 3 skipped) before committing. Scoped to `build.py`, `cli.py`, `errors.py`, `mapping.py`, `hosts.schema.json`, `queues.schema.json` (the `SizeOrNumber` schema type is tightly coupled to `mapping.py`'s new `parse_size_or_number`, so it had to travel with it), plus the three test files. **New discovery while scoping this:** `resources.yaml`'s diff also contains unrelated changes — new resource catalog entries (`r15m`, `ut`, `pg`, `ls`, `it`, `swp`, `io`) that the worker (`resource_sync.c`) already registers but the shipped default `resources.yaml` never declared, plus `mem`/`tmp` byte-unit description fixes matching `node_information.c`'s uncommitted change (worker-reported `mem`/`tmp` now actually returns bytes, not MB) and `xcecd.c`'s `maxswap`/`maxtmp` stub-value unit fix (bug-1554's stub, now byte-scaled). These three files (`resources.yaml`, `worker/src/node_information.c`, `worker/src/xcecd.c`) were deliberately left out of both commits so far — added as new item 10 below.
+3. **`.wolf/dashboard-token`** is untracked, holds a live-looking token, and is not covered by `.gitignore` — add a gitignore rule before it lands in history via a broad `git add`.
+4. **Triage the remaining uncommitted files** not obviously tied to items 1, 2, or 10 (`deploy-ha.sh`, `xce-build.sh`, `controller/testsuite/test.c` — the `xce_test` detailed-help-text feature, unrelated to bug-1557's one-line default fix it happens to share a file with).
+5. **bug-1553** (intermittent controller test segfault) is the most concerning open item — a real memory-safety bug in the shared test binary, unisolated. Needs a bisect/gdb session.
+6. **bug-1508** (possible `handle_fetchfile()` ownership/path-traversal gap) — flagged during a doc audit, never given a security-reviewer pass.
+7. **bug-1557** — one-line fix (`maxmem` default should be `64 * 1024 * 1024`, not `64`), still open.
+8. **bug-1554** — swap/tmp-disk detection deferred by explicit user request; needs a design discussion when picked back up.
+9. Decide which of the three new brief-stage specs (component-rejoin-resync, worker-rejoin-resync, scheduler-sim-diagnostics) to move into `/kiro-spec-requirements` next.
+10. **New (found while scoping item 2):** commit the worker-side byte-unit completion + resource-catalog sync (`resources.yaml`, `src/worker/src/node_information.c`, `src/worker/src/xcecd.c`) — finishes what `5bf123cc`/`e3b3ffaa` started (mem/tmp now genuinely report bytes, not just documented as bytes) and adds the 7 missing built-in resource entries (`r15m`, `ut`, `pg`, `ls`, `it`, `swp`, `io`) the worker already reports but the catalog never declared. Worth its own scoped commit with a build+test pass first, same as items 1 and 2.