myxce/reports/jul27-aug02-2026.md

# XCE Project Status — Jul 27 – Aug 01, 2026

## TL;DR

HA/Raft correctness work dominated the week: three permanent-wedge bugs closed (catch-up stall, quorum-loss deadlock, config-push race/UAF), `raft-adapter-conformance` spec landed complete, `admin-config-tool` and `config-upgrade-snapshot` both unblocked and near done. One exposed credential fixed. Scheduler work restarted: `scheduler-framework` decomposed into three child specs (`scheduler-core`, `scheduler-allocation`, `scheduler-preemption`), all currently at requirements-only (not yet approved for design).

## Commits by day

**Jul 27**
- `add report`, merge `working`→`config`, fix librtti package leak

**Jul 28** — HA wedge fixes
- fix(ha): close permanent catch-up wedge from duplicated replication-target formula (bug-1478)
- fix(ha): close permanent standby catch-up wedge after WAL_CHECKPOINT boundary (bug-1475)
- fix(config): serialize `load_primary_config` against concurrent invocations (bug-1476)
- fix(ha): retry `cluster_init_phase2` on failure instead of latching `phase2_done` unconditionally (bug-1477)
- docs: re-block config-upgrade-snapshot task 8.6 on bug-1477, then clear stale admin-config-tool blocker (bug-1468 already fixed)

**Jul 29** — raft-adapter-conformance build-out
- feat: term-adoption wired into primary-side config-push reply handler
- feat: `term` field added to `ReplConfigPushReply` schema
- feat: explicit reply on every config-push outcome
- feat: route post-connect vote/PreVote failures to production tally
- fix(ha): close permanent quorum-loss deadlock on log-conflicted standby (bug-1479)
- test: PreVote post-connect failure routing + fast-quorum coverage; full config-push reply integrity; dormant AppendEntries sender non-regression; WAL catch-up reject reply-construction lock
- chore(wolf): logged bug-1480, bug-1481, bug-1482, bug-1483 (audit findings, not all fixed yet)

**Jul 30** — conformance close-out + admin-config-tool E2E
- test(admin-config-tool): live single-node E2E test for config push
- fix(admin-config-tool): unwrap Request reply envelope before decoding as RequestReply
- docs(admin-config-tool): task 6.3 blocked pending flake reproduction (bug-1485)
- test(raft-adapter-conformance): batch payload persist-failure propagation regression lock
- docs: confirmed live soaks for tasks 7.1–7.4 (vote/PreVote post-connect failure, config-push rejection); extended controller conformance record; logged bug-1484

**Jul 31**
- fix(ha): retry chaotic-mode `verify_classify` before counting job loss (bug-1487) — false-positive job-loss fixed in soak harness, not controller
- fix(security): replace exposed `rshen/Letmein` admin credential in `auth.yaml` with placeholder; added `src/docs/deployment.md` full ops guide
- fix(waldump): decode `WAL_CONFIG_MARKER`, distinguish trimmed gaps from real loss
- docs(config-upgrade-snapshot): host admin-state frames, capnp migration, **bug-1418 blocker cleared**
- docs(raft-adapter-conformance): final design.md/research.md + spec.json state committed — **spec complete**
- **add scheduler related projects** — `scheduler-framework` decomposed into `scheduler-core` (1311-line requirements.md), `scheduler-allocation` (619-line), `scheduler-preemption` (109-line); all requirements-generated, none approved yet
- add wolf state (session memory checkpoint)

**Aug 01** — no commits yet; working tree has only uncommitted `.wolf/` session-state churn (memory.md, token-ledger.json, hooks/_session.json).

## Bugs opened/fixed this window

| Bug | Status | Summary |
|---|---|---|
| bug-1475 | **Fixed** (2026-07-27/28, 2 findings) | Standby catch-up permanently wedged across a WAL_CHECKPOINT boundary — 3-part fix (config_repl.c prev-tracker, ha.c JIT resync, persist.c `TERM_INDEX_UNSET` sentinel) plus a second snapshot-install variant of the same class, closed with the same sentinel |
| bug-1476 | **Fixed** (2026-07-28, re-fixed same day) | Heap-use-after-free race in `load_primary_config` — first landed fix left the lock un-acquired (RED), re-fixed for real with `myconf->lock` + decode-into-local-then-swap |
| bug-1477 | **Fixed** | `cluster_init_phase2` latched `phase2_done` on failure, blocking HA bootstrap retry |
| bug-1478 | **Fixed** | Duplicated replication-target formula caused permanent catch-up wedge |
| bug-1479 | **Fixed** | Permanent quorum-loss deadlock on log-conflicted standby |
| bug-1480 | **Fixed** (2026-07-29 09:43, `b7828eed`) | Config-push reply missing explicit reject+term — fixed same day, buglog `fix:` field was stale (written 07:05, before landing) until corrected 2026-08-01 |
| bug-1481 | **Fixed** (2026-07-29 13:53, `fce73b5b`) | `vote_ep_err_handler` bypassed production tally on post-connect failure — fixed same day, buglog `fix:` field was stale until corrected 2026-08-01 |
| bug-1482 | **Fixed** | Not a real UAF — erroneous `free_packet` call in a test (pkt already owned/consumed by `xce_decode`); test + doc comment corrected |
| bug-1483 | Open | Pre-existing test-mode leak in `ha_arm_heartbeat_timeout_vtask` (c==NULL calloc never freed); workaround only (stub+reset hook in every LOST-path test), not fixed at root |
| bug-1484 | Open, explicit residual | No test drives real `repl_ack_decode_and_dispatch` term-adoption for heartbeat-reply/WAL-NACK vectors; Req 7.5 "already verified" claim rests on unverified ground; out of scope for raft-adapter-conformance, needs a follow-up task |
| bug-1485 | Open, flaky, 24 occurrences | `test_live_config_push_against_primary_reaches_success_path` — admin-config-tool task 6.3 blocked on this |
| bug-1486 | Documented (workflow, no code fix) | Concurrent `xce-build.sh` sessions clobber shared build dir (rm -rf race) — mitigation: check `ps auxf` before trusting a stale background build log |
| bug-1487 | **Fixed** | 30-round chaotic soak false-positive job-loss — actually a 6.2s Figure-8 commit-barrier window, not real loss; fix adds bounded retry to `verify_classify` in the soak harness |

## Spec status changes

- **raft-adapter-conformance** — requirements/design/tasks all approved, `ready_for_implementation`, `blocked_on: []`. Final design.md/research.md committed Jul 31. Effectively **done**.
- **admin-config-tool** — both historical blockers (bug-1468 dispatch-inversion, bug-1461 capnp `Data` shadowing) now marked RESOLVED; task 6.3 HA E2E test still blocked on a flaky-test repro (bug-1485).
- **config-upgrade-snapshot** — `blocked_on` cleared to `[]` (bug-1418 confirmed fixed, no regression); host admin-state frames and capnp migration work added.
- **scheduler-framework** — formally decomposed (phase: `decomposed`) into three child specs, all at `requirements-generated` / not approved:
  - `scheduler-core` — 1311-line requirements.md
  - `scheduler-allocation` — 619-line requirements.md
  - `scheduler-preemption` — 109-line requirements.md
  - Next step for this line of work is design approval on whichever child spec proceeds first.

## Notable non-bug items

- **Security**: a live, checked-in `cluster_admin` credential (`rshen/Letmein`) in `auth.yaml` was replaced with a placeholder — flag for credential rotation if that value was ever deployed anywhere real.
- **New doc**: `src/docs/deployment.md` — full ops guide for building/running all four daemons + xceadmin, written from direct source verification + a live single-node deploy; explicitly documents production gaps (no process supervision, plaintext non-controller traffic, exposed dev creds).
- **waldump**: now decodes `WAL_CONFIG_MARKER`, so trimmed-segment gaps are no longer misreported as real data loss during investigations.

## Open threads heading into next session

1. `scheduler-core`/`scheduler-allocation`/`scheduler-preemption` need requirements review + design phase — nothing is implementation-ready yet.
2. bug-1480/1481/1482/1483/1484/1485 remain open (raft-adapter-conformance audit residue + one ASan UAF in netio.c).
3. Uncommitted `.wolf/` session-state changes sitting in working tree (harmless, not code).