# Work Summary: Jul 19 – Jul 26, 2026

## Branch: `config` | Project: XCE Cluster Management System

---

## Overview

Week focused on three major workstreams: completing `clusterconfig-schema-split` (Jul 21), shipping `admin-config-tool` (Jul 22), and executing `config-upgrade-snapshot` tasks 8.0–8.9 (Jul 24–26). Two critical bugs discovered during soak testing (bug-1475 standby liveness stall, bug-1476 config-reload UAF) remain open. Two bugs fixed and committed (bug-1472 worker replication, bug-1474 repl slot clobber).

---

## Day-by-Day Summary

### Jul 19 (Fri)

- **bug-1452** fixed: `commit_engine_fini`/`persist_stop` teardown race in controller
- **xce-audit-invariants** controller pass completed
- **bug-1453/1454/1455/1456**: scheduler + xcecli dead CA-cert plumbing removed, scheduler HA config bugs fixed

### Jul 20 (Sat)

- Continued scheduler/xcecli cleanups from Jul 19
- Git: `94f50ce6` — fix(scheduler,cli): bug-1453–1456 remove dead CA-cert plumbing

### Jul 21 (Mon) — `clusterconfig-schema-split` ship day

- **All 12 tasks (1.1–6.2) completed**, spec marked implementation-complete
- Dead fields removed from `ClusterConfig` capnp schema and `cluster_config_t`
- Two-tier port fallback (`cluster_open_listener`), dead port defaulting removed
- Orphaned `HashZArrayConfig`/`ResmgrConfig` structs removed
- CLI flags for hash-table tunables (optional Req 6)
- 3-node HA live soak verified clean
- Commits: 17 commits from `0205ac3c` through `519e11c7`

### Jul 22 (Tue) — `admin-config-tool` ship day

- **All 18 tasks (1.1–5.6) completed**, spec marked implementation-complete
- **New binary**: `xceadmin` — Python-based admin config build/push/discovery tool
- Package: `xceadmin.build` (YAML→capnp), `xceadmin.push` (controller push), `xceadmin.discovery` (auto-detect), `xceadmin.cli` (CLI entry)
- Library: `xcecommon.schema`, `xcecommon.validation`, `xcecommon.mapping`, `xcecommon.wire`
- Schema fix: `Data` struct renamed to `XceData` (unshadow capnp builtin blob type)
- Wired into meson + nix build, HA deploy/test skills, `deploy-ha.sh --build-config`
- Tests: golden-file build determinism, mock-controller push/dry-run coverage
- Commits: 18 commits from `b30a856a` through `b6bf7cb8`

### Jul 23 (Wed) — Stabilization + Spec expansion

- **config-upgrade-snapshot** spec expanded: 6 requirements, 26 ACs, 9 tasks (8.0–8.9)
- `blocked_on` cleared to `[]` after bug-1418 re-verified FIXED (10-round chaotic soak PASS)
- **bug-1462** fixed: late-joiner config-push stall — `ha_promotion_config_tag_floor` clamps `committed_config_tag` to `max(safe_tag, self_tag)`. 20+20 round soaks PASS
- Limits.yaml wired through `cluster_config` into controller (real config path)
- xcecli: restored dangling CA-cert cleanup, enabled https support
- **resource-distribution-mapping** spec scaffolded, scope deferred (3 candidate sizes)
- **cc-sdd to Gitea sync** skill added
- **bug-1464** identified: `xce-build.sh` shared build-dir clobber (per-component `rm -rf`)
- Commits: 12 commits from `bf6a9b34` through `cae4ef0c`

### Jul 24 (Thu) — config-upgrade-snapshot heavy lift

Tasks 8.0–8.5 executed:

- **Task 8.0**: HostAdminState/QueueAdminState capnp structs defined, snapshot format version bumped (`9e68335a`)
- **Task 8.1**: Host admin-state frames collected into checkpoint, queue frames migrated to capnp wire (`a71aa3f8`, `70ff3e5b`)
- **Task 8.2**: Embed active configuration bytes in every checkpoint — safety gate (`387ea35c`)
- **Task 8.3**: Restore host admin-state frame safely with create-if-missing (`5b1600a8`)
- **Task 8.4**: Configcompare tool + snapverify extended for byte-identical diff (`1d09631d`)
- **Task 8.5**: Add bug-857 regression test, correct design.md (`2bd44e8e`)
- **Materialization**: upgrade snapshot materialized on config-marker apply (`1da26b04`)
- **Activation fence**: hash-verification gate on materialized snapshot; primary serving/promotion gated on fence pass (`7b507156`, `89b85e16`)
- Removed restart/replay fallback for config activation (`41af0540`)

Commits: 10 commits from `9e68335a` through `1da26b04`

### Jul 25 (Fri) — Soak testing, bug discovery, task 8.6–8.9

- **Task 8.6**: Live config-upgrade soak infra built (`run-ha-soak.sh --config-upgrade`). Fixed bug-1468 (repl config-push duplicate-connect) and bug-1469 (config tag clamp). **Blocked on bug-1472** (out-of-scope HA replication defect for worker admin-status).
- **Task 8.7**: Fault-inject crash/restart + leadership change tests (`d4af605f`)
- **Task 8.8**: Overlapping config upgrades supersede cleanly (`b78efd03`)
- **Task 8.9**: Live WAL-trim-under-chaos soak infra, blocked on bug-1474 (`8b3e2ab0`)

**Major bugs discovered (open):**

| Bug | Severity | Description |
|-----|----------|-------------|
| **bug-1475** | HIGH | Standby catch-up liveness stall: WAL_CHECKPOINT at lsn=12 poisons applied-lsn advertisement. Root cause: `persist_write_checkpoint_marker` does NOT update term index → `on_get_entry_term(12)` returns 0 → raft library takes `ae_append` path not `ae_conflict_truncate`. Primary streams lsn=13 at raft index 12 → ctrl-1 applied_lsn never advances past 12 → infinite `ae_reject_logmismatch` loop. Full mechanism documented in claude-mem obs 16689–16706. |
| **bug-1476** | HIGH | `load_primary_config` concurrent UAF (heap-use-after-free). Two vCPU threads can concurrently free and decode `myconf->cluster_config` via `handle_repl_config_push` → ASAN crash in `decode_Resource_list`. No mutex protecting the read-free-decode sequence. |

**Build issues:**
- Ninja/Meson parallel-build race (5th recurrence logged)
- ASAN rebuild: 2 full retries needed, all 1033 tests pass clean after retry2

Commits: 7 commits from `959b0cda` through `8b3e2ab0`

### Jul 26 (Sat, early morning) — Bug fixes for 8.6/8.9 blockers

- **bug-1474** FIXED: two-pass slot allocation (virgin→oldest-disconnect) in `repl.c`. WAL-retention guard clobber prevented standby slot reuse. 2 tests RED→GREEN. Commit: `c0b0c7d1`
- **bug-1472** FIXED: worker admin-status replication gap. `worker_from_capnp` used INSERT semantics (new worker) instead of UPDATE (existing worker). +3 tests. Commit: `7547a8f1`
- **1033 tests all pass** (C + Go + Python layers, ASAN clean)

Commits: 2 commits

---

## Spec Status

| Spec | Phase | Status |
|------|-------|--------|
| **config-upgrade-snapshot** | Implementation | Tasks 8.0–8.9: 8.0–8.5 complete, 8.6 blocked on bug-1472 (now fixed), 8.7 done, 8.8 done, 8.9 blocked on bug-1474 (now fixed). Tasks 8.6 & 8.9 need soak re-verification. 6 reqs / 26 ACs. |
| **clusterconfig-schema-split** | **COMPLETE** | All 12 tasks done, live-soak verified |
| **admin-config-tool** | **COMPLETE** | All 18 tasks done, xceadmin binary shipped |
| **resource-distribution-mapping** | Scoping | Scaffolded, scope deferred |

---

## Bug Register (Jul 19–26)

| Bug | Date | Status | Description |
|-----|------|--------|-------------|
| bug-1452 | Jul 19 | **FIXED** | commit_engine_fini/persist_stop teardown race |
| bug-1453/4/5/6 | Jul 19–20 | **FIXED** | Dead CA-cert plumbing in scheduler/xcecli |
| bug-1418 | Jul 23 | **VERIFIED FIXED** | term desync + apply-divergent stall (re-verified with 10-round chaotic soak) |
| bug-1462 | Jul 23 | **FIXED** | Late-joiner config-push stall |
| bug-1464 | Jul 23 | **KNOWN** | xce-build.sh shared build-dir clobber (mitigation: use `/xce-build all`) |
| bug-1468 | Jul 25 | **FIXED** | Repl config-push duplicate-connect |
| bug-1469 | Jul 25 | **FIXED** | Config tag clamp on promotion |
| bug-1475 | Jul 25 | **OPEN** | Standby catch-up liveness stall (WAL_CHECKPOINT term-index gap) |
| bug-1476 | Jul 25 | **OPEN** | load_primary_config concurrent UAF |
| bug-1472 | Jul 25→26 | **FIXED** | Worker admin-status replication (INSERT→UPDATE semantics) |
| bug-1474 | Jul 25→26 | **FIXED** | Repl slot-reuse guard clobber (two-pass allocation) |

---

## Commits: 63 total (Jul 19–26)

```
c0b0c7d1 fix(repl): harden standby slot allocation (bug-1474)
7547a8f1 fix(worker): replicate admin host-control status (bug-1472)
8b3e2ab0 feat(config-upgrade-snapshot): task 8.9 WAL-trim-under-chaos soak infra
b78efd03 feat(config-upgrade-snapshot): task 8.8 overlapping config upgrades
d4af605f test(config-upgrade-snapshot): fault-inject crash/restart + leadership (task 8.7)
959b0cda feat(config-upgrade-snapshot): live config-upgrade soak + bug-1468/1469 (task 8.6)
2bd44e8e feat(config-upgrade-snapshot): bug-857 regression test (task 8.5)
1d09631d feat(config-upgrade-snapshot): configcompare tool + snapverify (task 8.4)
4aeae6e2 test(config-upgrade-snapshot): WAL-trim safety + fault-injection seam
41af0540 feat(config-upgrade-snapshot): remove restart/replay fallback
89b85e16 feat(config-upgrade-snapshot): gate primary on activation fence
7b507156 feat(config-upgrade-snapshot): activation fence materialization+hash
1da26b04 feat(config-upgrade-snapshot): materialize upgrade snapshot on config-marker apply
00cd26a9 feat(config-upgrade-snapshot): decode host/queue admin-state on checkpoint load
5b1600a8 feat(config-upgrade-snapshot): restore host admin-state safely
38735ea5 feat(config-upgrade-snapshot): embed config bytes in checkpoint
70ff3e5b feat(config-upgrade-snapshot): migrate queue admin-state to capnp
a71aa3f8 feat(config-upgrade-snapshot): collect host admin-state frames
9e68335a feat(config-upgrade-snapshot): HostAdminState/QueueAdminState capnp structs
e0c5e8b7 fix(nix): pin capnpc source to immutable commit
cae4ef0c chore: save wolf state
c921e59b feat: resource distribution map project
1705495a fix(admin-config-tool): xceadmin config build warnings
4ec40167 fix(xcecli): restore CA-cert cleanup, enable https
ff1be480 merge
b87d5e68 chore: commit wolf status
60c25c39 chore(openwolf): update session memory/buglog/anatomy
8ee98bee docs(config-upgrade-snapshot): expand spec to 6 reqs/26 ACs
bf6a9b34 feat(tooling): cc-sdd to Gitea sync skill
18e7a0e9 feat(controller): wire limits.yaml through cluster_config
0649e17f fix(controller): clamp committed_config_tag on promotion (bug-1462)
ee8473bd feat(admin-config-tool): drop ha/data/, match real client TLS
db2b5f17 fix(admin-config-tool): pin xceadmin python3 to meson resolution
d5330fc9 feat(admin-config-tool): wire xceadmin into HA deploy/test skills
893a01db feat(admin-config-tool): install standalone xceadmin binary via meson
e4fd9e1d feat(admin-config-tool): wire xceadmin into meson + nix build
b6bf7cb8 chore(admin-config-tool): mark implementation-complete (18 tasks done)
a1478510 docs(admin-config-tool): commit spec docs and example configs
67cc3611 feat(admin-config-tool): task 5.6 mock-controller push/dry-run coverage
48a5a0b4 feat(admin-config-tool): task 5.4 golden-file build determinism tests
b5040e08 feat(admin-config-tool): tasks 5.1-5.3 mapping/wire/schema coverage
ad27147a feat(admin-config-tool): task 4.2 deploy-ha.sh --build-config
eeda32c6 feat(admin-config-tool): task 4.1 xceadmin.cli
c33ef7d4 feat(admin-config-tool): task 3.3 xceadmin.push
e53c086e fix(schema): rename Data -> XceData
7ed3a394 feat(admin-config-tool): task 3.2 xceadmin.discovery
72798d4a feat(admin-config-tool): task 3.1 xceadmin.build
77ee831e feat(admin-config-tool): task 2.4 xcecommon.wire
666a838f feat(admin-config-tool): task 2.3 xcecommon.mapping
3a2094a4 merge: job_peek_lchen
dbf5c6c5 feat(admin-config-tool): task 2.2 xcecommon.schema
12b13a54 feat(admin-config-tool): task 2.1 xcecommon.validation
798f19b8 feat(admin-config-tool): task 1.3 nix + xce-build.sh integration
2e4ef4e2 feat(admin-config-tool): task 1.2 shared exception taxonomy
b30a856a feat(admin-config-tool): task 1.1 xceadmin package skeleton
519e11c7 chore(controller): clusterconfig-schema-split mark complete
a74eb008 docs(controller): clusterconfig-schema-split close validation gaps
553da7c7 feat(controller): clusterconfig-schema-split task 6.2 CLI tunables
c3107b95 feat(controller): clusterconfig-schema-split task 6.1 CLI flags
edd79562 chore(controller): clusterconfig-schema-split task 5.3 single-node live-verified
3e01fb1b chore(controller): clusterconfig-schema-split task 5.2 3-node HA soak
f503a43d feat(controller): clusterconfig-schema-split task 5.1 regenerate cluster.bin
29f9b026 chore(controller): clusterconfig-schema-split task 4 build+test verified
ba6fdc57 feat(controller): clusterconfig-schema-split task 3 fix test fixtures
1fa78443 feat(controller): clusterconfig-schema-split task 2.3 remove dead port default
81fecb49 feat(controller): clusterconfig-schema-split task 2.2 remove candidates debug-dump
8f047d93 feat(controller): clusterconfig-schema-split task 1.3 remove orphaned structs
1229f9a9 feat(controller): clusterconfig-schema-split task 2.1 two-tier port fallback
0205ac3c feat(controller): clusterconfig-schema-split task 1.2 remove dead members
51524820 feat(controller): clusterconfig-schema-split task 1.1 remove dead fields
94f50ce6 fix(scheduler,cli): bug-1453/1454/1455/1456 dead CA-cert + HA config
619f41f7 chore: update OpenWolf session state (bug-1452 fix)
d67ccc22 fix(controller): bug-1452 commit_engine_fini/persist_stop teardown race
e1f4d676 chore: update OpenWolf session state (xce-audit-invariants pass)
```

---

## Next Steps

1. **Soak re-verify tasks 8.6 and 8.9** with bug-1472 + bug-1474 fixes applied. Both blocking bugs now fixed, but soak must pass end-to-end.
2. **bug-1475** needs root-cause fix: `persist_write_checkpoint_marker` must update term index so `on_get_entry_term` returns correct term for WAL_CHECKPOINT slot.
3. **bug-1476** needs mutex guard on `load_primary_config`'s `myconf->cluster_config` read-free-decode sequence.
4. **Task 8.6** blocked_on cleared (bug-1472 fixed), can proceed with worker admin-status rotation verification.
5. **Task 8.9** blocked_on cleared (bug-1474 fixed), can proceed with WAL-trim-under-chaos verification.

---

*Generated 2026-07-26 | Branch: `config` | 63 commits, 4 bugs fixed, 2 specs shipped, 2 bugs open*