225 lines
14 KiB
Markdown
225 lines
14 KiB
Markdown
# Work Summary: Jul 19 – Jul 26, 2026
|
||
|
||
## Branch: `config` | Project: XCE Cluster Management System
|
||
|
||
---
|
||
|
||
## Overview
|
||
|
||
Week focused on three major workstreams: completing `clusterconfig-schema-split` (Jul 21), shipping `admin-config-tool` (Jul 22), and executing `config-upgrade-snapshot` tasks 8.0–8.9 (Jul 24–26). Two critical bugs discovered during soak testing (bug-1475 standby liveness stall, bug-1476 config-reload UAF) remain open. Two bugs fixed and committed (bug-1472 worker replication, bug-1474 repl slot clobber).
|
||
|
||
---
|
||
|
||
## Day-by-Day Summary
|
||
|
||
### Jul 19 (Fri)
|
||
|
||
- **bug-1452** fixed: `commit_engine_fini`/`persist_stop` teardown race in controller
|
||
- **xce-audit-invariants** controller pass completed
|
||
- **bug-1453/1454/1455/1456**: scheduler + xcecli dead CA-cert plumbing removed, scheduler HA config bugs fixed
|
||
|
||
### Jul 20 (Sat)
|
||
|
||
- Continued scheduler/xcecli cleanups from Jul 19
|
||
- Git: `94f50ce6` — fix(scheduler,cli): bug-1453–1456 remove dead CA-cert plumbing
|
||
|
||
### Jul 21 (Mon) — `clusterconfig-schema-split` ship day
|
||
|
||
- **All 12 tasks (1.1–6.2) completed**, spec marked implementation-complete
|
||
- Dead fields removed from `ClusterConfig` capnp schema and `cluster_config_t`
|
||
- Two-tier port fallback (`cluster_open_listener`), dead port defaulting removed
|
||
- Orphaned `HashZArrayConfig`/`ResmgrConfig` structs removed
|
||
- CLI flags for hash-table tunables (optional Req 6)
|
||
- 3-node HA live soak verified clean
|
||
- Commits: 17 commits from `0205ac3c` through `519e11c7`
|
||
|
||
### Jul 22 (Tue) — `admin-config-tool` ship day
|
||
|
||
- **All 18 tasks (1.1–5.6) completed**, spec marked implementation-complete
|
||
- **New binary**: `xceadmin` — Python-based admin config build/push/discovery tool
|
||
- Package: `xceadmin.build` (YAML→capnp), `xceadmin.push` (controller push), `xceadmin.discovery` (auto-detect), `xceadmin.cli` (CLI entry)
|
||
- Library: `xcecommon.schema`, `xcecommon.validation`, `xcecommon.mapping`, `xcecommon.wire`
|
||
- Schema fix: `Data` struct renamed to `XceData` (unshadow capnp builtin blob type)
|
||
- Wired into meson + nix build, HA deploy/test skills, `deploy-ha.sh --build-config`
|
||
- Tests: golden-file build determinism, mock-controller push/dry-run coverage
|
||
- Commits: 18 commits from `b30a856a` through `b6bf7cb8`
|
||
|
||
### Jul 23 (Wed) — Stabilization + Spec expansion
|
||
|
||
- **config-upgrade-snapshot** spec expanded: 6 requirements, 26 ACs, 9 tasks (8.0–8.9)
|
||
- `blocked_on` cleared to `[]` after bug-1418 re-verified FIXED (10-round chaotic soak PASS)
|
||
- **bug-1462** fixed: late-joiner config-push stall — `ha_promotion_config_tag_floor` clamps `committed_config_tag` to `max(safe_tag, self_tag)`. 20+20 round soaks PASS
|
||
- Limits.yaml wired through `cluster_config` into controller (real config path)
|
||
- xcecli: restored dangling CA-cert cleanup, enabled https support
|
||
- **resource-distribution-mapping** spec scaffolded, scope deferred (3 candidate sizes)
|
||
- **cc-sdd to Gitea sync** skill added
|
||
- **bug-1464** identified: `xce-build.sh` shared build-dir clobber (per-component `rm -rf`)
|
||
- Commits: 12 commits from `bf6a9b34` through `cae4ef0c`
|
||
|
||
### Jul 24 (Thu) — config-upgrade-snapshot heavy lift
|
||
|
||
Tasks 8.0–8.5 executed:
|
||
|
||
- **Task 8.0**: HostAdminState/QueueAdminState capnp structs defined, snapshot format version bumped (`9e68335a`)
|
||
- **Task 8.1**: Host admin-state frames collected into checkpoint, queue frames migrated to capnp wire (`a71aa3f8`, `70ff3e5b`)
|
||
- **Task 8.2**: Embed active configuration bytes in every checkpoint — safety gate (`387ea35c`)
|
||
- **Task 8.3**: Restore host admin-state frame safely with create-if-missing (`5b1600a8`)
|
||
- **Task 8.4**: Configcompare tool + snapverify extended for byte-identical diff (`1d09631d`)
|
||
- **Task 8.5**: Add bug-857 regression test, correct design.md (`2bd44e8e`)
|
||
- **Materialization**: upgrade snapshot materialized on config-marker apply (`1da26b04`)
|
||
- **Activation fence**: hash-verification gate on materialized snapshot; primary serving/promotion gated on fence pass (`7b507156`, `89b85e16`)
|
||
- Removed restart/replay fallback for config activation (`41af0540`)
|
||
|
||
Commits: 10 commits from `9e68335a` through `1da26b04`
|
||
|
||
### Jul 25 (Fri) — Soak testing, bug discovery, task 8.6–8.9
|
||
|
||
- **Task 8.6**: Live config-upgrade soak infra built (`run-ha-soak.sh --config-upgrade`). Fixed bug-1468 (repl config-push duplicate-connect) and bug-1469 (config tag clamp). **Blocked on bug-1472** (out-of-scope HA replication defect for worker admin-status).
|
||
- **Task 8.7**: Fault-inject crash/restart + leadership change tests (`d4af605f`)
|
||
- **Task 8.8**: Overlapping config upgrades supersede cleanly (`b78efd03`)
|
||
- **Task 8.9**: Live WAL-trim-under-chaos soak infra, blocked on bug-1474 (`8b3e2ab0`)
|
||
|
||
**Major bugs discovered (open):**
|
||
|
||
| Bug | Severity | Description |
|
||
|-----|----------|-------------|
|
||
| **bug-1475** | HIGH | Standby catch-up liveness stall: WAL_CHECKPOINT at lsn=12 poisons applied-lsn advertisement. Root cause: `persist_write_checkpoint_marker` does NOT update term index → `on_get_entry_term(12)` returns 0 → raft library takes `ae_append` path not `ae_conflict_truncate`. Primary streams lsn=13 at raft index 12 → ctrl-1 applied_lsn never advances past 12 → infinite `ae_reject_logmismatch` loop. Full mechanism documented in claude-mem obs 16689–16706. |
|
||
| **bug-1476** | HIGH | `load_primary_config` concurrent UAF (heap-use-after-free). Two vCPU threads can concurrently free and decode `myconf->cluster_config` via `handle_repl_config_push` → ASAN crash in `decode_Resource_list`. No mutex protecting the read-free-decode sequence. |
|
||
|
||
**Build issues:**
|
||
- Ninja/Meson parallel-build race (5th recurrence logged)
|
||
- ASAN rebuild: 2 full retries needed, all 1033 tests pass clean after retry2
|
||
|
||
Commits: 7 commits from `959b0cda` through `8b3e2ab0`
|
||
|
||
### Jul 26 (Sat, early morning) — Bug fixes for 8.6/8.9 blockers
|
||
|
||
- **bug-1474** FIXED: two-pass slot allocation (virgin→oldest-disconnect) in `repl.c`. WAL-retention guard clobber prevented standby slot reuse. 2 tests RED→GREEN. Commit: `c0b0c7d1`
|
||
- **bug-1472** FIXED: worker admin-status replication gap. `worker_from_capnp` used INSERT semantics (new worker) instead of UPDATE (existing worker). +3 tests. Commit: `7547a8f1`
|
||
- **1033 tests all pass** (C + Go + Python layers, ASAN clean)
|
||
|
||
Commits: 2 commits
|
||
|
||
---
|
||
|
||
## Spec Status
|
||
|
||
| Spec | Phase | Status |
|
||
|------|-------|--------|
|
||
| **config-upgrade-snapshot** | Implementation | Tasks 8.0–8.9: 8.0–8.5 complete, 8.6 blocked on bug-1472 (now fixed), 8.7 done, 8.8 done, 8.9 blocked on bug-1474 (now fixed). Tasks 8.6 & 8.9 need soak re-verification. 6 reqs / 26 ACs. |
|
||
| **clusterconfig-schema-split** | **COMPLETE** | All 12 tasks done, live-soak verified |
|
||
| **admin-config-tool** | **COMPLETE** | All 18 tasks done, xceadmin binary shipped |
|
||
| **resource-distribution-mapping** | Scoping | Scaffolded, scope deferred |
|
||
|
||
---
|
||
|
||
## Bug Register (Jul 19–26)
|
||
|
||
| Bug | Date | Status | Description |
|
||
|-----|------|--------|-------------|
|
||
| bug-1452 | Jul 19 | **FIXED** | commit_engine_fini/persist_stop teardown race |
|
||
| bug-1453/4/5/6 | Jul 19–20 | **FIXED** | Dead CA-cert plumbing in scheduler/xcecli |
|
||
| bug-1418 | Jul 23 | **VERIFIED FIXED** | term desync + apply-divergent stall (re-verified with 10-round chaotic soak) |
|
||
| bug-1462 | Jul 23 | **FIXED** | Late-joiner config-push stall |
|
||
| bug-1464 | Jul 23 | **KNOWN** | xce-build.sh shared build-dir clobber (mitigation: use `/xce-build all`) |
|
||
| bug-1468 | Jul 25 | **FIXED** | Repl config-push duplicate-connect |
|
||
| bug-1469 | Jul 25 | **FIXED** | Config tag clamp on promotion |
|
||
| bug-1475 | Jul 25 | **OPEN** | Standby catch-up liveness stall (WAL_CHECKPOINT term-index gap) |
|
||
| bug-1476 | Jul 25 | **OPEN** | load_primary_config concurrent UAF |
|
||
| bug-1472 | Jul 25→26 | **FIXED** | Worker admin-status replication (INSERT→UPDATE semantics) |
|
||
| bug-1474 | Jul 25→26 | **FIXED** | Repl slot-reuse guard clobber (two-pass allocation) |
|
||
|
||
---
|
||
|
||
## Commits: 63 total (Jul 19–26)
|
||
|
||
```
|
||
c0b0c7d1 fix(repl): harden standby slot allocation (bug-1474)
|
||
7547a8f1 fix(worker): replicate admin host-control status (bug-1472)
|
||
8b3e2ab0 feat(config-upgrade-snapshot): task 8.9 WAL-trim-under-chaos soak infra
|
||
b78efd03 feat(config-upgrade-snapshot): task 8.8 overlapping config upgrades
|
||
d4af605f test(config-upgrade-snapshot): fault-inject crash/restart + leadership (task 8.7)
|
||
959b0cda feat(config-upgrade-snapshot): live config-upgrade soak + bug-1468/1469 (task 8.6)
|
||
2bd44e8e feat(config-upgrade-snapshot): bug-857 regression test (task 8.5)
|
||
1d09631d feat(config-upgrade-snapshot): configcompare tool + snapverify (task 8.4)
|
||
4aeae6e2 test(config-upgrade-snapshot): WAL-trim safety + fault-injection seam
|
||
41af0540 feat(config-upgrade-snapshot): remove restart/replay fallback
|
||
89b85e16 feat(config-upgrade-snapshot): gate primary on activation fence
|
||
7b507156 feat(config-upgrade-snapshot): activation fence materialization+hash
|
||
1da26b04 feat(config-upgrade-snapshot): materialize upgrade snapshot on config-marker apply
|
||
00cd26a9 feat(config-upgrade-snapshot): decode host/queue admin-state on checkpoint load
|
||
5b1600a8 feat(config-upgrade-snapshot): restore host admin-state safely
|
||
38735ea5 feat(config-upgrade-snapshot): embed config bytes in checkpoint
|
||
70ff3e5b feat(config-upgrade-snapshot): migrate queue admin-state to capnp
|
||
a71aa3f8 feat(config-upgrade-snapshot): collect host admin-state frames
|
||
9e68335a feat(config-upgrade-snapshot): HostAdminState/QueueAdminState capnp structs
|
||
e0c5e8b7 fix(nix): pin capnpc source to immutable commit
|
||
cae4ef0c chore: save wolf state
|
||
c921e59b feat: resource distribution map project
|
||
1705495a fix(admin-config-tool): xceadmin config build warnings
|
||
4ec40167 fix(xcecli): restore CA-cert cleanup, enable https
|
||
ff1be480 merge
|
||
b87d5e68 chore: commit wolf status
|
||
60c25c39 chore(openwolf): update session memory/buglog/anatomy
|
||
8ee98bee docs(config-upgrade-snapshot): expand spec to 6 reqs/26 ACs
|
||
bf6a9b34 feat(tooling): cc-sdd to Gitea sync skill
|
||
18e7a0e9 feat(controller): wire limits.yaml through cluster_config
|
||
0649e17f fix(controller): clamp committed_config_tag on promotion (bug-1462)
|
||
ee8473bd feat(admin-config-tool): drop ha/data/, match real client TLS
|
||
db2b5f17 fix(admin-config-tool): pin xceadmin python3 to meson resolution
|
||
d5330fc9 feat(admin-config-tool): wire xceadmin into HA deploy/test skills
|
||
893a01db feat(admin-config-tool): install standalone xceadmin binary via meson
|
||
e4fd9e1d feat(admin-config-tool): wire xceadmin into meson + nix build
|
||
b6bf7cb8 chore(admin-config-tool): mark implementation-complete (18 tasks done)
|
||
a1478510 docs(admin-config-tool): commit spec docs and example configs
|
||
67cc3611 feat(admin-config-tool): task 5.6 mock-controller push/dry-run coverage
|
||
48a5a0b4 feat(admin-config-tool): task 5.4 golden-file build determinism tests
|
||
b5040e08 feat(admin-config-tool): tasks 5.1-5.3 mapping/wire/schema coverage
|
||
ad27147a feat(admin-config-tool): task 4.2 deploy-ha.sh --build-config
|
||
eeda32c6 feat(admin-config-tool): task 4.1 xceadmin.cli
|
||
c33ef7d4 feat(admin-config-tool): task 3.3 xceadmin.push
|
||
e53c086e fix(schema): rename Data -> XceData
|
||
7ed3a394 feat(admin-config-tool): task 3.2 xceadmin.discovery
|
||
72798d4a feat(admin-config-tool): task 3.1 xceadmin.build
|
||
77ee831e feat(admin-config-tool): task 2.4 xcecommon.wire
|
||
666a838f feat(admin-config-tool): task 2.3 xcecommon.mapping
|
||
3a2094a4 merge: job_peek_lchen
|
||
dbf5c6c5 feat(admin-config-tool): task 2.2 xcecommon.schema
|
||
12b13a54 feat(admin-config-tool): task 2.1 xcecommon.validation
|
||
798f19b8 feat(admin-config-tool): task 1.3 nix + xce-build.sh integration
|
||
2e4ef4e2 feat(admin-config-tool): task 1.2 shared exception taxonomy
|
||
b30a856a feat(admin-config-tool): task 1.1 xceadmin package skeleton
|
||
519e11c7 chore(controller): clusterconfig-schema-split mark complete
|
||
a74eb008 docs(controller): clusterconfig-schema-split close validation gaps
|
||
553da7c7 feat(controller): clusterconfig-schema-split task 6.2 CLI tunables
|
||
c3107b95 feat(controller): clusterconfig-schema-split task 6.1 CLI flags
|
||
edd79562 chore(controller): clusterconfig-schema-split task 5.3 single-node live-verified
|
||
3e01fb1b chore(controller): clusterconfig-schema-split task 5.2 3-node HA soak
|
||
f503a43d feat(controller): clusterconfig-schema-split task 5.1 regenerate cluster.bin
|
||
29f9b026 chore(controller): clusterconfig-schema-split task 4 build+test verified
|
||
ba6fdc57 feat(controller): clusterconfig-schema-split task 3 fix test fixtures
|
||
1fa78443 feat(controller): clusterconfig-schema-split task 2.3 remove dead port default
|
||
81fecb49 feat(controller): clusterconfig-schema-split task 2.2 remove candidates debug-dump
|
||
8f047d93 feat(controller): clusterconfig-schema-split task 1.3 remove orphaned structs
|
||
1229f9a9 feat(controller): clusterconfig-schema-split task 2.1 two-tier port fallback
|
||
0205ac3c feat(controller): clusterconfig-schema-split task 1.2 remove dead members
|
||
51524820 feat(controller): clusterconfig-schema-split task 1.1 remove dead fields
|
||
94f50ce6 fix(scheduler,cli): bug-1453/1454/1455/1456 dead CA-cert + HA config
|
||
619f41f7 chore: update OpenWolf session state (bug-1452 fix)
|
||
d67ccc22 fix(controller): bug-1452 commit_engine_fini/persist_stop teardown race
|
||
e1f4d676 chore: update OpenWolf session state (xce-audit-invariants pass)
|
||
```
|
||
|
||
---
|
||
|
||
## Next Steps
|
||
|
||
1. **Soak re-verify tasks 8.6 and 8.9** with bug-1472 + bug-1474 fixes applied. Both blocking bugs now fixed, but soak must pass end-to-end.
|
||
2. **bug-1475** needs root-cause fix: `persist_write_checkpoint_marker` must update term index so `on_get_entry_term` returns correct term for WAL_CHECKPOINT slot.
|
||
3. **bug-1476** needs mutex guard on `load_primary_config`'s `myconf->cluster_config` read-free-decode sequence.
|
||
4. **Task 8.6** blocked_on cleared (bug-1472 fixed), can proceed with worker admin-status rotation verification.
|
||
5. **Task 8.9** blocked_on cleared (bug-1474 fixed), can proceed with WAL-trim-under-chaos verification.
|
||
|
||
---
|
||
|
||
*Generated 2026-07-26 | Branch: `config` | 63 commits, 4 bugs fixed, 2 specs shipped, 2 bugs open*
|