From 6bc3fda2d487c27dc88c96d25082f4bc31f88f97 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Mon, 27 Jul 2026 16:10:32 +0800 Subject: [PATCH] add report --- reports/jul19-jul26-2026.md | 225 ++++++++++++++++++++++++++++++++++++ 1 file changed, 225 insertions(+) create mode 100644 reports/jul19-jul26-2026.md diff --git a/reports/jul19-jul26-2026.md b/reports/jul19-jul26-2026.md new file mode 100644 index 00000000..250d4222 --- /dev/null +++ b/reports/jul19-jul26-2026.md @@ -0,0 +1,225 @@ +# Work Summary: Jul 19 – Jul 26, 2026 + +## Branch: `config` | Project: XCE Cluster Management System + +--- + +## Overview + +Week focused on three major workstreams: completing `clusterconfig-schema-split` (Jul 21), shipping `admin-config-tool` (Jul 22), and executing `config-upgrade-snapshot` tasks 8.0–8.9 (Jul 24–26). Two critical bugs discovered during soak testing (bug-1475 standby liveness stall, bug-1476 config-reload UAF) remain open. Two bugs fixed and committed (bug-1472 worker replication, bug-1474 repl slot clobber). + +--- + +## Day-by-Day Summary + +### Jul 19 (Fri) + +- **bug-1452** fixed: `commit_engine_fini`/`persist_stop` teardown race in controller +- **xce-audit-invariants** controller pass completed +- **bug-1453/1454/1455/1456**: scheduler + xcecli dead CA-cert plumbing removed, scheduler HA config bugs fixed + +### Jul 20 (Sat) + +- Continued scheduler/xcecli cleanups from Jul 19 +- Git: `94f50ce6` — fix(scheduler,cli): bug-1453–1456 remove dead CA-cert plumbing + +### Jul 21 (Mon) — `clusterconfig-schema-split` ship day + +- **All 12 tasks (1.1–6.2) completed**, spec marked implementation-complete +- Dead fields removed from `ClusterConfig` capnp schema and `cluster_config_t` +- Two-tier port fallback (`cluster_open_listener`), dead port defaulting removed +- Orphaned `HashZArrayConfig`/`ResmgrConfig` structs removed +- CLI flags for hash-table tunables (optional Req 6) +- 3-node HA live soak verified clean +- Commits: 17 commits from `0205ac3c` through `519e11c7` + +### Jul 22 (Tue) — `admin-config-tool` ship day + +- **All 18 tasks (1.1–5.6) completed**, spec marked implementation-complete +- **New binary**: `xceadmin` — Python-based admin config build/push/discovery tool +- Package: `xceadmin.build` (YAML→capnp), `xceadmin.push` (controller push), `xceadmin.discovery` (auto-detect), `xceadmin.cli` (CLI entry) +- Library: `xcecommon.schema`, `xcecommon.validation`, `xcecommon.mapping`, `xcecommon.wire` +- Schema fix: `Data` struct renamed to `XceData` (unshadow capnp builtin blob type) +- Wired into meson + nix build, HA deploy/test skills, `deploy-ha.sh --build-config` +- Tests: golden-file build determinism, mock-controller push/dry-run coverage +- Commits: 18 commits from `b30a856a` through `b6bf7cb8` + +### Jul 23 (Wed) — Stabilization + Spec expansion + +- **config-upgrade-snapshot** spec expanded: 6 requirements, 26 ACs, 9 tasks (8.0–8.9) +- `blocked_on` cleared to `[]` after bug-1418 re-verified FIXED (10-round chaotic soak PASS) +- **bug-1462** fixed: late-joiner config-push stall — `ha_promotion_config_tag_floor` clamps `committed_config_tag` to `max(safe_tag, self_tag)`. 20+20 round soaks PASS +- Limits.yaml wired through `cluster_config` into controller (real config path) +- xcecli: restored dangling CA-cert cleanup, enabled https support +- **resource-distribution-mapping** spec scaffolded, scope deferred (3 candidate sizes) +- **cc-sdd to Gitea sync** skill added +- **bug-1464** identified: `xce-build.sh` shared build-dir clobber (per-component `rm -rf`) +- Commits: 12 commits from `bf6a9b34` through `cae4ef0c` + +### Jul 24 (Thu) — config-upgrade-snapshot heavy lift + +Tasks 8.0–8.5 executed: + +- **Task 8.0**: HostAdminState/QueueAdminState capnp structs defined, snapshot format version bumped (`9e68335a`) +- **Task 8.1**: Host admin-state frames collected into checkpoint, queue frames migrated to capnp wire (`a71aa3f8`, `70ff3e5b`) +- **Task 8.2**: Embed active configuration bytes in every checkpoint — safety gate (`387ea35c`) +- **Task 8.3**: Restore host admin-state frame safely with create-if-missing (`5b1600a8`) +- **Task 8.4**: Configcompare tool + snapverify extended for byte-identical diff (`1d09631d`) +- **Task 8.5**: Add bug-857 regression test, correct design.md (`2bd44e8e`) +- **Materialization**: upgrade snapshot materialized on config-marker apply (`1da26b04`) +- **Activation fence**: hash-verification gate on materialized snapshot; primary serving/promotion gated on fence pass (`7b507156`, `89b85e16`) +- Removed restart/replay fallback for config activation (`41af0540`) + +Commits: 10 commits from `9e68335a` through `1da26b04` + +### Jul 25 (Fri) — Soak testing, bug discovery, task 8.6–8.9 + +- **Task 8.6**: Live config-upgrade soak infra built (`run-ha-soak.sh --config-upgrade`). Fixed bug-1468 (repl config-push duplicate-connect) and bug-1469 (config tag clamp). **Blocked on bug-1472** (out-of-scope HA replication defect for worker admin-status). +- **Task 8.7**: Fault-inject crash/restart + leadership change tests (`d4af605f`) +- **Task 8.8**: Overlapping config upgrades supersede cleanly (`b78efd03`) +- **Task 8.9**: Live WAL-trim-under-chaos soak infra, blocked on bug-1474 (`8b3e2ab0`) + +**Major bugs discovered (open):** + +| Bug | Severity | Description | +|-----|----------|-------------| +| **bug-1475** | HIGH | Standby catch-up liveness stall: WAL_CHECKPOINT at lsn=12 poisons applied-lsn advertisement. Root cause: `persist_write_checkpoint_marker` does NOT update term index → `on_get_entry_term(12)` returns 0 → raft library takes `ae_append` path not `ae_conflict_truncate`. Primary streams lsn=13 at raft index 12 → ctrl-1 applied_lsn never advances past 12 → infinite `ae_reject_logmismatch` loop. Full mechanism documented in claude-mem obs 16689–16706. | +| **bug-1476** | HIGH | `load_primary_config` concurrent UAF (heap-use-after-free). Two vCPU threads can concurrently free and decode `myconf->cluster_config` via `handle_repl_config_push` → ASAN crash in `decode_Resource_list`. No mutex protecting the read-free-decode sequence. | + +**Build issues:** +- Ninja/Meson parallel-build race (5th recurrence logged) +- ASAN rebuild: 2 full retries needed, all 1033 tests pass clean after retry2 + +Commits: 7 commits from `959b0cda` through `8b3e2ab0` + +### Jul 26 (Sat, early morning) — Bug fixes for 8.6/8.9 blockers + +- **bug-1474** FIXED: two-pass slot allocation (virgin→oldest-disconnect) in `repl.c`. WAL-retention guard clobber prevented standby slot reuse. 2 tests RED→GREEN. Commit: `c0b0c7d1` +- **bug-1472** FIXED: worker admin-status replication gap. `worker_from_capnp` used INSERT semantics (new worker) instead of UPDATE (existing worker). +3 tests. Commit: `7547a8f1` +- **1033 tests all pass** (C + Go + Python layers, ASAN clean) + +Commits: 2 commits + +--- + +## Spec Status + +| Spec | Phase | Status | +|------|-------|--------| +| **config-upgrade-snapshot** | Implementation | Tasks 8.0–8.9: 8.0–8.5 complete, 8.6 blocked on bug-1472 (now fixed), 8.7 done, 8.8 done, 8.9 blocked on bug-1474 (now fixed). Tasks 8.6 & 8.9 need soak re-verification. 6 reqs / 26 ACs. | +| **clusterconfig-schema-split** | **COMPLETE** | All 12 tasks done, live-soak verified | +| **admin-config-tool** | **COMPLETE** | All 18 tasks done, xceadmin binary shipped | +| **resource-distribution-mapping** | Scoping | Scaffolded, scope deferred | + +--- + +## Bug Register (Jul 19–26) + +| Bug | Date | Status | Description | +|-----|------|--------|-------------| +| bug-1452 | Jul 19 | **FIXED** | commit_engine_fini/persist_stop teardown race | +| bug-1453/4/5/6 | Jul 19–20 | **FIXED** | Dead CA-cert plumbing in scheduler/xcecli | +| bug-1418 | Jul 23 | **VERIFIED FIXED** | term desync + apply-divergent stall (re-verified with 10-round chaotic soak) | +| bug-1462 | Jul 23 | **FIXED** | Late-joiner config-push stall | +| bug-1464 | Jul 23 | **KNOWN** | xce-build.sh shared build-dir clobber (mitigation: use `/xce-build all`) | +| bug-1468 | Jul 25 | **FIXED** | Repl config-push duplicate-connect | +| bug-1469 | Jul 25 | **FIXED** | Config tag clamp on promotion | +| bug-1475 | Jul 25 | **OPEN** | Standby catch-up liveness stall (WAL_CHECKPOINT term-index gap) | +| bug-1476 | Jul 25 | **OPEN** | load_primary_config concurrent UAF | +| bug-1472 | Jul 25→26 | **FIXED** | Worker admin-status replication (INSERT→UPDATE semantics) | +| bug-1474 | Jul 25→26 | **FIXED** | Repl slot-reuse guard clobber (two-pass allocation) | + +--- + +## Commits: 63 total (Jul 19–26) + +``` +c0b0c7d1 fix(repl): harden standby slot allocation (bug-1474) +7547a8f1 fix(worker): replicate admin host-control status (bug-1472) +8b3e2ab0 feat(config-upgrade-snapshot): task 8.9 WAL-trim-under-chaos soak infra +b78efd03 feat(config-upgrade-snapshot): task 8.8 overlapping config upgrades +d4af605f test(config-upgrade-snapshot): fault-inject crash/restart + leadership (task 8.7) +959b0cda feat(config-upgrade-snapshot): live config-upgrade soak + bug-1468/1469 (task 8.6) +2bd44e8e feat(config-upgrade-snapshot): bug-857 regression test (task 8.5) +1d09631d feat(config-upgrade-snapshot): configcompare tool + snapverify (task 8.4) +4aeae6e2 test(config-upgrade-snapshot): WAL-trim safety + fault-injection seam +41af0540 feat(config-upgrade-snapshot): remove restart/replay fallback +89b85e16 feat(config-upgrade-snapshot): gate primary on activation fence +7b507156 feat(config-upgrade-snapshot): activation fence materialization+hash +1da26b04 feat(config-upgrade-snapshot): materialize upgrade snapshot on config-marker apply +00cd26a9 feat(config-upgrade-snapshot): decode host/queue admin-state on checkpoint load +5b1600a8 feat(config-upgrade-snapshot): restore host admin-state safely +38735ea5 feat(config-upgrade-snapshot): embed config bytes in checkpoint +70ff3e5b feat(config-upgrade-snapshot): migrate queue admin-state to capnp +a71aa3f8 feat(config-upgrade-snapshot): collect host admin-state frames +9e68335a feat(config-upgrade-snapshot): HostAdminState/QueueAdminState capnp structs +e0c5e8b7 fix(nix): pin capnpc source to immutable commit +cae4ef0c chore: save wolf state +c921e59b feat: resource distribution map project +1705495a fix(admin-config-tool): xceadmin config build warnings +4ec40167 fix(xcecli): restore CA-cert cleanup, enable https +ff1be480 merge +b87d5e68 chore: commit wolf status +60c25c39 chore(openwolf): update session memory/buglog/anatomy +8ee98bee docs(config-upgrade-snapshot): expand spec to 6 reqs/26 ACs +bf6a9b34 feat(tooling): cc-sdd to Gitea sync skill +18e7a0e9 feat(controller): wire limits.yaml through cluster_config +0649e17f fix(controller): clamp committed_config_tag on promotion (bug-1462) +ee8473bd feat(admin-config-tool): drop ha/data/, match real client TLS +db2b5f17 fix(admin-config-tool): pin xceadmin python3 to meson resolution +d5330fc9 feat(admin-config-tool): wire xceadmin into HA deploy/test skills +893a01db feat(admin-config-tool): install standalone xceadmin binary via meson +e4fd9e1d feat(admin-config-tool): wire xceadmin into meson + nix build +b6bf7cb8 chore(admin-config-tool): mark implementation-complete (18 tasks done) +a1478510 docs(admin-config-tool): commit spec docs and example configs +67cc3611 feat(admin-config-tool): task 5.6 mock-controller push/dry-run coverage +48a5a0b4 feat(admin-config-tool): task 5.4 golden-file build determinism tests +b5040e08 feat(admin-config-tool): tasks 5.1-5.3 mapping/wire/schema coverage +ad27147a feat(admin-config-tool): task 4.2 deploy-ha.sh --build-config +eeda32c6 feat(admin-config-tool): task 4.1 xceadmin.cli +c33ef7d4 feat(admin-config-tool): task 3.3 xceadmin.push +e53c086e fix(schema): rename Data -> XceData +7ed3a394 feat(admin-config-tool): task 3.2 xceadmin.discovery +72798d4a feat(admin-config-tool): task 3.1 xceadmin.build +77ee831e feat(admin-config-tool): task 2.4 xcecommon.wire +666a838f feat(admin-config-tool): task 2.3 xcecommon.mapping +3a2094a4 merge: job_peek_lchen +dbf5c6c5 feat(admin-config-tool): task 2.2 xcecommon.schema +12b13a54 feat(admin-config-tool): task 2.1 xcecommon.validation +798f19b8 feat(admin-config-tool): task 1.3 nix + xce-build.sh integration +2e4ef4e2 feat(admin-config-tool): task 1.2 shared exception taxonomy +b30a856a feat(admin-config-tool): task 1.1 xceadmin package skeleton +519e11c7 chore(controller): clusterconfig-schema-split mark complete +a74eb008 docs(controller): clusterconfig-schema-split close validation gaps +553da7c7 feat(controller): clusterconfig-schema-split task 6.2 CLI tunables +c3107b95 feat(controller): clusterconfig-schema-split task 6.1 CLI flags +edd79562 chore(controller): clusterconfig-schema-split task 5.3 single-node live-verified +3e01fb1b chore(controller): clusterconfig-schema-split task 5.2 3-node HA soak +f503a43d feat(controller): clusterconfig-schema-split task 5.1 regenerate cluster.bin +29f9b026 chore(controller): clusterconfig-schema-split task 4 build+test verified +ba6fdc57 feat(controller): clusterconfig-schema-split task 3 fix test fixtures +1fa78443 feat(controller): clusterconfig-schema-split task 2.3 remove dead port default +81fecb49 feat(controller): clusterconfig-schema-split task 2.2 remove candidates debug-dump +8f047d93 feat(controller): clusterconfig-schema-split task 1.3 remove orphaned structs +1229f9a9 feat(controller): clusterconfig-schema-split task 2.1 two-tier port fallback +0205ac3c feat(controller): clusterconfig-schema-split task 1.2 remove dead members +51524820 feat(controller): clusterconfig-schema-split task 1.1 remove dead fields +94f50ce6 fix(scheduler,cli): bug-1453/1454/1455/1456 dead CA-cert + HA config +619f41f7 chore: update OpenWolf session state (bug-1452 fix) +d67ccc22 fix(controller): bug-1452 commit_engine_fini/persist_stop teardown race +e1f4d676 chore: update OpenWolf session state (xce-audit-invariants pass) +``` + +--- + +## Next Steps + +1. **Soak re-verify tasks 8.6 and 8.9** with bug-1472 + bug-1474 fixes applied. Both blocking bugs now fixed, but soak must pass end-to-end. +2. **bug-1475** needs root-cause fix: `persist_write_checkpoint_marker` must update term index so `on_get_entry_term` returns correct term for WAL_CHECKPOINT slot. +3. **bug-1476** needs mutex guard on `load_primary_config`'s `myconf->cluster_config` read-free-decode sequence. +4. **Task 8.6** blocked_on cleared (bug-1472 fixed), can proceed with worker admin-status rotation verification. +5. **Task 8.9** blocked_on cleared (bug-1474 fixed), can proceed with WAL-trim-under-chaos verification. + +--- + +*Generated 2026-07-26 | Branch: `config` | 63 commits, 4 bugs fixed, 2 specs shipped, 2 bugs open*