wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity
myxce/reports/jul19-jul26-2026.md
Rongsong Shen 6bc3fda2d4 add report
2026-07-27 16:10:32 +08:00

14 KiB
Raw Blame History

Work Summary: Jul 19 – Jul 26, 2026

Branch: config | Project: XCE Cluster Management System

Overview

Week focused on three major workstreams: completing clusterconfig-schema-split (Jul 21), shipping admin-config-tool (Jul 22), and executing config-upgrade-snapshot tasks 8.0–8.9 (Jul 24–26). Two critical bugs discovered during soak testing (bug-1475 standby liveness stall, bug-1476 config-reload UAF) remain open. Two bugs fixed and committed (bug-1472 worker replication, bug-1474 repl slot clobber).

Day-by-Day Summary

Jul 19 (Fri)

  • bug-1452 fixed: commit_engine_fini/persist_stop teardown race in controller
  • xce-audit-invariants controller pass completed
  • bug-1453/1454/1455/1456: scheduler + xcecli dead CA-cert plumbing removed, scheduler HA config bugs fixed

Jul 20 (Sat)

  • Continued scheduler/xcecli cleanups from Jul 19
  • Git: 94f50ce6 — fix(scheduler,cli): bug-1453–1456 remove dead CA-cert plumbing

Jul 21 (Mon) — clusterconfig-schema-split ship day

  • All 12 tasks (1.1–6.2) completed, spec marked implementation-complete
  • Dead fields removed from ClusterConfig capnp schema and cluster_config_t
  • Two-tier port fallback (cluster_open_listener), dead port defaulting removed
  • Orphaned HashZArrayConfig/ResmgrConfig structs removed
  • CLI flags for hash-table tunables (optional Req 6)
  • 3-node HA live soak verified clean
  • Commits: 17 commits from 0205ac3c through 519e11c7

Jul 22 (Tue) — admin-config-tool ship day

  • All 18 tasks (1.1–5.6) completed, spec marked implementation-complete
  • New binary: xceadmin — Python-based admin config build/push/discovery tool
  • Package: xceadmin.build (YAML→capnp), xceadmin.push (controller push), xceadmin.discovery (auto-detect), xceadmin.cli (CLI entry)
  • Library: xcecommon.schema, xcecommon.validation, xcecommon.mapping, xcecommon.wire
  • Schema fix: Data struct renamed to XceData (unshadow capnp builtin blob type)
  • Wired into meson + nix build, HA deploy/test skills, deploy-ha.sh --build-config
  • Tests: golden-file build determinism, mock-controller push/dry-run coverage
  • Commits: 18 commits from b30a856a through b6bf7cb8

Jul 23 (Wed) — Stabilization + Spec expansion

  • config-upgrade-snapshot spec expanded: 6 requirements, 26 ACs, 9 tasks (8.0–8.9)
  • blocked_on cleared to [] after bug-1418 re-verified FIXED (10-round chaotic soak PASS)
  • bug-1462 fixed: late-joiner config-push stall — ha_promotion_config_tag_floor clamps committed_config_tag to max(safe_tag, self_tag). 20+20 round soaks PASS
  • Limits.yaml wired through cluster_config into controller (real config path)
  • xcecli: restored dangling CA-cert cleanup, enabled https support
  • resource-distribution-mapping spec scaffolded, scope deferred (3 candidate sizes)
  • cc-sdd to Gitea sync skill added
  • bug-1464 identified: xce-build.sh shared build-dir clobber (per-component rm -rf)
  • Commits: 12 commits from bf6a9b34 through cae4ef0c

Jul 24 (Thu) — config-upgrade-snapshot heavy lift

Tasks 8.0–8.5 executed:

  • Task 8.0: HostAdminState/QueueAdminState capnp structs defined, snapshot format version bumped (9e68335a)
  • Task 8.1: Host admin-state frames collected into checkpoint, queue frames migrated to capnp wire (a71aa3f8, 70ff3e5b)
  • Task 8.2: Embed active configuration bytes in every checkpoint — safety gate (387ea35c)
  • Task 8.3: Restore host admin-state frame safely with create-if-missing (5b1600a8)
  • Task 8.4: Configcompare tool + snapverify extended for byte-identical diff (1d09631d)
  • Task 8.5: Add bug-857 regression test, correct design.md (2bd44e8e)
  • Materialization: upgrade snapshot materialized on config-marker apply (1da26b04)
  • Activation fence: hash-verification gate on materialized snapshot; primary serving/promotion gated on fence pass (7b507156, 89b85e16)
  • Removed restart/replay fallback for config activation (41af0540)

Commits: 10 commits from 9e68335a through 1da26b04

Jul 25 (Fri) — Soak testing, bug discovery, task 8.6–8.9

  • Task 8.6: Live config-upgrade soak infra built (run-ha-soak.sh --config-upgrade). Fixed bug-1468 (repl config-push duplicate-connect) and bug-1469 (config tag clamp). Blocked on bug-1472 (out-of-scope HA replication defect for worker admin-status).
  • Task 8.7: Fault-inject crash/restart + leadership change tests (d4af605f)
  • Task 8.8: Overlapping config upgrades supersede cleanly (b78efd03)
  • Task 8.9: Live WAL-trim-under-chaos soak infra, blocked on bug-1474 (8b3e2ab0)

Major bugs discovered (open):

Bug Severity Description
bug-1475 HIGH Standby catch-up liveness stall: WAL_CHECKPOINT at lsn=12 poisons applied-lsn advertisement. Root cause: persist_write_checkpoint_marker does NOT update term index → on_get_entry_term(12) returns 0 → raft library takes ae_append path not ae_conflict_truncate. Primary streams lsn=13 at raft index 12 → ctrl-1 applied_lsn never advances past 12 → infinite ae_reject_logmismatch loop. Full mechanism documented in claude-mem obs 16689–16706.
bug-1476 HIGH load_primary_config concurrent UAF (heap-use-after-free). Two vCPU threads can concurrently free and decode myconf->cluster_config via handle_repl_config_push → ASAN crash in decode_Resource_list. No mutex protecting the read-free-decode sequence.

Build issues:

  • Ninja/Meson parallel-build race (5th recurrence logged)
  • ASAN rebuild: 2 full retries needed, all 1033 tests pass clean after retry2

Commits: 7 commits from 959b0cda through 8b3e2ab0

Jul 26 (Sat, early morning) — Bug fixes for 8.6/8.9 blockers

  • bug-1474 FIXED: two-pass slot allocation (virgin→oldest-disconnect) in repl.c. WAL-retention guard clobber prevented standby slot reuse. 2 tests RED→GREEN. Commit: c0b0c7d1
  • bug-1472 FIXED: worker admin-status replication gap. worker_from_capnp used INSERT semantics (new worker) instead of UPDATE (existing worker). +3 tests. Commit: 7547a8f1
  • 1033 tests all pass (C + Go + Python layers, ASAN clean)

Commits: 2 commits

Spec Status

Spec Phase Status
config-upgrade-snapshot Implementation Tasks 8.0–8.9: 8.0–8.5 complete, 8.6 blocked on bug-1472 (now fixed), 8.7 done, 8.8 done, 8.9 blocked on bug-1474 (now fixed). Tasks 8.6 & 8.9 need soak re-verification. 6 reqs / 26 ACs.
clusterconfig-schema-split COMPLETE All 12 tasks done, live-soak verified
admin-config-tool COMPLETE All 18 tasks done, xceadmin binary shipped
resource-distribution-mapping Scoping Scaffolded, scope deferred

Bug Register (Jul 19–26)

Bug Date Status Description
bug-1452 Jul 19 FIXED commit_engine_fini/persist_stop teardown race
bug-1453/4/5/6 Jul 19–20 FIXED Dead CA-cert plumbing in scheduler/xcecli
bug-1418 Jul 23 VERIFIED FIXED term desync + apply-divergent stall (re-verified with 10-round chaotic soak)
bug-1462 Jul 23 FIXED Late-joiner config-push stall
bug-1464 Jul 23 KNOWN xce-build.sh shared build-dir clobber (mitigation: use /xce-build all)
bug-1468 Jul 25 FIXED Repl config-push duplicate-connect
bug-1469 Jul 25 FIXED Config tag clamp on promotion
bug-1475 Jul 25 OPEN Standby catch-up liveness stall (WAL_CHECKPOINT term-index gap)
bug-1476 Jul 25 OPEN load_primary_config concurrent UAF
bug-1472 Jul 25→26 FIXED Worker admin-status replication (INSERT→UPDATE semantics)
bug-1474 Jul 25→26 FIXED Repl slot-reuse guard clobber (two-pass allocation)

Commits: 63 total (Jul 19–26)

c0b0c7d1 fix(repl): harden standby slot allocation (bug-1474)
7547a8f1 fix(worker): replicate admin host-control status (bug-1472)
8b3e2ab0 feat(config-upgrade-snapshot): task 8.9 WAL-trim-under-chaos soak infra
b78efd03 feat(config-upgrade-snapshot): task 8.8 overlapping config upgrades
d4af605f test(config-upgrade-snapshot): fault-inject crash/restart + leadership (task 8.7)
959b0cda feat(config-upgrade-snapshot): live config-upgrade soak + bug-1468/1469 (task 8.6)
2bd44e8e feat(config-upgrade-snapshot): bug-857 regression test (task 8.5)
1d09631d feat(config-upgrade-snapshot): configcompare tool + snapverify (task 8.4)
4aeae6e2 test(config-upgrade-snapshot): WAL-trim safety + fault-injection seam
41af0540 feat(config-upgrade-snapshot): remove restart/replay fallback
89b85e16 feat(config-upgrade-snapshot): gate primary on activation fence
7b507156 feat(config-upgrade-snapshot): activation fence materialization+hash
1da26b04 feat(config-upgrade-snapshot): materialize upgrade snapshot on config-marker apply
00cd26a9 feat(config-upgrade-snapshot): decode host/queue admin-state on checkpoint load
5b1600a8 feat(config-upgrade-snapshot): restore host admin-state safely
38735ea5 feat(config-upgrade-snapshot): embed config bytes in checkpoint
70ff3e5b feat(config-upgrade-snapshot): migrate queue admin-state to capnp
a71aa3f8 feat(config-upgrade-snapshot): collect host admin-state frames
9e68335a feat(config-upgrade-snapshot): HostAdminState/QueueAdminState capnp structs
e0c5e8b7 fix(nix): pin capnpc source to immutable commit
cae4ef0c chore: save wolf state
c921e59b feat: resource distribution map project
1705495a fix(admin-config-tool): xceadmin config build warnings
4ec40167 fix(xcecli): restore CA-cert cleanup, enable https
ff1be480 merge
b87d5e68 chore: commit wolf status
60c25c39 chore(openwolf): update session memory/buglog/anatomy
8ee98bee docs(config-upgrade-snapshot): expand spec to 6 reqs/26 ACs
bf6a9b34 feat(tooling): cc-sdd to Gitea sync skill
18e7a0e9 feat(controller): wire limits.yaml through cluster_config
0649e17f fix(controller): clamp committed_config_tag on promotion (bug-1462)
ee8473bd feat(admin-config-tool): drop ha/data/, match real client TLS
db2b5f17 fix(admin-config-tool): pin xceadmin python3 to meson resolution
d5330fc9 feat(admin-config-tool): wire xceadmin into HA deploy/test skills
893a01db feat(admin-config-tool): install standalone xceadmin binary via meson
e4fd9e1d feat(admin-config-tool): wire xceadmin into meson + nix build
b6bf7cb8 chore(admin-config-tool): mark implementation-complete (18 tasks done)
a1478510 docs(admin-config-tool): commit spec docs and example configs
67cc3611 feat(admin-config-tool): task 5.6 mock-controller push/dry-run coverage
48a5a0b4 feat(admin-config-tool): task 5.4 golden-file build determinism tests
b5040e08 feat(admin-config-tool): tasks 5.1-5.3 mapping/wire/schema coverage
ad27147a feat(admin-config-tool): task 4.2 deploy-ha.sh --build-config
eeda32c6 feat(admin-config-tool): task 4.1 xceadmin.cli
c33ef7d4 feat(admin-config-tool): task 3.3 xceadmin.push
e53c086e fix(schema): rename Data -> XceData
7ed3a394 feat(admin-config-tool): task 3.2 xceadmin.discovery
72798d4a feat(admin-config-tool): task 3.1 xceadmin.build
77ee831e feat(admin-config-tool): task 2.4 xcecommon.wire
666a838f feat(admin-config-tool): task 2.3 xcecommon.mapping
3a2094a4 merge: job_peek_lchen
dbf5c6c5 feat(admin-config-tool): task 2.2 xcecommon.schema
12b13a54 feat(admin-config-tool): task 2.1 xcecommon.validation
798f19b8 feat(admin-config-tool): task 1.3 nix + xce-build.sh integration
2e4ef4e2 feat(admin-config-tool): task 1.2 shared exception taxonomy
b30a856a feat(admin-config-tool): task 1.1 xceadmin package skeleton
519e11c7 chore(controller): clusterconfig-schema-split mark complete
a74eb008 docs(controller): clusterconfig-schema-split close validation gaps
553da7c7 feat(controller): clusterconfig-schema-split task 6.2 CLI tunables
c3107b95 feat(controller): clusterconfig-schema-split task 6.1 CLI flags
edd79562 chore(controller): clusterconfig-schema-split task 5.3 single-node live-verified
3e01fb1b chore(controller): clusterconfig-schema-split task 5.2 3-node HA soak
f503a43d feat(controller): clusterconfig-schema-split task 5.1 regenerate cluster.bin
29f9b026 chore(controller): clusterconfig-schema-split task 4 build+test verified
ba6fdc57 feat(controller): clusterconfig-schema-split task 3 fix test fixtures
1fa78443 feat(controller): clusterconfig-schema-split task 2.3 remove dead port default
81fecb49 feat(controller): clusterconfig-schema-split task 2.2 remove candidates debug-dump
8f047d93 feat(controller): clusterconfig-schema-split task 1.3 remove orphaned structs
1229f9a9 feat(controller): clusterconfig-schema-split task 2.1 two-tier port fallback
0205ac3c feat(controller): clusterconfig-schema-split task 1.2 remove dead members
51524820 feat(controller): clusterconfig-schema-split task 1.1 remove dead fields
94f50ce6 fix(scheduler,cli): bug-1453/1454/1455/1456 dead CA-cert + HA config
619f41f7 chore: update OpenWolf session state (bug-1452 fix)
d67ccc22 fix(controller): bug-1452 commit_engine_fini/persist_stop teardown race
e1f4d676 chore: update OpenWolf session state (xce-audit-invariants pass)

Next Steps

  1. Soak re-verify tasks 8.6 and 8.9 with bug-1472 + bug-1474 fixes applied. Both blocking bugs now fixed, but soak must pass end-to-end.
  2. bug-1475 needs root-cause fix: persist_write_checkpoint_marker must update term index so on_get_entry_term returns correct term for WAL_CHECKPOINT slot.
  3. bug-1476 needs mutex guard on load_primary_config's myconf->cluster_config read-free-decode sequence.
  4. Task 8.6 blocked_on cleared (bug-1472 fixed), can proceed with worker admin-status rotation verification.
  5. Task 8.9 blocked_on cleared (bug-1474 fixed), can proceed with WAL-trim-under-chaos verification.

Generated 2026-07-26 | Branch: config | 63 commits, 4 bugs fixed, 2 specs shipped, 2 bugs open