From d683f5fc44733a80a0f9e7edf16b910295e03d12 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Sun, 2 Aug 2026 04:37:28 +0000 Subject: [PATCH] docs: add Jul27-Aug01 2026 project status report Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01LXkenK8rqE8mdS1YKjsq78 --- reports/jul27-aug02-2026.md | 85 +++++++++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 reports/jul27-aug02-2026.md diff --git a/reports/jul27-aug02-2026.md b/reports/jul27-aug02-2026.md new file mode 100644 index 00000000..b4888064 --- /dev/null +++ b/reports/jul27-aug02-2026.md @@ -0,0 +1,85 @@ +# XCE Project Status — Jul 27 – Aug 01, 2026 + +## TL;DR + +HA/Raft correctness work dominated the week: three permanent-wedge bugs closed (catch-up stall, quorum-loss deadlock, config-push race/UAF), `raft-adapter-conformance` spec landed complete, `admin-config-tool` and `config-upgrade-snapshot` both unblocked and near done. One exposed credential fixed. Scheduler work restarted: `scheduler-framework` decomposed into three child specs (`scheduler-core`, `scheduler-allocation`, `scheduler-preemption`), all currently at requirements-only (not yet approved for design). + +## Commits by day + +**Jul 27** +- `add report`, merge `working`→`config`, fix librtti package leak + +**Jul 28** — HA wedge fixes +- fix(ha): close permanent catch-up wedge from duplicated replication-target formula (bug-1478) +- fix(ha): close permanent standby catch-up wedge after WAL_CHECKPOINT boundary (bug-1475) +- fix(config): serialize `load_primary_config` against concurrent invocations (bug-1476) +- fix(ha): retry `cluster_init_phase2` on failure instead of latching `phase2_done` unconditionally (bug-1477) +- docs: re-block config-upgrade-snapshot task 8.6 on bug-1477, then clear stale admin-config-tool blocker (bug-1468 already fixed) + +**Jul 29** — raft-adapter-conformance build-out +- feat: term-adoption wired into primary-side config-push reply handler +- feat: `term` field added to `ReplConfigPushReply` schema +- feat: explicit reply on every config-push outcome +- feat: route post-connect vote/PreVote failures to production tally +- fix(ha): close permanent quorum-loss deadlock on log-conflicted standby (bug-1479) +- test: PreVote post-connect failure routing + fast-quorum coverage; full config-push reply integrity; dormant AppendEntries sender non-regression; WAL catch-up reject reply-construction lock +- chore(wolf): logged bug-1480, bug-1481, bug-1482, bug-1483 (audit findings, not all fixed yet) + +**Jul 30** — conformance close-out + admin-config-tool E2E +- test(admin-config-tool): live single-node E2E test for config push +- fix(admin-config-tool): unwrap Request reply envelope before decoding as RequestReply +- docs(admin-config-tool): task 6.3 blocked pending flake reproduction (bug-1485) +- test(raft-adapter-conformance): batch payload persist-failure propagation regression lock +- docs: confirmed live soaks for tasks 7.1–7.4 (vote/PreVote post-connect failure, config-push rejection); extended controller conformance record; logged bug-1484 + +**Jul 31** +- fix(ha): retry chaotic-mode `verify_classify` before counting job loss (bug-1487) — false-positive job-loss fixed in soak harness, not controller +- fix(security): replace exposed `rshen/Letmein` admin credential in `auth.yaml` with placeholder; added `src/docs/deployment.md` full ops guide +- fix(waldump): decode `WAL_CONFIG_MARKER`, distinguish trimmed gaps from real loss +- docs(config-upgrade-snapshot): host admin-state frames, capnp migration, **bug-1418 blocker cleared** +- docs(raft-adapter-conformance): final design.md/research.md + spec.json state committed — **spec complete** +- **add scheduler related projects** — `scheduler-framework` decomposed into `scheduler-core` (1311-line requirements.md), `scheduler-allocation` (619-line), `scheduler-preemption` (109-line); all requirements-generated, none approved yet +- add wolf state (session memory checkpoint) + +**Aug 01** — no commits yet; working tree has only uncommitted `.wolf/` session-state churn (memory.md, token-ledger.json, hooks/_session.json). + +## Bugs opened/fixed this window + +| Bug | Status | Summary | +|---|---|---| +| bug-1475 | **Fixed** (2026-07-27/28, 2 findings) | Standby catch-up permanently wedged across a WAL_CHECKPOINT boundary — 3-part fix (config_repl.c prev-tracker, ha.c JIT resync, persist.c `TERM_INDEX_UNSET` sentinel) plus a second snapshot-install variant of the same class, closed with the same sentinel | +| bug-1476 | **Fixed** (2026-07-28, re-fixed same day) | Heap-use-after-free race in `load_primary_config` — first landed fix left the lock un-acquired (RED), re-fixed for real with `myconf->lock` + decode-into-local-then-swap | +| bug-1477 | **Fixed** | `cluster_init_phase2` latched `phase2_done` on failure, blocking HA bootstrap retry | +| bug-1478 | **Fixed** | Duplicated replication-target formula caused permanent catch-up wedge | +| bug-1479 | **Fixed** | Permanent quorum-loss deadlock on log-conflicted standby | +| bug-1480 | **Fixed** (2026-07-29 09:43, `b7828eed`) | Config-push reply missing explicit reject+term — fixed same day, buglog `fix:` field was stale (written 07:05, before landing) until corrected 2026-08-01 | +| bug-1481 | **Fixed** (2026-07-29 13:53, `fce73b5b`) | `vote_ep_err_handler` bypassed production tally on post-connect failure — fixed same day, buglog `fix:` field was stale until corrected 2026-08-01 | +| bug-1482 | **Fixed** | Not a real UAF — erroneous `free_packet` call in a test (pkt already owned/consumed by `xce_decode`); test + doc comment corrected | +| bug-1483 | Open | Pre-existing test-mode leak in `ha_arm_heartbeat_timeout_vtask` (c==NULL calloc never freed); workaround only (stub+reset hook in every LOST-path test), not fixed at root | +| bug-1484 | Open, explicit residual | No test drives real `repl_ack_decode_and_dispatch` term-adoption for heartbeat-reply/WAL-NACK vectors; Req 7.5 "already verified" claim rests on unverified ground; out of scope for raft-adapter-conformance, needs a follow-up task | +| bug-1485 | Open, flaky, 24 occurrences | `test_live_config_push_against_primary_reaches_success_path` — admin-config-tool task 6.3 blocked on this | +| bug-1486 | Documented (workflow, no code fix) | Concurrent `xce-build.sh` sessions clobber shared build dir (rm -rf race) — mitigation: check `ps auxf` before trusting a stale background build log | +| bug-1487 | **Fixed** | 30-round chaotic soak false-positive job-loss — actually a 6.2s Figure-8 commit-barrier window, not real loss; fix adds bounded retry to `verify_classify` in the soak harness | + +## Spec status changes + +- **raft-adapter-conformance** — requirements/design/tasks all approved, `ready_for_implementation`, `blocked_on: []`. Final design.md/research.md committed Jul 31. Effectively **done**. +- **admin-config-tool** — both historical blockers (bug-1468 dispatch-inversion, bug-1461 capnp `Data` shadowing) now marked RESOLVED; task 6.3 HA E2E test still blocked on a flaky-test repro (bug-1485). +- **config-upgrade-snapshot** — `blocked_on` cleared to `[]` (bug-1418 confirmed fixed, no regression); host admin-state frames and capnp migration work added. +- **scheduler-framework** — formally decomposed (phase: `decomposed`) into three child specs, all at `requirements-generated` / not approved: + - `scheduler-core` — 1311-line requirements.md + - `scheduler-allocation` — 619-line requirements.md + - `scheduler-preemption` — 109-line requirements.md + - Next step for this line of work is design approval on whichever child spec proceeds first. + +## Notable non-bug items + +- **Security**: a live, checked-in `cluster_admin` credential (`rshen/Letmein`) in `auth.yaml` was replaced with a placeholder — flag for credential rotation if that value was ever deployed anywhere real. +- **New doc**: `src/docs/deployment.md` — full ops guide for building/running all four daemons + xceadmin, written from direct source verification + a live single-node deploy; explicitly documents production gaps (no process supervision, plaintext non-controller traffic, exposed dev creds). +- **waldump**: now decodes `WAL_CONFIG_MARKER`, so trimmed-segment gaps are no longer misreported as real data loss during investigations. + +## Open threads heading into next session + +1. `scheduler-core`/`scheduler-allocation`/`scheduler-preemption` need requirements review + design phase — nothing is implementation-ready yet. +2. bug-1480/1481/1482/1483/1484/1485 remain open (raft-adapter-conformance audit residue + one ASan UAF in netio.c). +3. Uncommitted `.wolf/` session-state changes sitting in working tree (harmless, not code).