myxce/reports/jul27-aug02-2026.md

XCE Project Status — Jul 27 – Aug 01, 2026

TL;DR

HA/Raft correctness work dominated the week: three permanent-wedge bugs closed (catch-up stall, quorum-loss deadlock, config-push race/UAF), raft-adapter-conformance spec landed complete, admin-config-tool and config-upgrade-snapshot both unblocked and near done. One exposed credential fixed. Scheduler work restarted: scheduler-framework decomposed into three child specs (scheduler-core, scheduler-allocation, scheduler-preemption), all currently at requirements-only (not yet approved for design).

Commits by day

Jul 27

• add report, merge working→config, fix librtti package leak

Jul 28 — HA wedge fixes

• fix(ha): close permanent catch-up wedge from duplicated replication-target formula (bug-1478)
• fix(ha): close permanent standby catch-up wedge after WAL_CHECKPOINT boundary (bug-1475)
• fix(config): serialize load_primary_config against concurrent invocations (bug-1476)
• fix(ha): retry cluster_init_phase2 on failure instead of latching phase2_done unconditionally (bug-1477)
• docs: re-block config-upgrade-snapshot task 8.6 on bug-1477, then clear stale admin-config-tool blocker (bug-1468 already fixed)

Jul 29 — raft-adapter-conformance build-out

• feat: term-adoption wired into primary-side config-push reply handler
• feat: term field added to ReplConfigPushReply schema
• feat: explicit reply on every config-push outcome
• feat: route post-connect vote/PreVote failures to production tally
• fix(ha): close permanent quorum-loss deadlock on log-conflicted standby (bug-1479)
• test: PreVote post-connect failure routing + fast-quorum coverage; full config-push reply integrity; dormant AppendEntries sender non-regression; WAL catch-up reject reply-construction lock
• chore(wolf): logged bug-1480, bug-1481, bug-1482, bug-1483 (audit findings, not all fixed yet)

Jul 30 — conformance close-out + admin-config-tool E2E

• test(admin-config-tool): live single-node E2E test for config push
• fix(admin-config-tool): unwrap Request reply envelope before decoding as RequestReply
• docs(admin-config-tool): task 6.3 blocked pending flake reproduction (bug-1485)
• test(raft-adapter-conformance): batch payload persist-failure propagation regression lock
• docs: confirmed live soaks for tasks 7.1–7.4 (vote/PreVote post-connect failure, config-push rejection); extended controller conformance record; logged bug-1484

Jul 31

• fix(ha): retry chaotic-mode verify_classify before counting job loss (bug-1487) — false-positive job-loss fixed in soak harness, not controller
• fix(security): replace exposed rshen/Letmein admin credential in auth.yaml with placeholder; added src/docs/deployment.md full ops guide
• fix(waldump): decode WAL_CONFIG_MARKER, distinguish trimmed gaps from real loss
• docs(config-upgrade-snapshot): host admin-state frames, capnp migration, bug-1418 blocker cleared
• docs(raft-adapter-conformance): final design.md/research.md + spec.json state committed — spec complete
• add scheduler related projects — scheduler-framework decomposed into scheduler-core (1311-line requirements.md), scheduler-allocation (619-line), scheduler-preemption (109-line); all requirements-generated, none approved yet
• add wolf state (session memory checkpoint)

Aug 01 — no commits yet; working tree has only uncommitted .wolf/ session-state churn (memory.md, token-ledger.json, hooks/_session.json).

Bugs opened/fixed this window

Bug	Status	Summary
bug-1475	Fixed (2026-07-27/28, 2 findings)	Standby catch-up permanently wedged across a WAL_CHECKPOINT boundary — 3-part fix (config_repl.c prev-tracker, ha.c JIT resync, persist.c TERM_INDEX_UNSET sentinel) plus a second snapshot-install variant of the same class, closed with the same sentinel
bug-1476	Fixed (2026-07-28, re-fixed same day)	Heap-use-after-free race in load_primary_config — first landed fix left the lock un-acquired (RED), re-fixed for real with myconf->lock + decode-into-local-then-swap
bug-1477	Fixed	cluster_init_phase2 latched phase2_done on failure, blocking HA bootstrap retry
bug-1478	Fixed	Duplicated replication-target formula caused permanent catch-up wedge
bug-1479	Fixed	Permanent quorum-loss deadlock on log-conflicted standby
bug-1480	Fixed (2026-07-29 09:43, b7828eed)	Config-push reply missing explicit reject+term — fixed same day, buglog fix: field was stale (written 07:05, before landing) until corrected 2026-08-01
bug-1481	Fixed (2026-07-29 13:53, fce73b5b)	vote_ep_err_handler bypassed production tally on post-connect failure — fixed same day, buglog fix: field was stale until corrected 2026-08-01
bug-1482	Fixed	Not a real UAF — erroneous free_packet call in a test (pkt already owned/consumed by xce_decode); test + doc comment corrected
bug-1483	Open	Pre-existing test-mode leak in ha_arm_heartbeat_timeout_vtask (c==NULL calloc never freed); workaround only (stub+reset hook in every LOST-path test), not fixed at root
bug-1484	Open, explicit residual	No test drives real repl_ack_decode_and_dispatch term-adoption for heartbeat-reply/WAL-NACK vectors; Req 7.5 "already verified" claim rests on unverified ground; out of scope for raft-adapter-conformance, needs a follow-up task
bug-1485	Open, flaky, 24 occurrences	test_live_config_push_against_primary_reaches_success_path — admin-config-tool task 6.3 blocked on this
bug-1486	Documented (workflow, no code fix)	Concurrent xce-build.sh sessions clobber shared build dir (rm -rf race) — mitigation: check ps auxf before trusting a stale background build log
bug-1487	Fixed	30-round chaotic soak false-positive job-loss — actually a 6.2s Figure-8 commit-barrier window, not real loss; fix adds bounded retry to verify_classify in the soak harness

Spec status changes

• raft-adapter-conformance — requirements/design/tasks all approved, ready_for_implementation, blocked_on: []. Final design.md/research.md committed Jul 31. Effectively done.
• admin-config-tool — both historical blockers (bug-1468 dispatch-inversion, bug-1461 capnp Data shadowing) now marked RESOLVED; task 6.3 HA E2E test still blocked on a flaky-test repro (bug-1485).
• config-upgrade-snapshot — blocked_on cleared to [] (bug-1418 confirmed fixed, no regression); host admin-state frames and capnp migration work added.
• scheduler-framework — formally decomposed (phase: decomposed) into three child specs, all at requirements-generated / not approved:
  • scheduler-core — 1311-line requirements.md
  • scheduler-allocation — 619-line requirements.md
  • scheduler-preemption — 109-line requirements.md
  • Next step for this line of work is design approval on whichever child spec proceeds first.

Notable non-bug items

• Security: a live, checked-in cluster_admin credential (rshen/Letmein) in auth.yaml was replaced with a placeholder — flag for credential rotation if that value was ever deployed anywhere real.
• New doc: src/docs/deployment.md — full ops guide for building/running all four daemons + xceadmin, written from direct source verification + a live single-node deploy; explicitly documents production gaps (no process supervision, plaintext non-controller traffic, exposed dev creds).
• waldump: now decodes WAL_CONFIG_MARKER, so trimmed-segment gaps are no longer misreported as real data loss during investigations.

Open threads heading into next session

1. scheduler-core/scheduler-allocation/scheduler-preemption need requirements review + design phase — nothing is implementation-ready yet.
2. bug-1480/1481/1482/1483/1484/1485 remain open (raft-adapter-conformance audit residue + one ASan UAF in netio.c).
3. Uncommitted .wolf/ session-state changes sitting in working tree (harmless, not code).