test_live_config_push_against_standby_returns_standby_rejection_not_redirect
locked in the OLD controller behavior (a standby flatly rejects
OP_upgradeConfig with no redirect fields). That gap is now closed:
myreq_handler's gate-rejection path calls fill_redirect_reply() on
every -XCE_ERR_STANDBY rejection (bug-1700), and handle_ctrlhello() now
actually populates primary_host/primary_port from the static --peers
list (bug-1701) -- a standby pushing config now redirects to the
primary and the push succeeds. Renamed to
test_live_config_push_against_standby_redirects_and_succeeds and
rewritten to assert the new behavior.
_assert_primary_committed() changed from a single-shot log read to a
poll loop (up to 5s) for the "config activated from committed
marker..." line -- accounts for the documented async gap between the
majority-ack-gated client reply and the primary's own local WAL-apply-
pump activation, a real race the old single-shot read could hit.
meson-build.sh: wire src/tools and each build profile's controller
subdir into the isolated repo-mirror the E2E tests run against, and
export XCEADMIN_BIN explicitly, so these live tests actually find real
binaries instead of always skipping.
Verified: xceadmin suite 288/288 (incl. both tests in this file);
controller suite 1220/1222 (only the pre-existing bug-1561 flake).
Updated admin-config-tool design.md and roadmap.md to mark the
dispatch-gate and redirect-wiring gaps RESOLVED, matching this test's
new assertions and the live evidence.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
Adds the previously-uncommitted spec.json/requirements.md/design.md/research.md
for scheduler-snapshot-job-allocation (feature now fully implemented across
tasks 1.1-4.5) and the in-progress xce-test-cli-ux spec's requirements.md/
spec.json. Updates brief.md, a security.md note on an unrelated deferred RBAC
persistence gap found during this feature's cross-component survey, and syncs
OpenWolf's anatomy/memory/token-ledger/daemon-log bookkeeping.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SSuXfK6nD1Nue5EjjdhjNZ
Prompted by rediscovering the same bug class in two places this
session: the controller's WAL already has this documented for one
mechanism (persistence.md's APPLY_CLEAN/bug-1678 section, xce-audit-
invariants controller dimension 7) but the scheduler's separate
snapshot save/load mechanism had the identical gap (job->allocs, this
session's scheduler-snapshot-job-allocation brief) with no equivalent
tracking. Closes that gap and makes the invariant self-reinforcing
going forward rather than something that has to be independently
rediscovered per persistence mechanism:
- xce-audit-invariants: new scheduler dimension 4 ("Field-complete
snapshot save/load parity"), sibling of controller dimension 7,
scoped to snapshot.c/snapshot_wire.c/snapshot_load.c.
- persistence.md: new "Snapshot Save/Load Field-Completeness
(Scheduler)" section cross-referencing the WAL-side bug-1678 writeup
and the new scheduler dimension.
- CLAUDE.md: new top-level Key Rules bullet stating the invariant
plainly, plus adding persistence.md to the steering-file pointer
table (it was missing).
- kiro-spec-design's design-review-gate.md (the mandatory pre-write
gate) and kiro-validate-design's design-review.md (the optional deep
review): new conditional review sections -- applies only when a
design touches a save/restore mechanism, requires enumerating every
persisted field's capture status rather than trusting an aggregate
"state looks right" check. Catches this class of gap at design time,
before implementation, rather than only via a later audit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
/kiro-discovery investigation of removing src/frontend/Makefile in favor
of Meson as the sole build system. Records the quick investigation's
findings (5 boundary seams, current-state evidence that build-cli/
build-server are already dead from /xce-build's perspective and
build-server is already broken today) and the user's confirmed decision
to fully remove the Makefile rather than keep a compat wrapper.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
CLAUDE.md: replace the stale @.wolf/OPENWOLF.md import with a pointer to
the openwolf skill / rules file, matching how the project's OpenWolf
setup actually resolves the protocol now.
api-standards.md: add checklist item 6 for new primary-only opcodes --
confirm ha_dispatch_gate's -XCE_ERR_STANDBY rejection actually populates
a redirect via fill_redirect_reply(), not just that the client has a
generic redirect-follower. Directly informed by bug-1700, where the
identical gap was independently rediscovered for OPCode_schedAdmin after
already being documented once for OPCode_upgradeConfig.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
persistence.md's APPLY_CLEAN verdict only checks entity count and apply-
cursor position, not per-field completeness -- bug-1678 (job allocation
silently dropped on restart replay despite APPLY_CLEAN reporting healthy)
fell through exactly this gap. Documents the sharper invariant online-
apply and replay-apply must produce field-complete equivalent state, not
just matching counts/cursor, and adds it as a standing audit dimension
(#7) to xce-audit-invariants' controller dispatch list.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Elevates the normal-mode/simulation-mode driver equivalence guarantee
from a one-line architectural description in CLAUDE.md to an enforced
steering constraint (scheduler-driver-equivalence.md), backed by
bug-1611/bug-1613 as concrete evidence of violation. Scopes the fix as
its own spec (scheduler-sim-decision-parity, requirements + design)
rather than reopening the closed scheduler-core spec or folding it
into scheduler-sim-repl-verbosity's explicitly narrower boundary.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Full spec scaffold (requirements/design/research/tasks all approved,
ready_for_implementation=true) plus a roadmap backlog item recorded from
the dirty-content-fingerprint spec's task 4.3 findings (Makefile/meson
fingerprint-computation duplication risk).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Surfaced while investigating a user request to show total/current values
for GLOBAL/SHARED resources in the scheduler REPL's "info resource"
command -- researched and confirmed that concept doesn't exist anywhere
in this codebase (resourcemap config field dropped at build time, capnp
fields commented out, worker.c actively rejects XCE_RESF_SHARED resources
today). This spec already owns closing that gap; it just never made it
into the roadmap's backlog list.
- xce_test shell command support: was already implemented (xce-test-e2e-
primitives task 2.1, closed 2026-06-19) -- stale "not started" entry
corrected.
- scheduler-core tasks 2.1-2.6 _Verified: backfill: was already done
(commit d7b49690, 2026-08-08) -- corrects a status line I myself wrote
incorrectly earlier today by trusting the original 2026-08-07 finding's
wording without re-checking the file.
- job_ctrl direct-mutate-path investigation: completed. Real risk
confirmed (pre-commit state pushed live to scheduler + apiserver/clients
with no rollback on commit failure), scope wider than originally framed
(on_job_stop/on_job_resume/on_job_ckpt share the same pattern, fed by
ordinary traffic not just rare admin commands), plus a separate
read-lock-during-mutation bug found along the way. Not promoted to a
spec yet -- scope/framing corrected first per this session's decision.
- OP_upgradeConfig standby redirect verification: investigated. The
item's own premise doesn't match production (a standby never actually
returns a redirect for this opcode -- a documented, deliberate feature
gap, not a missing test); coverage confirmed already complete on both
the client mock side and a live 3-node regression test.
design.md's own "Open Questions/Risks" list (items 5 and 6) still described
the OPCode_schedAdmin admission gap and the tlog activation gap as open/
deferred, even though both were fixed earlier in this session -- marked
FIXED with pointers to the actual change. research.md's schedAdmin
decision record and its Risks & Mitigations bullet got the same treatment
(SUPERSEDES note, historical record preserved rather than rewritten).
tasks.md's 2026-08-07 GO/NO-GO follow-up list gained a status-update line:
follow-up 1 (tlog) is now fixed, follow-up 2 (fcfs.c/builtin_fcfs.c
traceability) is only partially closed (design.md text was already
correct; tasks 2.1-2.6 still lack _Verified: lines -- flagged, not
fixed here since I have no fresh verification evidence for those tasks).
roadmap.md's scheduler-core summary line corrected to attribute each gap
to the right source (2026-08-07 validate-impl vs today's separate
design-review pass) instead of conflating them.
Every early-exit branch in the worker_list scan either re-tested the same
node forever (bare `continue`) or returned out of the whole function on the
first DONE/EXITED worker, so job array elements past the head of the list
never got their on_job_start report. Capture `next` before touching each
node and always advance to it. Affects xcecd_virtual and real xcecd alike
(same DBUS_MANAGER-gated function).
Also records a roadmap follow-up (xcecd_virtual concurrent-job support,
MXJ > 1) discovered while investigating this bug.
- config-upgrade-snapshot, raft-adapter-conformance: correct stale
phase metadata (both showed "tasks-generated" though every task was
already complete) and record their validate-impl GO decisions with
known-limitations notes (bug-1471/bug-1472-remainder, bug-1474
angle-b) for future spec ownership.
- component-failover-adoption: catch up design/requirements/research
docs to the Phase 6 fix (bug-1548/bug-1549, commit d479b707) that
landed after this spec was already marked closed.
- Promote 3 roadmap backlog items to specs (2026-08-10 discovery):
component-rejoin-resync (apiserver/scheduler rejoin resync),
worker-rejoin-resync (worker job-fact resync, hard prereq for
raft-apply-decoupling's facts half), scheduler-sim-diagnostics
(open-ended scheduler REPL debug/troubleshoot capability).
- roadmap.md updated to reflect all of the above.
- kiro-steering/kiro-steering-custom SKILL.md: fix stale template path
(templates/steering/ -> .kiro/settings/templates/steering/) plus
minor doc corrections in the other kiro-spec-* skills.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
These edits (three-thread picker correction in brief.md, record_dir
path-validation and kvmgr-audit requirements in requirements.md,
OPCode_schedAdmin folded into the peer-authentication-hardening
backlog item in roadmap.md) were made during earlier requirements
review and design work but left uncommitted. Reviewed for accuracy
against the completed implementation — all still correct, persisting
as-is.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bug-1463: commit 94f50ce6 (bug-1453 CA-cert-plumbing removal) missed
GetJobOutput's dead tls.Config/loadRootCAs block in request.go, plus
peek.go's SetRootCAs call and serverInfo.CACert field access (the
field no longer exists on APIServerInfo) -- broke the xcecli build.
Removed all three dangling references. Also enabled NewAPIClient's
previously-commented-out InsecureSkipVerify TLS config, since no
xcecli method set any TLS config at all -- any https:// apiserver URL
would have failed cert verification, not just GetJobOutput. Matches
bug-1453's documented design (server-side TLS only, no peer
verification, mTLS deferred as P3 hardening).
Documented both the api-server (http+https) and controller
(server-side-cert-only, no client certs) TLS postures in
.kiro/steering/security.md, replacing a stale line claiming TLS is
handled at the network layer, plus a follow-up note for the planned
HAProxy TLS-termination reverse proxy once its cert type and
apiserver's own listener fate are decided.
- ha-async-io.md: document the same-batch endpoint lifetime hazard
(bug-1447) and its ref-whole-batch-before-dispatch fix pattern
- xce-audit-invariants SKILL.md: add object-lifetime-within-a-dispatch-batch
as a 6th audit dimension, so a future pass checks other batch loops
(vtaskmgr drain, peer-list iteration) for the same shape
- product.md: replace the stale "HA consistency family is the current
focus" framing with the actual 2026-07-18 status (umbrella complete,
9-spec validate-impl batch closed, job-array-single-wal-record shipped)
- roadmap.md: correct 5 entries that were stale relative to actual spec
status (ha-consistency-invariants/ha-wal-replication/libxce-failover
pulled out of "In Flight", ha-term-safety and raft-drive-async marked
implemented+GO, raft-sim-conformance and job-array-single-wal-record
given closure annotations) -- struck through, not deleted, per this
file's own additive-update convention
- testing.md: document that meson's aggregate Fail:0 can mask Unity
sub-group failures -- always check the standalone binary's own summary
- src/controller/CLAUDE.md: fix a stale line-citation for the votedFor
atomicity invariant (had drifted onto unrelated post-election code)
bug-1433: raft_cb_become_leader runs synchronously off peer_ctrl_dispatch_to_pool,
which hardcodes VTASK_LIVENESS for every inbound peer-control packet including
vote replies. The bug-1431 round-2 fix (flusher_flush_once force-flush in
persist_promotion_advance_committed) was reachable from that never-blocks task
class -- same bug family as bug-1198. Removed the force-flush; round-3
(durable-capped only) plus the already-shipped bug-1432 immediate-first-heartbeat
fix close the visibility-latency gap non-blockingly. TRACKED RESIDUAL:
persist_apply_pump_run_sync is still reachable on the same VTASK_LIVENESS path
and does bounded pread() -- lower risk (page-cache-hot) but strictly still
blocking I/O; not safe to remove outright (reopens bug-1043-family job loss),
needs its own follow-up (reclassify peer_ctrl_classify's winning-vote-reply case
or dispatch promotion tail onto a fresh VTASK_LIVE_BLOCK vtask).
bug-1434: persist_promotion_advance_committed and ha_standby_advance_committed_and_apply
raced on persist_t.committed_lsn via an unlocked load-then-conditional-store during
the promotion window (before ha->role flips to CTRL_PRIMARY) -- a losing racer could
regress committed_lsn. Added persist_committed_lsn_advance_atomic (CAS retry loop,
lock-free) and routed both call sites through it.
Validated: 19/19 unit test suites, 6-round chaotic soak (0 loss/divergence) + 6-round
sequential+leadership-coverage soak, both PASS. Also folds in raft-apply-decoupling
and raft-ha-core-safety validate-impl phase bumps to "implemented" and clears
config-upgrade-snapshot's now-satisfied entity-bearing-snapshots block.
raft-apply-decoupling: 32/32 tasks complete, validate-impl GO (79/79 test
groups, live smoke boot verified) -- update roadmap's In Flight entry with
outcome, residuals, and follow-ups.
raft-drive-async: all 17 sub-tasks were already [x] but the 4 phase-header
checkboxes were never ticked, leaving tasks.md looking incomplete. Confirmed
via source (g_drive_async_configpush_offload defaults to 1) and prior
session notes that all 4 phases genuinely landed and flipped ON in
production; checkboxes now match reality.
Adds the Raft Apply-Ordering & Hook Decoupling roadmap phase: canonical
pre-hooks -> raft commit -> apply -> notify ordering, with the raft protocol
step lifted out of the notification hook fan-out (root of the bug-795/828/
851/1060 divergence family). Decision: apply-after-commit via a single
shared apply_committed_entry() seam reused by online apply and restart
replay.
- raft-drive-async: full spec docs (brief/requirements/research/design)
committed alongside its already-tracked tasks.md; async-drive prerequisite
for the apply-decoupling work, sequenced to start first (threatens
failover liveness today).
- raft-apply-decoupling: new spec (brief + facts-in-raft.md) covering the
apply seam and the facts (WAL_MODE_RETAIN) apply-timing question (open
decision D1).
- job-array-single-wal-record: promoted from backlog to a tracked spec,
sequenced after raft-apply-decoupling.
- ha-consistency-invariants: add criterion 2.9 (Facts-Apply invariant),
documenting the current on-receipt fact-reaction deviation this family
will reconcile.
- raft-commit-safety / raft-log-replication: reconcile spec.json phase to
"implemented" (both were mislabeled tasks-generated despite full 18/18
and 24/24 task completion + raft_sim GO).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
Per design invariant (a controller exits only on crash or admin kill), replace the
self-abort recovery terminal (tasks 3.3/4.3/4.4) with a loud, permanent, non-promotable
fence that stays alive and keeps retrying catch-up (self-heals on peer return; no manual
admin recovery). Records the never-self-terminate invariant as project-wide guidance in
src/controller/CLAUDE.md + .kiro/steering/ha-async-io.md.
Full controller suite green under ASAN (508 PASS / 0 FAIL / 0 sanitizer
aborts, independently re-censused). All 16 tasks complete; barrier
before reply proven across all restructured opcodes; classification
guard active; barrier-before-state-removal precondition for
ha-single-leader-role satisfied. spec phase=implemented; roadmap
updated. Sub-spec 2 of 4 of the ha-consistency-invariants umbrella.
