Move set_resval()'s resource-name lookup table into a new, dependency-light
resval_typeresolve.c/.h module so it's independently unit-testable (test.c
can't be white-box included into Unity). Adds RED tests for the not-yet-
implemented numeric-fallback/type-override behavior (task 2.1).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ha-config-bootstrap, scheduler-controller-msg-gap, and resval-id-decoupling
were the only 3 specs (of ~40) carrying a nonstandard status:"active"
field, each paired with a note about residual gap-audit findings that
never got refreshed once later work actually resolved them:
- ha-config-bootstrap: all 3 known deviations resolved/superseded
(redirect fields -> libxce-failover; WAL majority-commit scope -> the
deliberate wal_classify() WAIT/RETAIN design; commit-timeout durability
-> bug-661 fix via ha-commit-durability, live-reverified 2026-07-18).
One narrower, genuinely new gap found in passing (OP_upgradeConfig
replies lack redirect wiring, admin-config-tool task 6.3) is flagged
in the note for a future spec, not folded back into this one.
- scheduler-controller-msg-gap: both open items (sched_pend_update_killer
leak, test coverage) were fixed via bug-765 across 4 commits on
2026-06-19, verified directly in current scheduler.c and the test
suite -- the note just never caught up.
- resval-id-decoupling: not just an investigation -- the recommended
name-keyed collect_resval() fix shipped one day after the spec was
created (commit d47b3375), verified as the sole implementation left
in current code. Description no longer named the deleted function;
status:"active" flipped to "closed" and a note added to match its
sibling specs' convention.
All three verified via direct code/commit/buglog inspection (not just
trusting stale spec text) before closing. status:"active" -> "closed"
in all three; phase stays "implemented" (already correct).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Task 6.3's own "before deciding" gate (re-run the flaky primary-push
E2E 20-25x standalone with real stderr capture) was already satisfied
by bug-1485's 2026-08-01 follow-up: 40/40 clean runs, zero captured
crash evidence, after replacing the DEVNULL stderr redirect with real
per-node log capture. Decision reached there (mergeable as-is, bug-1485
downgraded to low-priority/environment-flake) was never reflected back
into tasks.md/spec.json. Marks 6.3 and its parent task-group 6 [x], and
reconciles spec.json's stale "tasks-generated" phase (21/21 tasks were
already complete) to "implementation-complete" with a validate_impl
block recording the known limitations (bug-1485 stays open/low-priority,
single-node ackedCount==0 exit-1 behavior, no CI live-E2E run yet).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- config-upgrade-snapshot, raft-adapter-conformance: correct stale
phase metadata (both showed "tasks-generated" though every task was
already complete) and record their validate-impl GO decisions with
known-limitations notes (bug-1471/bug-1472-remainder, bug-1474
angle-b) for future spec ownership.
- component-failover-adoption: catch up design/requirements/research
docs to the Phase 6 fix (bug-1548/bug-1549, commit d479b707) that
landed after this spec was already marked closed.
- Promote 3 roadmap backlog items to specs (2026-08-10 discovery):
component-rejoin-resync (apiserver/scheduler rejoin resync),
worker-rejoin-resync (worker job-fact resync, hard prereq for
raft-apply-decoupling's facts half), scheduler-sim-diagnostics
(open-ended scheduler REPL debug/troubleshoot capability).
- roadmap.md updated to reflect all of the above.
- kiro-steering/kiro-steering-custom SKILL.md: fix stale template path
(templates/steering/ -> .kiro/settings/templates/steering/) plus
minor doc corrections in the other kiro-spec-* skills.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
XCECLI_FF_CALL (backing xcecli_heartbeat) can silently rotate+reconnect
the transport entirely inside a single call and still return success, so
heartbeat.go's failure-to-success transition detector could never observe
a reconnect that completes within one heartbeat tick -- the common case,
since elections finish in single-digit seconds against a default 30s
interval. apiserver's application-level rejoin never fired on a real
idle-period failover, and job pushes silently stopped reaching it.
Adds a monotonic reconnect_gen counter to endpoint_t (netio.h), bumped by
xce_ctx_reconnect() (xcectx.c) on every successful reconnect regardless of
trigger path. heartbeat.go now compares this generation every tick instead
of tracking success/failure transitions. Re-validated live against task
5.2's exact scenario (apiserver alone, zero write traffic): rejoin fired
22s after a primary kill, 12s after election.
Reopens and closes out component-failover-adoption's own NO-GO -> GO
validation cycle from 2026-08-15.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Final regression sweep (task 5.4) confirms zero new regressions across
the full stack (lib/controller/scheduler/worker/xceadmin/apiserver/xcecli)
and proves the single-candidate path is identical to the multi-candidate
path across all three components (Req 1.5), closing out all 21 tasks
(1.1-5.4) of component-failover-adoption.
Two production bugs found during live E2E validation (5.1/5.2) remain
open and out of this spec's scope: bug-039 (apiserver idle-period rejoin
structurally unobservable on a real single-election failover) and
bug-040 (job pushes not reaching apiserver's local store, likely
downstream of bug-039).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Live E2E of apiserver's backend-health circuit breaker (task 4.5):
killed all 3 controller nodes (not just the primary, unlike 5.1/5.2 --
this scenario needs a call to genuinely exhaust its retry budget, not
successfully rotate to a survivor). A write call burns through the
20-attempt budget in ~3ms (ECONNREFUSED classifies as zero-backoff
XCE_FO_RETRY_NOW, not the ~70-100s backoff track reserved for a
reachable-but-electing controller), returning the real -E_XCE_NOT_READY
(-10013) and triggering MarkDegraded(10s). Immediately after, a second
write call fast-fails in ~1ms via the literal ErrBackendUnavailable
string -- proving the real C call was skipped entirely, not just fast
-- while a concurrent GET /ping (confirmed by source read to never
touch the backend) keeps succeeding normally throughout.
Cross-corroborated across two independently-generated log subsystems
(apiserver's Go zap log and libxce's own C-level retry log) matching
to the millisecond. Bonus: confirmed the 10s TTL's half-open auto-clear
correctly re-attempts the real call afterward.
No src/ changes -- live validation only.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Heartbeat-only idle-period E2E (apiserver alone, zero write traffic
during a 268s+ post-kill observation window) designed as a targeted,
cleaner confirmation experiment for task 5.1's bug-039 finding.
Result: bug-039 reproduces a second time, more cleanly than task 5.1
-- zero rejoin log lines throughout the observation window, and even
a post-window diagnostic write call (which itself succeeds against
the confirmed-new primary) triggers zero rejoin activity, ruling out
"task 5.1 was just unlucky timing." Controller-side logs corroborate
independently: zero "apiserver" references anywhere in the new
primary's log, meaning no join attempt of any kind ever reached it.
This task's own literal pass bar was not met -- approved on the same
basis as task 5.1: the validation itself succeeded rigorously and
honestly surfaced a real, now doubly-confirmed gap in the already-
shipped task 4.3/4.4 rejoin mechanism, which is the task's actual
purpose. A dedicated follow-up to fix heartbeat.go's transition
detection is warranted, separate from this spec's remaining scope.
No src/ changes -- live validation only.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Live 4-process E2E run (3-node controller cluster + xcecd_virtual
worker + xce_scheduler + xce_apiserver, all on the full 3-candidate
list) with the primary killed mid-session. Worker reconnect is
solidly proven with real log evidence and unchanged PIDs across all
three components throughout. Scheduler's rejoin path was never
naturally exercised (no job reached its picker/allocator) -- same
partial-coverage shape already accepted for task 3.3.
Surfaced a likely-real, structural gap in the already-approved task
4.3/4.4 apiserver rejoin mechanism (bug-039): XCECLI_FF_CALL can
silently self-heal the transport within a single heartbeat tick,
making the cross-tick failure->success transition invisible to
heartbeat.go's detector on any failover fast enough to complete
within one heartbeat interval -- which is most real single-election
failovers in this codebase. Also found bug-040 (job pushes never
reach apiserver's store), very likely a downstream symptom of
bug-039's rejoin gap via on_apiserver_join()'s per-process,
unreplicated apiserver registration list. Both logged with detailed,
independently-verified root-cause analysis for a dedicated follow-up
-- fixing them is outside this task's cross-component-validation
boundary.
No src/ changes -- live validation only.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Audit of apiserver's three task-4.6 coverage areas (degraded-to-healthy
transitions, concurrent-request safety, candidate-list parity):
degraded-to-healthy and candidate-list parity were already fully
satisfied by tasks 4.1/4.5's own tests. Found one genuine gap: reconnect.go's
needsRejoin/checkAndPerformRejoin (task 4.3) had zero concurrent-access
test coverage despite being reached from 5 real concurrent HTTP-wrapper
call sites plus the heartbeat goroutine -- health.go's existing
50-goroutine stress test only covered the separate degraded-until
timestamp. One new test closes it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Adds an atomic degraded-until state (health.go: IsDegraded/MarkDegraded/
MarkHealthy) that lets request handlers fail fast with a clear
unavailable error instead of blocking on xcecli's own ~100s retry
budget when the backend is known-degraded.
Consolidates what tasks 4.3/4.4 left as duplicated inline
"call xcecli_*; checkAndPerformRejoin()" pairs across five wrapper
functions into one shared callBackend() helper: checks IsDegraded()
before every call (fail fast, skip the C call and the rejoin check
entirely on the degraded path -- checking rejoin there would itself
block up to ~100s, defeating the fail-fast purpose), marks degraded on
-E_XCE_NOT_READY, marks healthy at checkAndPerformRejoin()'s existing
success point (the single call site both the write path and the
idle-period heartbeat path already funnel through).
The circuit breaker's own TTL-based state is deliberately never read
by the rejoin-retry mechanism -- checkAndPerformRejoin() has no
IsDegraded() dependency, so a fast-fail window never suppresses
retry-until-actually-joined; heartbeat.go's unconditional per-tick
call keeps that cadence alive regardless of degraded state, and a
10s TTL (well under the ~100s retry budget) lets write traffic resume
contributing once it clears.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
xcecli_heartbeat() is XCECLI_FF_CALL-wrapped -- a deliberately
non-retrying fire-and-forget path that does its own best-effort
one-shot reconnect but never calls xcecli_invoke_rejoin() (that only
fires from the write-call retry path). Left as-is, a controller
failover with no in-flight write request would silently repair the
transport connection without ever re-joining, permanently starving
push notifications.
runHeartbeat() gains two callback hooks fired at the right points --
onFailureToSuccess on a genuine transition, onSuccessTick on every
successful tick thereafter -- wired directly to task 4.3's existing
setNeedsRejoin()/checkAndPerformRejoin() from StartHeartbeat(). No
second rejoin mechanism: checkAndPerformRejoin() is a cheap no-op when
nothing is pending, so calling it unconditionally on every successful
tick is safe and matches the requirement that retry isn't gated on the
transition edge alone.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Registers a rejoin flag-setter (go_backend_rejoin_handler) as the
client library's reconnect callback: zero I/O, sets an atomic flag,
returns immediately -- avoiding self-deadlock on the library's
per-ctx failover lock the same way worker's and scheduler's rejoin
callbacks already do. The C shim, its extern declaration, and the
xcecli_set_reconnect_handler() registration call all live together in
clib.go's xce_backend_init(), placed after xcecli_start()'s success
check (xcecli_start() wipes any earlier-registered handler as part of
its own init -- the same ordering bug found and fixed in worker's/
scheduler's task 2.3/3.1 work).
A shared checkAndPerformRejoin() helper wraps every gCtx.ctx-based
backend call (SubmitJob, ControlJob, ControlQueue, ControlHost,
FetchFile): after the call returns, if a rejoin is pending, issue
xcecli_join() as a separate step and clear the flag only on success --
on failure it stays set so the next backend call's return retries it,
no separate retry loop.
needsRejoin is a proper sync/atomic flag rather than a plain bool,
since apiserver (unlike the single-threaded C daemons) serves
concurrent HTTP requests that can all reach the shared helper at once.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Replaces apiserver's two connect mechanisms -- a per-peer Go loop that
hardcoded nmasters=1 per attempt, and a separate direct-construction
path for the legacy single-host config -- with one pipeline:
buildClusterConfig() parses (via task 4.1's C parser) or synthesizes a
single-candidate config, validates it, then ConfigConverter
(xce_cluster_config_to_client_config) produces the client config for
one xce_backend_init() call. A 3-candidate config now actually
exercises the C library's own failover/standby-redirect handling on
initial connect, matching worker's and scheduler's task 2.1/3.1 work,
instead of a Go-level per-candidate retry loop that never built a real
multi-master config.
Along the way: found and fixed a genuine use-after-free. The deleted
tryConnect() freed its C.CString-allocated host address immediately
after its single connect call returned, but xcecli_ctx_init() stores a
raw pointer to the config forever (client.c:38) and every later
failover rotation re-dereferences it (client_failover.c) -- so any
reconnect after the first would read freed memory. Fixed by giving the
cluster/client config structs package-level, process-lifetime storage
(gClusterCfg/gClientCfg), mirroring worker's and scheduler's file-scope
static pattern, with host/from/key written into fixed-size C arrays
instead of freed heap pointers.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Replaces apiserver's hand-written Go parsePeer() with a CGO wrapper
around the shared C parser+validator pair (xce_parse_peers then
xce_cluster_config_validate, both src/lib/cluster.c) that worker and
scheduler already use from their own main.c's. Gives apiserver the same
[name=]host:port[,...] syntax (including the name= prefix the old Go
parser could not express) and identical malformed-candidate rejection
(empty host, port <= 0, non-numeric port) -- round-1 review caught the
first pass skipping the validate() half, silently accepting inputs
worker/scheduler both reject.
A malformed entry now rejects the whole candidate list rather than
being skipped with a warning (matches xce_parse_peers()'s own
all-or-nothing construction and worker/scheduler's precedent) --
surfaced loudly via RegisterToBackend's error, not silently.
Parsing only -- RegisterToBackend()'s connect loop and tryConnect()
are unchanged; collapsing them into a single connect call is task
4.2's job.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Proves task 3.2's retry-until-success-or-shutdown wrapper's guarantees
hold across multiple queued decisions drained through the real
xce_sched_driver_on_decision(), not just a single isolated call:
decisions queued behind a stuck one publish afterward in original
order (Req 7 AC3), and shutdown mid-retry costs at most one retry
cycle regardless of queue depth -- decisions behind the stuck one see
core.stop already true and are never attempted (Req 7 AC2). Also adds
a source-scan confirming this stays scoped to the scheduler's own
publish direction (Req 7 AC5) and a fresh-instance test confirming a
restart still loses in-flight queued decisions, unchanged by this work
(Req 7 AC6).
Pure test-coverage task -- zero production changes, matching the
review's independently-reproduced RED->GREEN evidence that the
existing task-3.2 implementation already satisfies these guarantees.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Adds a worst-case-occupancy test proving the scheduler's picker/allocator
pipeline buffers briefly past a stalled publish (decisions cbuf cap 5,
jbuf cap 1) then genuinely blocks until connectivity recovers, with zero
job loss (Req 3 AC2). Adds a source-scan regression guard confirming the
scheduler makes no periodic heartbeat call of its own (Req 3 AC3).
Also fixes a real gap found along the way: sched_init()'s
all-candidates-unreachable and join-failure log lines named only a
candidate count, never which candidates were attempted (same class of
bug worker's task 2.3 fixed). Added sched_format_candidate_list(),
covered by a real dup2()-based stdout capture around a live sched_init()
failure path (round-1 review caught this diagnostic shipping with zero
test coverage; fixed in round 2).
Multi-candidate join and kill-primary/reconnect were validated manually
against a live 3-node cluster rather than committed as an automated
test, following worker's task 2.3 precedent -- no automated live-deploy
E2E harness exists for any C daemon in this repo.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Wraps sched_publish_dispatch()/sched_publish_pend_reason() in a shared
retry loop that keeps retrying a -E_XCE_NOT_READY-exhausted publish
until it succeeds or the scheduler is shutting down (core.stop checked
before each attempt, not only after), composed with the pending-rejoin
follow-up established in task 3.1. Extended-retry success logs
distinctly from an ordinary first-attempt success (Req 7 AC4).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Scheduler's unified connect pipeline is implemented, tested (RED→GREEN
verified), and live-validated against a 3-node HA controller cluster
(standby-first candidate list joins correctly via xcecli failover). See
a9f283d.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FYSEeLmzjtD4RLDuUDhLoK
Confirm connector thread is isolated from worker_process_update() (main
thread) and resource_sync_thread() (own thread) via code read plus a real
pthread concurrency test. Add unit and live 3-node E2E coverage for
candidate/reconnect behavior, all-candidates-unreachable diagnostics, and
liveness cadence under concurrent reconnect.
The E2E run surfaced two production bugs the 2.1/2.2 unit tests missed:
xcecli_start() resets the reconnect callback to NULL after it runs, so
registering it before xcecli_start() silently no-ops the whole rejoin
mechanism; and a successful worker join returns its assigned slot index
as reply.status (any rc >= 0), not always exactly 0, so checking_connection()'s
rc == 0 success check never fired on a real non-zero-slot join. Both fixed,
plus per-candidate names added to the all-unreachable failure log.
Root-check bypass needed for the E2E run is gated at compile time via the
existing VIRTUAL_CLUSTER flag, not a runtime env var, so the real xcecd
binary's root requirement has zero bypass surface.
Thread cluster_cfg by value through xcecd_config_t -> xcecd_context_t ->
cc_ctx so it's populated before controller_connector_start() fires the
first join, instead of after xcecd_start() returns. checking_connection()
now retries a failed post-reconnect xcecli_join() on the next liveness
probe instead of leaving needs_rejoin cleared on mere attempt.
Replace worker's separate default/single-host and multi-candidate connect
branches with one path: synthesize-or-accept a candidate list, validate,
convert via xce_cluster_config_to_client_config(), connect once. Register
a flag-only rejoin callback (no I/O, no client-lib reentry) and remove the
liveness probe's bounded give-up so the library owns retry/recovery.
Task 1.3's full checklist (constant-equality assertion, candidate
order, one-candidate-no-branch, truncation-with-warning, NULL/empty
rejection) was already satisfied by tasks 1.1 and 1.2's test
coverage. Verified fresh against the current test suite; no
implementer/reviewer cycle needed since there is no new diff.
xce_cluster_config_to_client_config() converts a validated candidate
list into a fully-populated xce_client_config_t, copying candidates
in order with no special-casing for a single candidate. The
truncate-with-warning safety net stays in place as a defensive guard
now that N_MAX_MASTERS and MAX_HA_CANDIDATES match.
Also removes a WARN check in xce_cluster_config_validate() that
became unreachable dead code once task 1.1 reconciled the two
candidate-count constants (both checks tested the same field against
the same constant with no mutation between them).
Task 1.2 of component-failover-adoption (Requirement 1 AC1-AC7).
N_MAX_MASTERS (client.h) now derives from config.h's MAX_HA_CANDIDATES
instead of an independently-maintained literal (3 vs 4 previously).
Fixes worker's and scheduler's hardcoded `> 3` startup checks to
reference the constant symbolically, and updates the client-retry
test fixture so it doesn't exercise an uninitialized 4th slot.
Task 1.1 of component-failover-adoption (Requirement 1 AC7).
These edits (three-thread picker correction in brief.md, record_dir
path-validation and kvmgr-audit requirements in requirements.md,
OPCode_schedAdmin folded into the peer-authentication-hardening
backlog item in roadmap.md) were made during earlier requirements
review and design work but left uncommitted. Reviewed for accuracy
against the completed implementation — all still correct, persisting
as-is.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found by /kiro-validate-impl: tasks 2.1-2.6 (builtin_fcfs picker) were
implemented, reviewed, and committed on 2026-08-03 but never got a
_Verified: line, unlike every other task group. Backfills each from
git history (commits f7921b96, b7214ebe, fcb32a8b, 7ee9a16f, 9d91d470,
604a5649) and direct source inspection.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both files existed only on disk since the design phase (2026-08-03) and
were never committed despite ~40 implementation commits landing since.
Also corrects design.md drift found by /kiro-validate-impl: the File
Structure Plan and BuiltinFcfsPicker sections claimed plugins/fcfs.c
would be rewritten in place, but the actual implementation added a new
plugins/builtin_fcfs.c and deliberately kept fcfs.c untouched as the
job_elect_fn compat-wrapper's exercise target. Updates the directory
tree to the real file layout (core_events/session/clock.c, sim.c/
sim_loop.c/sim_record.c/sim_verify.c, snapshot/recording/repl splits,
test_e2e_capnp.c vs the planned test_replay_format.c, test_sched_admin_dispatch.c
vs test_sched_admin.c), and adds two Open Questions/Risks entries for
the AC 3 tlog-logging-integration gap and this fcfs.c/builtin_fcfs.c
documentation gap, both found during validation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
/kiro-validate-impl pass: GO. Two non-blocking follow-ups found and
documented — AC 3's tlog virtual-clock logging integration is built
but never activated (root-caused to sim.c's sched_sim_driver_start,
deferred pending a matching deactivation/fini design), and
plugins/fcfs.c vs builtin_fcfs.c is functionally correct but
undocumented against design.md and missing _Verified notes on 2.1-2.6.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Clean rebuild (asan, all components) then 15-round sequential and
15-round chaotic HA soak against the fully integrated feature. Both
PASS with zero job loss, zero committed-prefix divergence, zero
liveness stall. Closes scheduler-core: all 45 tasks complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
controller_unit_test crashed deterministically in write_WalCatchupEntry
(test_wal_catchup_append_term_fields_round_trip) -- the same bug class
as bug-1504 (SnapshotData.internalState): this codebase's capnpc-c
backend silently no-ops Data-field encoding in generated encode_XXX()
composite helpers without zero-initializing the caller's local struct
first, leaving an uninitialized capn_ptr written onto the wire.
Confirmed zero production exposure before fixing: xce_wal_batch_encode_
raw()/wal_entry_to_wire() (request.c) never call the generic
encode_WalCatchupEntry() at all -- they memset() their own struct and
call write_WalCatchupEntry() directly with every field hand-set, so
this was 100% test-only exposure (grepped every caller of the generic
encode/decode pair; only the crashing test uses it).
Fix mirrors bug-1504: WalCatchupEntry.payload (@2) and batchPayloads
(@10) switched from Data to List(UInt8) in controller.capnp (new
batchPayloadsLen @11 companion; payload reuses the existing
wirePayloadLen @9), typemap.h's wal_catchup_entry_t field types
updated, and request.c's extract_data_bytes() widened to take a raw
capn_ptr (it already only used .p internally) so its call sites work
regardless of which Data-shaped wrapper the generated accessor
returns now. batchPayloads' single-top-level-blob shape confirmed
this does not reintroduce bug-847's per-element cross-segment
far-pointer loss -- that issue is inherent to per-element data
regardless of Data vs List(UInt8) spelling.
Strengthened test_wal_catchup_append_term_fields_round_trip with real
payload/batchPayloads content assertions -- previously never
exercised, since the test's src fields were left NULL.
Verified: controller suite 3/3 (was 2/3, crashing), scheduler suite
unaffected at 37/37 (unrelated schema area).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPPXeKSUXUqcqvzNXY1rzJ
Adds the final task of "Core: Interactive REPL" (Req 5 AC 22): proves
the REPL's own documented invariant -- "step/continue to completion
produces byte-identical output to batch --sim" (design.md SimDriver
Invariants).
test_repl_equivalence.c runs the same sched_sim_driver_run() on the
same 3-session/9-decision fixture twice: once batch (interactive=
false), once REPL-driven via freopen()'d scripted "step 1\ncontinue\n"
stdin into the real repl_command_loop()/readline() path (same
mechanism test_repl_execution_control.c established) -- then
structurally decodes both runs' --sim-output and compares decision
content. Test-only; zero production code changed.
RED-phase probe (planted in xce_repl_on_event(), reverted, zero
residue) confirmed the comparison is real: corrupting one job's id
under interactive mode broke exactly this test plus the 3 other REPL
tests that share the same hook, none else.
Scheduler suite: 37/37 clean (was 36/36). Task group 12 (Interactive
REPL, tasks 12.1-12.4) is now complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPPXeKSUXUqcqvzNXY1rzJ
Adds `snapshot [FILE]` and `set speed|output|verify ...`/`info config`
(Req 5 AC 22 "Snapshots"/"Simulation parameters") to the interactive
simulation REPL.
`snapshot` reuses sim_record.c's existing sim_record_write_header_and_
snapshot() (task 11.5) rather than reimplementing capnp encoding, so
the produced file is genuinely loadable via a fresh --sim invocation
-- the task's own Observable. `set speed`/`set output`/`set verify`
mutate fields sim_loop.c and core_clock.c already read fresh on every
event/session, so no new hook points were needed and sim_loop.c stays
untouched at exactly 800/800 lines (this project's hard per-file cap).
New repl_params.c is a fourth sibling in the repl.c/repl_break.c/
repl_info.c split (task 12.2's precedent) -- neither existing file had
headroom left under the 800-line cap for this task's own logic.
Scheduler suite: 36/36 clean (was 35/35). Review independently
reproduced the RED-phase probe and additionally verified the
snapshot round-trip Observable is non-vacuous.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPPXeKSUXUqcqvzNXY1rzJ
Adds the interactive simulation REPL's breakpoint and state-inspection
commands (Req 5 AC 22): `break <event_type> [key=value ...]` / `info
break` / `delete <id>|all` on event and decision types, and `info
session/job/queue/worker/resource/summary`.
repl.c (370 lines after task 12.1) couldn't hold this task's full
surface under the project's 800-line cap, so it's split three ways
(matching the snapshot.c/snapshot_wire.c/snapshot_load.c precedent):
repl.c (execution control + dispatch), repl_break.c (matching/CRUD +
the three new driver-facing hooks), repl_info.c (info dumps), sharing
state via repl_internal.h. sim_loop.c gained 3 new bare hook calls
(repl.h inline wrappers, same pattern as task 12.1's two) at the
points design.md's pseudocode requires -- per decoded event before it
joins the batch, per session boundary, and per drained decision --
landing sim_loop.c at exactly 800/800 lines.
Two real bugs found and fixed during development (bug-1506, bug-1507):
an uninitialized xstr_table left breakpoint filter fields silently
NULL instead of failing loud, and repl_breakpoint_free() called a bare
free() on an RTTI-refcounted makeString() string, corrupting the heap.
Review found one gap: `info job` printed only a resreq presence flag
instead of the actual resreq text, when format_resreq() already
existed in this repo for exactly this purpose. Fixed to match.
Scheduler suite: 35/35 clean (was 34/34). RED-phase evidence (2 real
bugs plus 2 synthetic probes, all reverted with confirmed zero
residue) recorded in tasks.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPPXeKSUXUqcqvzNXY1rzJ
Adds the interactive simulation REPL's execution-control slice (Req 5
AC 22): a single-threaded input loop (readline-based) supporting
step [N], continue, quit, and help, pausing before the first replay
event and blocking the driver loop while active.
Review round 1 raised two findings, both resolved here:
- GNU readline (GPL) was linked directly into xce_scheduler, a shipped
production daemon. Switched the whole project to libedit (BSD) for
line editing -- scheduler and controller (xce_test) both now use
editline/readline.h, API-compatible (readline()/add_history()).
- Getting a clean regression run after the libedit swap required a
full clean rebuild, which exposed a genuine pre-existing SIGSEGV
(bug-1504): SnapshotData.internalState was a capnp Data field, and
this codebase's capnpc-c backend silently no-ops Data-field
encoding without zero-initializing the caller's stack struct,
leaving an uninitialized capn_ptr written onto the wire. Fixed by
switching internalState to List(UInt8) (with a new internalStateLen
wire field), matching the working FetchFileReply.content/bytesRead
precedent -- no hand-written accessor code needed.
A second, unrelated instance of the same bug class was found in the
controller's WAL-catchup path (WalCatchupEntry.payload) and
deliberately left unfixed as bug-1505, out of scheduler-core's
boundary; needs its own project-wide Data-field audit.
Scheduler suite: 34/34 clean. RED-phase reproduction recorded in
tasks.md (repl_should_pause() step-budget check disabled -> 2/4
tests fail as expected -> reverted -> 34/34 green).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPPXeKSUXUqcqvzNXY1rzJ
Chain the already-shipped recording (task 9.2) and sim-output (tasks
11.1-11.5) pipelines: record a synthetic multi-queue session through the
normal-mode driver, replay it through the simulation driver with
--sim-output, and assert the two .replay files' decisions match in full
structural content. This is the spec's central validation contract (Req 5
AC 7.1) -- if normal and simulation modes diverge, the core/driver
separation is broken.
Honestly scoped to PEND-only decisions: sim_loop.c's picker thread
(task 11.2) has no code path that can emit a live DISPATCH today, a
pre-existing gap that belongs to the not-yet-designed scheduler-allocation
spec, not something this task can close.
Add sim_record.c (scheduler_t-free .replay writer: ReplayHeader + loaded
SnapshotEntry once, then per-event pass-through and per-session
ReplaySession finalization) and wire --sim-alt-events event-source
selection into sim_loop.c's drain loop, coexisting with task 11.2's
stdout default and task 11.4's --sim-verify. Extract scheduler_t-free
halves out of recording_session.c's begin/end so sim-side code can reuse
the decision-encoding path without gaining an xce_scheduler_t* dependency,
preserving Req 5 AC 17 isolation. Per Req 5 AC 16/19.
Add sim_verify.c (sim_decision_eq/sim_log_mismatch: real structural
comparison of dispatch/pend decision payloads, not a jobid shortcut) and
wire it into sim_loop.c's ReplaySession drain loop via decode_ReplaySession_ptr()
(first real consumer of the generated capnp decode path). --sim-verify now
detects and reports genuine decision divergence with logical-time + detail,
per Req 5 AC 15/18.
Three-layer regression guard (test_sim_isolation.c) proving sim.h/sim.c/
sim_loop.c hold no controller-facing client/cluster-config/capnp-socket
state: source-text scan, struct-body field scan, and nm(1) link-time
undefined-symbol audit of a new isolated sched_sim_isolation_lib target.
Add sim_loop.c implementing sched_sim_driver_run(): decode .replay
messages in file order, jump the virtual clock to each ReplayEvent's
exact timestamp as it's queued, and run a real push-events/go/drain/
end-session cycle on each ReplaySession boundary (AC 14, AC 18).
Task 11.1 left no picker registered against the isolated core and no
thread backing its cbufs, which deadlocks the decision drain the moment
a real event loop exists. Add plugin_register_core() (registers a
picker against a bare xce_sched_core_t*, no xce_scheduler_t* required)
and extract builtin_fcfs_new_plugin() from register_builtin_fcfs() as a
behavior-preserving factory, plus a picker-driving thread for the sim
driver. Since real resource allocation is unreachable without a live
xce_scheduler_t* (scheduler-allocation spec's job), the sim picker
thread conservatively PENDs every candidate rather than dispatching.
28/28 scheduler suite, 32/32 whole-project tests green.
Add sim.h/sim.c implementing AC 13 steps 1-4: open+validate ReplayHeader,
locate the most recent SnapshotEntry at/before --sim-start-ts, load it via
xce_sched_core_load_snapshot(), and set the virtual clock anchors. Wire
all seven simulation CLI flags into main.c's task-5.2 mode-selection
branch. Event loop itself (step 5) deferred to task 11.2. Fixes two real
bugs found during TDD: a SnapshotEntry_get_ts() use-after-free and a
pend_updates NULL-deref in xce_sched_core_fini() for bare-core callers.
27/27 scheduler suite green, full project build clean.
Add real CliRunner-through-real-mock-TLS-server coverage for scheduler
record start/stop/status (incl. standby->primary redirect handoff) and
dump, closing the gap task 10.5 left open (which only monkeypatched the
CLI's function bindings). 227/227 xceadmin tests green. Closes Group 10.
Add scheduler command group with nested record sub-group (start/stop/status)
and sibling dump command, delegating to the client functions from 10.3/10.4
with no business logic in the CLI layer. 220/220 xceadmin tests green.
Adds dump() to scheduler.py -- a pure local .replay file decoder, zero
network I/O. Key finding: pycapnp's from_bytes() does not validate a
message against the schema type you ask it to decode as (trial-decoding
SnapshotEntry bytes as ReplayEvent raises nothing, returns plausible
garbage), ruling out trial-decode-and-catch for message-type
discrimination. Instead reads each message's raw segment-table/
root-pointer framing to extract its (dataWordCount, pointerCount) shape,
computed live from the schema, and matches against the four replay
message types' distinct shapes before decoding. AC 20's stream-ordering
rules (msg 1 = ReplayHeader, msg 2 = SnapshotEntry) are enforced
structurally, not inferred.
Adversarial review reproduced a real (if forward-looking) gap: an actual
modified schema with a shape collision caused an uncaught raw capnp
exception to escape dump(). Closed with a pairwise-distinctness
assertion right after the four shapes are computed, raising a typed
ReplayFileError before any message is decoded if a future schema change
ever makes two shapes collide -- restoring dump()'s own docstring
guarantee unconditionally. New regression test simulates the exact
collision the review found.
201 passed, 3 skipped (up from 188 pre-task).
Adds record_start()/record_stop()/record_status() to new
src/xceadmin/src/xceadmin/scheduler.py, following push_config()'s exact
pipeline shape (load_schema -> follow_redirects -> decode_reply). Deliberately
diverges from push_config()'s "return failure as data" convention because
this task's own Observable text requires raising a typed error on an error
reply, not just connection/decode failure -- a new SchedAdminError
(xcecommon/errors.py) carries the full reply state (error/recording/file/
event_count/started_at) when SchedAdminReply.ok is false.
test_scheduler.py mirrors test_config_push_mock.py's real mock-TLS-server
technique, covering all three actions, redirect-following (standby-to-
primary), and the error-reply-raises-SchedAdminError path across the
error codes reference table.
188 passed, 3 skipped via the real xceadmin test runner (meson compile
-C xce.build xceadmin).
Closes the two genuine gaps beyond task 10.1's own coverage (standby and
disconnected-scheduler rejection were already tested):
- Forward timeout: adds an XCE_TESTING-only timeout-override seam
(SCHEDADMIN_TIMEOUT_MS) in cluster.c, verified inert in production
builds (-DXCE_TESTING absent from xce_controller's target and every
project-wide arg; the non-test branch is a bare 5000L literal).
test_schedadmin_forward_timeout drives the real 5s (shrunk to 200ms
under test) pthread_cond_timedwait to genuine expiry and asserts the
exact {ok=false, error="request timeout"} shape from AC 10.
- Round-trip coverage: two more outcome-branch tests
(rpc_status_only, business_error) using the same real xce_encode/
xce_decode functions production uses. A literal two-binary
controller<->scheduler integration was judged out of scope (separate
meson targets, incompatible fixtures) since task 9.4's own
test_admin_request.c already white-box-drives the real scheduler
handler rigorously.
Controller suite 3/3 (1093/1093 assertions, up from 1090), scheduler
26/26 unaffected, clang-format clean, zero diff-hygiene noise (git diff
matches git diff -w for cluster.c).
Adds the spec's one and only src/controller/ change: a new
case OPCode_schedAdmin in myreq_handler() (cluster.c) that mirrors the
existing OPCode_upgradeConfig case -- primary-only gate, explicit reply
on every path, stateless pass-through with no WAL entry.
Forwards to the scheduler via a new xce_schedadmin_req() (async send,
mirrors xce_fetchfile_req) and blocks the vtask thread on a condvar with
a 5s deadline while holding c->sched.lock for the round-trip, matching
this spec's own research-phase design decision (parity with
OPCode_upgradeConfig's no-RBAC posture, blocking wait accepted for
low-frequency admin traffic). A new branch in the existing
OPCode_reply case demuxes the scheduler's SchedAdminReply to a
process-global single-slot pending-waiter mechanism, serialized by
c->sched.lock, matching job.c's existing fetchfile-proxy pattern.
Adversarial review independently traced double-free/cleanup, the
timeout/signal race, and reply-opcode demux correctness against source;
found and a follow-up round closed a narrow-window leak via a carefully
lock-ordered unlock->unregister->relock sequence (avoiding an AB-BA
inversion against schedadmin_signal()'s lock acquisition order).
New test_sched_admin_dispatch.c drives the real myreq_handler via the
existing myreq_handler_for_test seam: not-primary rejection (proven via
a poison-pointer sched.ep that would crash if forwarding were ever
attempted), not-connected rejection, and a genuine two-thread round-trip
test exercising the real signal/demux path.
Controller suite 3/3 (1090/1090 assertions), scheduler suite 26/26
unaffected, clang-format clean.
