Task 12.5 (REPL live-operator-testing fixes) was implemented and its bugs
fixed in an earlier commit, but the tasks.md checkbox was never flipped and
carried no _Verified: line, unlike every other completed task in this spec.
Ran /kiro-verify-completion against it with fresh evidence rather than
trusting the earlier self-report: fresh build+full suite (41/42, only the
pre-existing unrelated bug-1560 failing), a live subprocess smoke test of
the real xce_scheduler binary for all five amendment gaps (including a
hand-built .replay fixture with resource data to close the one gap the
first pass left unit-tested-only), and direct source inspection of each
fix. All five confirmed correct; checkbox flipped, _Verified: line added.
Two retrospective design-review findings:
- design.md's System Flows sequence diagram showed the controller-layer
resval check as an independently-launched client session, implying no
dependency on the injecting worker staying connected. As-built (bug-1579),
host-attrs resources reset to baseline the instant the worker disconnects,
so the check actually runs nested inside the still-open worker session via
xce_test's `!` shell-escape. Updated the diagram and prose to show this,
and documented the still-open residual risk for custom host-attrs
resource names in Error Handling.
- Added explicit "do not change this format" warning comments at both
trace-log call sites (core_events.c's RESVALS_DELTA case, apiserver's
OnResourceLocationUpdated) that xce-test-resources' log-grep depends on --
an unrelated logging cleanup at either site would otherwise silently turn
every future verification run into a false UNREACHABLE/timeout.
No behavior change in either source file -- comments only.
The schedAdmin dispatch case blocks its vtask handler thread for up to 5s
waiting on a cross-process reply while holding c->sched.lock -- bounded
blocking on a vtask thread, which ha-async-io.md permits only under an
admission reservation. The real risk isn't just the wait itself: since
c->sched.lock serializes concurrent requests one at a time, a burst of N
concurrent schedAdmin calls could already occupy N vtask threads for up to
N*5s each before this fix.
Now acquires vtaskmgr_blocking_acquire(&c->vtaskmgr) before ever locking
c->sched.lock, releasing on every exit path -- the same shared admission
budget commit.c's commit-wait barrier already gates on (wired via
commit_set_pool). Admission-exhausted requests fast-reject with
{ok=false, error="server busy, try again"} rather than queuing. New test:
test_schedadmin_fast_rejects_when_admission_exhausted, using a poison
c->sched.ep sentinel to prove the fast-reject happens before any
lock/forward attempt.
Bugs fixed, discovered across a day of live-operator testing against a
real deployed cluster:
- bug-1584: clock anchors never initialized to real wall-clock values,
timestamps showed system uptime instead of Unix epoch.
- bug-1585: heap-use-after-free in the normal-mode driver's periodic
resval update path -- core_events.c's RESVALS_FULL/_DELTA handlers do a
shallow zarray_put() of the wire-decoded xce_val_t*, but driver.c's
release_translated_payload() unconditionally freed the same decoded
array right after, so the next periodic report for the same
worker+resource triggered a UAF read via kill_sched_resval(). New
regression test: test_resval_ownership.c.
- bug-1587/1588/1591: raw-ns/raw-uint64 jobid/timestamp prints across the
REPL/SIM family converted to the same human-readable formatting already
used elsewhere (fmt_jobid(), timestamp formatting).
- bug-1590: repl_info.c's XCE_VAL_NUMERIC checks corrected to
Value_u_fvalue, fixing `info worker`/`info resource` falling back to
0.00 for real numeric resvals.
task 12.5 (Req 5.22 amendment) -- five REPL gaps surfaced by a live
operator session, each with its own bug id:
- bug-1594: `continue`/`step` reaching EOF with no breakpoint hit now
re-enters the REPL via xce_repl_on_replay_complete() instead of exiting
the process; step/continue issued after replay is already complete
re-prompt instead of silently exiting.
- bug-1595: `step`/`continue` now echo each input event (not just
decisions) via repl_print_event_summary(), in real event/decision
order.
- bug-1593: `info resource <name>|all` now reports resource definition
metadata restricted to XCE_RESF_GLOBAL/XCE_RESF_SHARED resources, read
from core->sharedInst, instead of aggregating host-based resources
across every worker.
- bug-1596: repl_split_tokens() strips one matching pair of surrounding
quote characters from each token.
See requirements.md's Req 5 AC 22 "Amendment (2026-08-19)" for the full
rationale on each of the five task 12.5 gaps.
Full scheduler suite green (41/42; the one remaining failure,
test_recordstart_symlink_escape_replies_error, is the pre-existing
unrelated bug-1560).
Every early-exit branch in the worker_list scan either re-tested the same
node forever (bare `continue`) or returned out of the whole function on the
first DONE/EXITED worker, so job array elements past the head of the list
never got their on_job_start report. Capture `next` before touching each
node and always advance to it. Affects xcecd_virtual and real xcecd alike
(same DBUS_MANAGER-gated function).
Also records a roadmap follow-up (xcecd_virtual concurrent-job support,
MXJ > 1) discovered while investigating this bug.
requirements.md, design.md, research.md, and spec.json were never committed
during implementation (only tasks.md was, per the per-task checkpoint
convention). Committing them now that all 15 tasks are implemented,
reviewed, and validated GO by /kiro-validate-impl.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
verdict-logic-selftest.sh sources the real aggregate_verdict()/
build_verdict_record() (via run-resource-verify.sh's BASH_SOURCE guard, no
live deployment triggered) and exercises them with synthetic per-layer
records: all-match/PASS positive control, a single-resource single-layer
MISMATCH within a multi-resource batch (asserting both correct attribution
of the failure AND that the other resources independently still report
PASS), UNREACHABLE distinct from MISMATCH, REJECTED, and a zero-records
degenerate case. 53 assertions total.
Proven non-tautological: breaking aggregate_verdict()'s comparison logic
(two different ways, independently reproduced by review) makes the selftest
genuinely fail; restoring makes it pass again.
This completes the resource-value-verification spec - all 15 tasks (1.1
through 8.2) implemented, reviewed, and committed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Validation-only, no code changes. Confirmed the complete spec end-to-end
against a freshly built controller+scheduler+apiserver: default 9-resource
set + one deliberately-unregistered name -> VERDICT resources=10
rejected=1 result=FAIL (exit 1), attributable specifically to the rejected
name, not a false PASS; positive control (defaults only) -> VERDICT
resources=9 rejected=0 result=PASS (exit 0). All 27 individual layer-checks
(9 resources x 3 layers) MATCH; raw scheduler.log/apiserver.log trace lines
independently verified against tasks 4.1/4.2's documented format; rejection
confirmed non-silent across 4 independent log sources; cleanup and log
preservation confirmed.
Surfaced (but correctly did not fix, out of boundary) bug-1583: the
apiserver stores resource-location updates with zero name validation, even
for names the controller/scheduler both reject. Confirmed this does not
affect this feature's own correctness - accept/reject is determined
entirely by cmd_resupdate()'s controller-side follow-up query, before any
layer is checked.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
aggregate_verdict()/build_verdict_record() combine the three per-layer
check results into a single per-resource comparison, self-contained and
invokable with synthetic records (task 8.2). Uses ASCII Unit Separator
(\x1f) as the record delimiter after discovering bash's IFS tab-splitting
silently collapses adjacent empty fields even with IFS set to only a tab
(bug-1581), which corrupted UNREACHABLE-record field alignment.
Emits one machine-parseable VERDICT line (resources/layers_checked/
mismatches/unreachable/rejected/result) plus the terminal PASS/FAIL oracle
line and matching exit code. Cleanup is the same trap-based mechanism task
5.1 registered for the abort path - logs always preserved regardless of
--keep-running.
Round-2 fix: main()'s live-data wiring (build_verdict_records_from_tsv())
derived a layer's status from array-key existence rather than the actual
recorded outcome - since tasks 6.3/6.4 always set the key (even on a
genuine timeout, just with an empty value), every real UNREACHABLE was
silently misclassified as MISMATCH. Fixed to check SCHEDULER_CHECK_RESULTS/
APISERVER_CHECK_RESULTS string content instead (bug-1582). Live-confirmed
by killing the scheduler mid-run: VERDICT now correctly reports
unreachable=9 mismatches=0, not the reverse.
SKILL.md rewritten to describe the now-complete, functional tool.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
poll_apiserver_log()/query_apiserver_resvals() mirror task 6.3's
scheduler-check pattern: bounded-wait polling (batched across all pending
resources per iteration) of apiserver.log for task 4.2's "resource-location
stored" trace line, parsing the trailing zap JSON block. Reports MATCH /
MISMATCH (both values) / UNREACHABLE (timeout, distinct from mismatch),
tracked in APISERVER_OBSERVED/APISERVER_CHECK_RESULTS +
apiserver-check-results.tsv.
This completes all three per-layer checks. Live-confirmed: the full default
resource set (including cpuf, post-bug-1579-fix) MATCHes simultaneously at
the controller, scheduler, and apiserver layers.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Previously worked around (task 6.2) by removing the one host-attrs resource
(cpuf) from the default set - a custom resource=value arg using any
host-attrs name would still false-mismatch at the controller layer, since
inject_resources() disconnected the worker before the separate controller-
check session ran, and the controller resets host-attrs resources on
disconnect.
Real fix: xce_test's `!<shell-command>` escape is handled in the single
top-level REPL loop before context-specific dispatch, so it works in any
context - including worker mode - without calling leave_ctx()/disconnecting.
inject_resources() now shells out via `!` to run the controller check
in-line, between resupdate and quit, while the worker's connection is still
open. query_controller_resvals()'s comparison logic is unchanged - this is
a pure sequencing fix. cpuf restored to DEFAULT_RESOURCE_ARGS as live proof.
Live-verified: cpuf now MATCHes at the controller layer (was always a false
MISMATCH before); adversarial shell-injection attempt via a malicious
resource name confirmed safe (printf %q quoting); SIGTERM delivered exactly
during the nested check confirmed zero orphaned/zombie processes across 4
independent runs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
poll_scheduler_log()/query_scheduler_resvals() tail the live scheduler
daemon's log (task 4.1's trace line) for each accepted injected resource,
with a bounded ~15s wait batched across all pending resources per poll
iteration (not N times the timeout). Reports MATCH / MISMATCH (both
values) / UNREACHABLE (timeout, distinct from mismatch), tracked in
SCHEDULER_OBSERVED/SCHEDULER_CHECK_RESULTS + scheduler-check-results.tsv.
Live-proven immune to bug-1579 (unlike the controller check): injecting a
host-attrs resource (cpuf) simultaneously produces a controller MISMATCH
and a scheduler MATCH, since this check reads task 4.1's trace log (written
once, at apply-time) rather than re-querying live state after the worker
disconnects.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
query_controller_resvals()/controller_values_match() query the controller
via resval for every accepted (not rejected) injected resource/worker pair
and report MATCH/MISMATCH with both values on a mismatch, tracked in
CONTROLLER_OBSERVED/CONTROLLER_CHECK_RESULTS + controller-check-results.tsv
for tasks 6.3/6.4/7.1.
Also fixes bug-1579 (round-2, in default set only): cpuf, a host-attrs
resource the controller resets to baseline on worker disconnect, always
false-mismatched since inject_resources() disconnects before this check
runs. Swapped for r1m (periodic/dynamic, disconnect-safe) in
DEFAULT_RESOURCE_ARGS. Custom host-attrs resource=value args still hit this
- documented as a known follow-up, not fixed here (would need a C-level
xce_test change or a coprocess restructuring).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
inject_resources() joins a synthetic worker through xce_test's worker
context and runs resupdate for the requested resource=value pairs
(default: Requirement 1's numeric gap set + mem/cpuf regression names,
deliberately excluding type/model due to bug-1574's live segfault risk).
Per-name accept/reject results captured into INJECTED_VALUES/
INJECTED_RESULTS plus injection-results.tsv for tasks 6.2-6.4.
Also suppresses the real xcecd_virtual worker daemon that deploy-ha.sh's
pre-existing worker branch would otherwise silently start whenever the
build dir contains one (XCE_WORKER_BIN=/bin/true), which would have
violated Requirement 2.1 - bug-1577.
Blocked pending a separate prerequisite fix (bug-1578, commit b406dd7):
live testing surfaced a pre-existing controller crash on worker disconnect,
unrelated to this feature, now fixed with regression coverage. Live-
reconfirmed post-fix: controller survives the full join+resupdate+quit flow
indefinitely.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
run-resource-verify.sh deploys controller+scheduler(-vvv)+apiserver(debug
logging) via xce-deploy-ha's deploy-ha.sh -> start-all.sh into a fresh
scratch dir, confirms all three are genuinely up, and cleans up via a trap
registered before any process starts - covering both normal completion and
a mid-deployment abort. Logs always preserved regardless of --keep-running.
Independently stress-tested with 8+ SIGTERM interrupts at varying
deployment points and a genuine crashing-apiserver scenario - zero leftover
processes in every case. Single-node by default (propagation testing, not
HA failover), overridable via $XCE_RESVERIFY_PEERS.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
One logger.Debug("resource-location stored", ...) call in
OnResourceLocationUpdated, immediately after store.ResourceLocations.Put,
logging resource/host/value as zap fields. Debug-level only, matches this
package's existing logging convention (reuses convert.go's field names).
Live-confirmed against a real deployment with logging.level: debug.
Also surfaced (but did not fix, out of boundary - pure Go change, no
possible causal link) a pre-existing xce_test segfault on string-typed
resupdate (type/model) - uninitialized stack memory, non-deterministic,
invisible to ASan. Logged as bug-1574; flagged in tasks.md as a risk for
task 7.1's default resource set and task 8.1's live e2e run.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
One TRACE (tlog_debug3) line in xce_core_push_events()'s RESVALS_DELTA case,
logging host/resource/value in a fixed format ("resval applied host=...
resource=... value=..."), immediately after the value is written to the
worker's live state. Debug-level only, no control-flow change.
Explicit svalue/fvalue branching (round-2 fix after review) - never reads a
Value_u_t union member without confirming .which matches first, since
Value_u_bvalue (boolean resources) is reachable at this path in production
and would otherwise be misread as a garbage float.
Live-confirmed against a real deployment at -vvv. Also surfaced (but did not
fix, out of boundary) a pre-existing double-free crash on a resource's
second periodic report - logged as bug-1572.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
deploy-ha.sh Step 3 gains an opt-in-by-presence apiserver step (gated on a
pre-built /tmp/apiserver binary, not built by this script) that generates
start-apiserver.sh with a debug-level logger.yaml, plus an always-generated
start-all.sh sequencing controller-health-check-then-worker/scheduler
(-vvv)/apiserver. Purely additive - callers without a built apiserver
(e.g. xce-test-ha) see zero behavior change.
Live-confirmed both scenarios end-to-end: no-apiserver correctly skips the
step; with-apiserver reaches full backend registration with debug logging
active and scheduler emitting DEBUG3 output.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
cmd_resupdate() now issues a follow-up xcecli_resval() query per submitted
(resource, worker) pair after the update RPC, since update_resval_list()'s
reply can't carry per-item status. Reports accepted/rejected per name, with
a bounded retry (5x100ms) against propagation races.
Live-confirmed: `resupdate -n mem -v 5 -n not-a-real-resource -v 1` reports
mem accepted and not-a-real-resource rejected in the same invocation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
set_resval() now delegates type resolution to resval_typeresolve.c; unknown
resource names default to numeric, overridable via a new -t <numeric|string>
flag. cmd_resupdate()'s getopt loop defers per-pair finalization so a
trailing -t still applies to its pair, while preserving the exact original
duplicate-value rejection behavior (round-2 fix after review caught a
last-write-wins regression in that refactor).
Live-confirmed: resupdate -n r15m -v 1.5 (previously silently dropped) now
succeeds and is visible via resval.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Documentation-only stub: usage, VERDICT line format, and PASS/FAIL/cleanup
contract per design.md, explicitly marked not-yet-functional. The actual
run-resource-verify.sh orchestration script is built incrementally by
tasks 3.1-7.1.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Move set_resval()'s resource-name lookup table into a new, dependency-light
resval_typeresolve.c/.h module so it's independently unit-testable (test.c
can't be white-box included into Unity). Adds RED tests for the not-yet-
implemented numeric-fallback/type-override behavior (task 2.1).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ha-config-bootstrap, scheduler-controller-msg-gap, and resval-id-decoupling
were the only 3 specs (of ~40) carrying a nonstandard status:"active"
field, each paired with a note about residual gap-audit findings that
never got refreshed once later work actually resolved them:
- ha-config-bootstrap: all 3 known deviations resolved/superseded
(redirect fields -> libxce-failover; WAL majority-commit scope -> the
deliberate wal_classify() WAIT/RETAIN design; commit-timeout durability
-> bug-661 fix via ha-commit-durability, live-reverified 2026-07-18).
One narrower, genuinely new gap found in passing (OP_upgradeConfig
replies lack redirect wiring, admin-config-tool task 6.3) is flagged
in the note for a future spec, not folded back into this one.
- scheduler-controller-msg-gap: both open items (sched_pend_update_killer
leak, test coverage) were fixed via bug-765 across 4 commits on
2026-06-19, verified directly in current scheduler.c and the test
suite -- the note just never caught up.
- resval-id-decoupling: not just an investigation -- the recommended
name-keyed collect_resval() fix shipped one day after the spec was
created (commit d47b3375), verified as the sole implementation left
in current code. Description no longer named the deleted function;
status:"active" flipped to "closed" and a note added to match its
sibling specs' convention.
All three verified via direct code/commit/buglog inspection (not just
trusting stale spec text) before closing. status:"active" -> "closed"
in all three; phase stays "implemented" (already correct).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Task 6.3's own "before deciding" gate (re-run the flaky primary-push
E2E 20-25x standalone with real stderr capture) was already satisfied
by bug-1485's 2026-08-01 follow-up: 40/40 clean runs, zero captured
crash evidence, after replacing the DEVNULL stderr redirect with real
per-node log capture. Decision reached there (mergeable as-is, bug-1485
downgraded to low-priority/environment-flake) was never reflected back
into tasks.md/spec.json. Marks 6.3 and its parent task-group 6 [x], and
reconciles spec.json's stale "tasks-generated" phase (21/21 tasks were
already complete) to "implementation-complete" with a validate_impl
block recording the known limitations (bug-1485 stays open/low-priority,
single-node ackedCount==0 exit-1 behavior, no CI live-E2E run yet).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- config-upgrade-snapshot, raft-adapter-conformance: correct stale
phase metadata (both showed "tasks-generated" though every task was
already complete) and record their validate-impl GO decisions with
known-limitations notes (bug-1471/bug-1472-remainder, bug-1474
angle-b) for future spec ownership.
- component-failover-adoption: catch up design/requirements/research
docs to the Phase 6 fix (bug-1548/bug-1549, commit d479b707) that
landed after this spec was already marked closed.
- Promote 3 roadmap backlog items to specs (2026-08-10 discovery):
component-rejoin-resync (apiserver/scheduler rejoin resync),
worker-rejoin-resync (worker job-fact resync, hard prereq for
raft-apply-decoupling's facts half), scheduler-sim-diagnostics
(open-ended scheduler REPL debug/troubleshoot capability).
- roadmap.md updated to reflect all of the above.
- kiro-steering/kiro-steering-custom SKILL.md: fix stale template path
(templates/steering/ -> .kiro/settings/templates/steering/) plus
minor doc corrections in the other kiro-spec-* skills.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
XCECLI_FF_CALL (backing xcecli_heartbeat) can silently rotate+reconnect
the transport entirely inside a single call and still return success, so
heartbeat.go's failure-to-success transition detector could never observe
a reconnect that completes within one heartbeat tick -- the common case,
since elections finish in single-digit seconds against a default 30s
interval. apiserver's application-level rejoin never fired on a real
idle-period failover, and job pushes silently stopped reaching it.
Adds a monotonic reconnect_gen counter to endpoint_t (netio.h), bumped by
xce_ctx_reconnect() (xcectx.c) on every successful reconnect regardless of
trigger path. heartbeat.go now compares this generation every tick instead
of tracking success/failure transitions. Re-validated live against task
5.2's exact scenario (apiserver alone, zero write traffic): rejoin fired
22s after a primary kill, 12s after election.
Reopens and closes out component-failover-adoption's own NO-GO -> GO
validation cycle from 2026-08-15.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Final regression sweep (task 5.4) confirms zero new regressions across
the full stack (lib/controller/scheduler/worker/xceadmin/apiserver/xcecli)
and proves the single-candidate path is identical to the multi-candidate
path across all three components (Req 1.5), closing out all 21 tasks
(1.1-5.4) of component-failover-adoption.
Two production bugs found during live E2E validation (5.1/5.2) remain
open and out of this spec's scope: bug-039 (apiserver idle-period rejoin
structurally unobservable on a real single-election failover) and
bug-040 (job pushes not reaching apiserver's local store, likely
downstream of bug-039).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Live E2E of apiserver's backend-health circuit breaker (task 4.5):
killed all 3 controller nodes (not just the primary, unlike 5.1/5.2 --
this scenario needs a call to genuinely exhaust its retry budget, not
successfully rotate to a survivor). A write call burns through the
20-attempt budget in ~3ms (ECONNREFUSED classifies as zero-backoff
XCE_FO_RETRY_NOW, not the ~70-100s backoff track reserved for a
reachable-but-electing controller), returning the real -E_XCE_NOT_READY
(-10013) and triggering MarkDegraded(10s). Immediately after, a second
write call fast-fails in ~1ms via the literal ErrBackendUnavailable
string -- proving the real C call was skipped entirely, not just fast
-- while a concurrent GET /ping (confirmed by source read to never
touch the backend) keeps succeeding normally throughout.
Cross-corroborated across two independently-generated log subsystems
(apiserver's Go zap log and libxce's own C-level retry log) matching
to the millisecond. Bonus: confirmed the 10s TTL's half-open auto-clear
correctly re-attempts the real call afterward.
No src/ changes -- live validation only.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Heartbeat-only idle-period E2E (apiserver alone, zero write traffic
during a 268s+ post-kill observation window) designed as a targeted,
cleaner confirmation experiment for task 5.1's bug-039 finding.
Result: bug-039 reproduces a second time, more cleanly than task 5.1
-- zero rejoin log lines throughout the observation window, and even
a post-window diagnostic write call (which itself succeeds against
the confirmed-new primary) triggers zero rejoin activity, ruling out
"task 5.1 was just unlucky timing." Controller-side logs corroborate
independently: zero "apiserver" references anywhere in the new
primary's log, meaning no join attempt of any kind ever reached it.
This task's own literal pass bar was not met -- approved on the same
basis as task 5.1: the validation itself succeeded rigorously and
honestly surfaced a real, now doubly-confirmed gap in the already-
shipped task 4.3/4.4 rejoin mechanism, which is the task's actual
purpose. A dedicated follow-up to fix heartbeat.go's transition
detection is warranted, separate from this spec's remaining scope.
No src/ changes -- live validation only.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Live 4-process E2E run (3-node controller cluster + xcecd_virtual
worker + xce_scheduler + xce_apiserver, all on the full 3-candidate
list) with the primary killed mid-session. Worker reconnect is
solidly proven with real log evidence and unchanged PIDs across all
three components throughout. Scheduler's rejoin path was never
naturally exercised (no job reached its picker/allocator) -- same
partial-coverage shape already accepted for task 3.3.
Surfaced a likely-real, structural gap in the already-approved task
4.3/4.4 apiserver rejoin mechanism (bug-039): XCECLI_FF_CALL can
silently self-heal the transport within a single heartbeat tick,
making the cross-tick failure->success transition invisible to
heartbeat.go's detector on any failover fast enough to complete
within one heartbeat interval -- which is most real single-election
failovers in this codebase. Also found bug-040 (job pushes never
reach apiserver's store), very likely a downstream symptom of
bug-039's rejoin gap via on_apiserver_join()'s per-process,
unreplicated apiserver registration list. Both logged with detailed,
independently-verified root-cause analysis for a dedicated follow-up
-- fixing them is outside this task's cross-component-validation
boundary.
No src/ changes -- live validation only.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Audit of apiserver's three task-4.6 coverage areas (degraded-to-healthy
transitions, concurrent-request safety, candidate-list parity):
degraded-to-healthy and candidate-list parity were already fully
satisfied by tasks 4.1/4.5's own tests. Found one genuine gap: reconnect.go's
needsRejoin/checkAndPerformRejoin (task 4.3) had zero concurrent-access
test coverage despite being reached from 5 real concurrent HTTP-wrapper
call sites plus the heartbeat goroutine -- health.go's existing
50-goroutine stress test only covered the separate degraded-until
timestamp. One new test closes it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Adds an atomic degraded-until state (health.go: IsDegraded/MarkDegraded/
MarkHealthy) that lets request handlers fail fast with a clear
unavailable error instead of blocking on xcecli's own ~100s retry
budget when the backend is known-degraded.
Consolidates what tasks 4.3/4.4 left as duplicated inline
"call xcecli_*; checkAndPerformRejoin()" pairs across five wrapper
functions into one shared callBackend() helper: checks IsDegraded()
before every call (fail fast, skip the C call and the rejoin check
entirely on the degraded path -- checking rejoin there would itself
block up to ~100s, defeating the fail-fast purpose), marks degraded on
-E_XCE_NOT_READY, marks healthy at checkAndPerformRejoin()'s existing
success point (the single call site both the write path and the
idle-period heartbeat path already funnel through).
The circuit breaker's own TTL-based state is deliberately never read
by the rejoin-retry mechanism -- checkAndPerformRejoin() has no
IsDegraded() dependency, so a fast-fail window never suppresses
retry-until-actually-joined; heartbeat.go's unconditional per-tick
call keeps that cadence alive regardless of degraded state, and a
10s TTL (well under the ~100s retry budget) lets write traffic resume
contributing once it clears.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
xcecli_heartbeat() is XCECLI_FF_CALL-wrapped -- a deliberately
non-retrying fire-and-forget path that does its own best-effort
one-shot reconnect but never calls xcecli_invoke_rejoin() (that only
fires from the write-call retry path). Left as-is, a controller
failover with no in-flight write request would silently repair the
transport connection without ever re-joining, permanently starving
push notifications.
runHeartbeat() gains two callback hooks fired at the right points --
onFailureToSuccess on a genuine transition, onSuccessTick on every
successful tick thereafter -- wired directly to task 4.3's existing
setNeedsRejoin()/checkAndPerformRejoin() from StartHeartbeat(). No
second rejoin mechanism: checkAndPerformRejoin() is a cheap no-op when
nothing is pending, so calling it unconditionally on every successful
tick is safe and matches the requirement that retry isn't gated on the
transition edge alone.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Registers a rejoin flag-setter (go_backend_rejoin_handler) as the
client library's reconnect callback: zero I/O, sets an atomic flag,
returns immediately -- avoiding self-deadlock on the library's
per-ctx failover lock the same way worker's and scheduler's rejoin
callbacks already do. The C shim, its extern declaration, and the
xcecli_set_reconnect_handler() registration call all live together in
clib.go's xce_backend_init(), placed after xcecli_start()'s success
check (xcecli_start() wipes any earlier-registered handler as part of
its own init -- the same ordering bug found and fixed in worker's/
scheduler's task 2.3/3.1 work).
A shared checkAndPerformRejoin() helper wraps every gCtx.ctx-based
backend call (SubmitJob, ControlJob, ControlQueue, ControlHost,
FetchFile): after the call returns, if a rejoin is pending, issue
xcecli_join() as a separate step and clear the flag only on success --
on failure it stays set so the next backend call's return retries it,
no separate retry loop.
needsRejoin is a proper sync/atomic flag rather than a plain bool,
since apiserver (unlike the single-threaded C daemons) serves
concurrent HTTP requests that can all reach the shared helper at once.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Replaces apiserver's two connect mechanisms -- a per-peer Go loop that
hardcoded nmasters=1 per attempt, and a separate direct-construction
path for the legacy single-host config -- with one pipeline:
buildClusterConfig() parses (via task 4.1's C parser) or synthesizes a
single-candidate config, validates it, then ConfigConverter
(xce_cluster_config_to_client_config) produces the client config for
one xce_backend_init() call. A 3-candidate config now actually
exercises the C library's own failover/standby-redirect handling on
initial connect, matching worker's and scheduler's task 2.1/3.1 work,
instead of a Go-level per-candidate retry loop that never built a real
multi-master config.
Along the way: found and fixed a genuine use-after-free. The deleted
tryConnect() freed its C.CString-allocated host address immediately
after its single connect call returned, but xcecli_ctx_init() stores a
raw pointer to the config forever (client.c:38) and every later
failover rotation re-dereferences it (client_failover.c) -- so any
reconnect after the first would read freed memory. Fixed by giving the
cluster/client config structs package-level, process-lifetime storage
(gClusterCfg/gClientCfg), mirroring worker's and scheduler's file-scope
static pattern, with host/from/key written into fixed-size C arrays
instead of freed heap pointers.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Replaces apiserver's hand-written Go parsePeer() with a CGO wrapper
around the shared C parser+validator pair (xce_parse_peers then
xce_cluster_config_validate, both src/lib/cluster.c) that worker and
scheduler already use from their own main.c's. Gives apiserver the same
[name=]host:port[,...] syntax (including the name= prefix the old Go
parser could not express) and identical malformed-candidate rejection
(empty host, port <= 0, non-numeric port) -- round-1 review caught the
first pass skipping the validate() half, silently accepting inputs
worker/scheduler both reject.
A malformed entry now rejects the whole candidate list rather than
being skipped with a warning (matches xce_parse_peers()'s own
all-or-nothing construction and worker/scheduler's precedent) --
surfaced loudly via RegisterToBackend's error, not silently.
Parsing only -- RegisterToBackend()'s connect loop and tryConnect()
are unchanged; collapsing them into a single connect call is task
4.2's job.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Proves task 3.2's retry-until-success-or-shutdown wrapper's guarantees
hold across multiple queued decisions drained through the real
xce_sched_driver_on_decision(), not just a single isolated call:
decisions queued behind a stuck one publish afterward in original
order (Req 7 AC3), and shutdown mid-retry costs at most one retry
cycle regardless of queue depth -- decisions behind the stuck one see
core.stop already true and are never attempted (Req 7 AC2). Also adds
a source-scan confirming this stays scoped to the scheduler's own
publish direction (Req 7 AC5) and a fresh-instance test confirming a
restart still loses in-flight queued decisions, unchanged by this work
(Req 7 AC6).
Pure test-coverage task -- zero production changes, matching the
review's independently-reproduced RED->GREEN evidence that the
existing task-3.2 implementation already satisfies these guarantees.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Adds a worst-case-occupancy test proving the scheduler's picker/allocator
pipeline buffers briefly past a stalled publish (decisions cbuf cap 5,
jbuf cap 1) then genuinely blocks until connectivity recovers, with zero
job loss (Req 3 AC2). Adds a source-scan regression guard confirming the
scheduler makes no periodic heartbeat call of its own (Req 3 AC3).
Also fixes a real gap found along the way: sched_init()'s
all-candidates-unreachable and join-failure log lines named only a
candidate count, never which candidates were attempted (same class of
bug worker's task 2.3 fixed). Added sched_format_candidate_list(),
covered by a real dup2()-based stdout capture around a live sched_init()
failure path (round-1 review caught this diagnostic shipping with zero
test coverage; fixed in round 2).
Multi-candidate join and kill-primary/reconnect were validated manually
against a live 3-node cluster rather than committed as an automated
test, following worker's task 2.3 precedent -- no automated live-deploy
E2E harness exists for any C daemon in this repo.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Wraps sched_publish_dispatch()/sched_publish_pend_reason() in a shared
retry loop that keeps retrying a -E_XCE_NOT_READY-exhausted publish
until it succeeds or the scheduler is shutting down (core.stop checked
before each attempt, not only after), composed with the pending-rejoin
follow-up established in task 3.1. Extended-retry success logs
distinctly from an ordinary first-attempt success (Req 7 AC4).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Scheduler's unified connect pipeline is implemented, tested (RED→GREEN
verified), and live-validated against a 3-node HA controller cluster
(standby-first candidate list joins correctly via xcecli failover). See
a9f283d.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FYSEeLmzjtD4RLDuUDhLoK
Confirm connector thread is isolated from worker_process_update() (main
thread) and resource_sync_thread() (own thread) via code read plus a real
pthread concurrency test. Add unit and live 3-node E2E coverage for
candidate/reconnect behavior, all-candidates-unreachable diagnostics, and
liveness cadence under concurrent reconnect.
The E2E run surfaced two production bugs the 2.1/2.2 unit tests missed:
xcecli_start() resets the reconnect callback to NULL after it runs, so
registering it before xcecli_start() silently no-ops the whole rejoin
mechanism; and a successful worker join returns its assigned slot index
as reply.status (any rc >= 0), not always exactly 0, so checking_connection()'s
rc == 0 success check never fired on a real non-zero-slot join. Both fixed,
plus per-candidate names added to the all-unreachable failure log.
Root-check bypass needed for the E2E run is gated at compile time via the
existing VIRTUAL_CLUSTER flag, not a runtime env var, so the real xcecd
binary's root requirement has zero bypass surface.
Thread cluster_cfg by value through xcecd_config_t -> xcecd_context_t ->
cc_ctx so it's populated before controller_connector_start() fires the
first join, instead of after xcecd_start() returns. checking_connection()
now retries a failed post-reconnect xcecli_join() on the next liveness
probe instead of leaving needs_rejoin cleared on mere attempt.
Replace worker's separate default/single-host and multi-candidate connect
branches with one path: synthesize-or-accept a candidate list, validate,
convert via xce_cluster_config_to_client_config(), connect once. Register
a flag-only rejoin callback (no I/O, no client-lib reentry) and remove the
liveness probe's bounded give-up so the library owns retry/recovery.
Task 1.3's full checklist (constant-equality assertion, candidate
order, one-candidate-no-branch, truncation-with-warning, NULL/empty
rejection) was already satisfied by tasks 1.1 and 1.2's test
coverage. Verified fresh against the current test suite; no
implementer/reviewer cycle needed since there is no new diff.
xce_cluster_config_to_client_config() converts a validated candidate
list into a fully-populated xce_client_config_t, copying candidates
in order with no special-casing for a single candidate. The
truncate-with-warning safety net stays in place as a defensive guard
now that N_MAX_MASTERS and MAX_HA_CANDIDATES match.
Also removes a WARN check in xce_cluster_config_validate() that
became unreachable dead code once task 1.1 reconciled the two
candidate-count constants (both checks tested the same field against
the same constant with no mutation between them).
Task 1.2 of component-failover-adoption (Requirement 1 AC1-AC7).
N_MAX_MASTERS (client.h) now derives from config.h's MAX_HA_CANDIDATES
instead of an independently-maintained literal (3 vs 4 previously).
Fixes worker's and scheduler's hardcoded `> 3` startup checks to
reference the constant symbolically, and updates the client-retry
test fixture so it doesn't exercise an uninitialized 4th slot.
Task 1.1 of component-failover-adoption (Requirement 1 AC7).
These edits (three-thread picker correction in brief.md, record_dir
path-validation and kvmgr-audit requirements in requirements.md,
OPCode_schedAdmin folded into the peer-authentication-hardening
backlog item in roadmap.md) were made during earlier requirements
review and design work but left uncommitted. Reviewed for accuracy
against the completed implementation — all still correct, persisting
as-is.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found by /kiro-validate-impl: tasks 2.1-2.6 (builtin_fcfs picker) were
implemented, reviewed, and committed on 2026-08-03 but never got a
_Verified: line, unlike every other task group. Backfills each from
git history (commits f7921b96, b7214ebe, fcb32a8b, 7ee9a16f, 9d91d470,
604a5649) and direct source inspection.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
