Commit graph

7 commits

Author SHA1 Message Date
80d5a06212
feat(worker-rejoin-resync): task 5 - exclude reconciling host from dispatch
Extend the IS_WORKER_UNAVAIL-style candidate-host filter to also skip
a host with XCE_WORKER_RECONCILING set (Req 5.1), naturally
re-including it once the bit clears with no separate mechanism (Req
5.2). Leaves the scheduler's decision-publish code and its
symbol-guard regression test untouched.

Fixes bug-1751: design.md's cited filter site (scheduler.c's
sched_reset_worker_(), via sched_begin_session()) is dead code on both
live drivers - driver.c/sim_loop.c actually call
xce_core_begin_session() -> core_session.c's core_reset_workers(), an
independently-maintained mirror not listed in design.md's File
Structure Plan. Applied the identical disjunct to both sites so the
task's own observable-completion criterion holds in production, not
just on the literally-cited dead path.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
2026-09-01 18:38:07 +00:00
eec2596c5d
feat(worker-rejoin-resync): apply reported outcomes idempotently (task 3.2)
Add resync_apply_report()/resync_apply_running_entry()/
resync_apply_finished_entry()/resync_apply_unrecognized_entry() to
resync.c: routes each entry in a worker's job-state report to the
correct existing idempotent path (still-running -> clear from outstanding
set, no fact; finished-success/failure -> record_job_finish_fact_at/
on_job_fail; unrecognized -> record_job_reconcile_pending_fact;
unqueried extra -> record_job_reconcile_assign_fact or no-op), discards
reports for superseded rounds (Req 6.1/6.3), and resolves the round once
its outstanding set empties.

Also fixes bug-1747 (Req 4.2): adds record_job_finish_fact_at(),
additive alongside the unmodified record_job_finish_fact, so a
finished-success reconciliation entry records the WORKER-reported
finish time instead of reconciliation-processing time -- proven via a
real WAL-write + persist_recover() restart-replay test. The FAIL-side
half of the same gap (WAL_JOB_FAIL carries no fail-time wire slot at
all, for any caller) is confirmed pre-existing and systemic, correctly
left out of this task's scope.

Three review rounds surfaced and fixed two real, distinct bugs in the
unqueried-extra-job routing guard (see tasks.md's Implementation Notes
for the full account and the general lesson about re-deriving a full
truth table after a partial boolean-guard fix): round 1 caught the
wrong axis (terminal-vs-non-terminal instead of assignment-status,
Req 4.8); round 2 caught that the first fix's short-circuit clobbered
an already-terminal job's DONE/EXIT back to RUN (Req 4.4), including
confirming the implementer's own test had certified the bug as correct;
round 3 independently re-derived the final three-way guard from scratch
and found no further defect. bug-1748 logged for the guard fix.

Full controller suite: 1239 tests, 0 failures.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
2026-09-01 15:24:29 +00:00
ecfb4c942f
feat(worker-rejoin-resync): reconcile unqueried job assignment (task 2.2)
Add record_job_reconcile_assign_fact(cluster_t*, job_t*, worker_t*,
xce_job_state_entry_t*) to job.c: eagerly sets job->job.state to match
the worker's reported outcome (mirroring dispatch_a_job_'s inline
mutation, minus allocation fields -- there are none to set, job->job.allocs
stays NULL per Requirement 4.8's documented scope). Calls the existing
persist_submit_job_dispatch_fact(..., alloc=NULL) directly, then leaves
the post-unlock persist_notify_job_updated_already_persisted() call to
the caller, same convention as task 2.1.

Fixes bug-1746 along the way: capnp_encode_joballoc() (persist.c, part
of bug-1678's original fix, predates this feature) unconditionally
dereferenced a NULL alloc -- never triggered before since
dispatch_a_job_ (the only prior caller) always passes non-NULL. Fixed
with a minimal guard producing a true zero-length buffer, matching
apply_committed_entry's alloc_len > 0 decode gate exactly. This
directly contradicted design.md's "tested, backward-compatible on both
seams" claim (true only for decode, never actually exercised for
encode) -- corrected in 4 places.

2 tests: applies-running-no-allocs-and-notifies, and a real restart-
replay round trip (write to real disk, fresh persist_recover(), confirm
jobid/host/state/NULL-allocs survive) -- proving this task's own
"durably survives a restart replay" completion criterion literally, not
just the in-memory mutation.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
2026-09-01 11:40:47 +00:00
b7ed6de2a1
docs(worker-rejoin-resync): correct design.md's NULL-alloc encode claim
Found live during task 2.2 implementation (bug-1746): design.md claimed
persist_submit_job_dispatch_fact's NULL-alloc case was "explicit,
tested, backward-compatible" on both apply seams -- true for the DECODE
side (apply_committed_entry/apply_wal_entry, bug-1678's actual fix
target), but the ENCODE side (capnp_encode_joballoc) was never
exercised with NULL before this feature and had a latent unconditional
NULL dereference. Corrected all references to distinguish encode vs.
decode safety and note the bug-1746 fix task 2.2 required.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
2026-09-01 10:47:46 +00:00
9fbbd12b36
docs(worker-rejoin-resync): add task 1.5, xceadmin config-build wiring
Discovered live during task 1.4 development: xceadmin's controller.capnp
copy is freshly synced from src/schema/ on every /xce-build xceadmin run
(meson-build.sh), so task 1.1's two new ClusterConfig fields already
broke the checked-in golden-fixture tests (test_golden_fixture_matches_
checked_in_expected_bin[minimal/full]) by changing the binary's wire
layout. Deeper gap: neither field has a YAML->binary configuration path
at all (unlike their own cited precedent, heartbeatInterval/
inactiveHeartbeatCount, which is wired into both xce.schema.json and
mapping.py) -- the original design review never scoped xceadmin as a
boundary. Added design.md Boundary Commitments addendum + tasks.md
task 1.5 to close both gaps together.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
2026-09-01 09:26:15 +00:00
6c908d5017
docs(worker-rejoin-resync): fix design review findings; add tasks.md
Design validation (2026-09-01) found and fixed 4 issues: a stale bug-1678
citation blocking reuse of persist_submit_job_dispatch_fact, missing
XCE_WORKER_RECONCILING accounting against scheduler-dispatch-commit's
dispatch-fail retry/terminal-fail classification, bug-1631 reactivation
(job.allocs never released on RUN->PEND reconciliation), and an eager-vs-
deferred mutation inconsistency between the new job.c functions and the
bug-1679 explicit-notify precedent. design.approved and tasks.approved
both set; tasks.md drafted (7 major/19 sub-tasks) and passed two rounds
of independent task-graph sanity review.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
2026-09-01 08:03:10 +00:00
f2ff45f18e
docs(worker-rejoin-resync): add requirements, design, and research docs
Requirements approved; design generated (not yet approved -- pending
/kiro-validate-design fixes).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
2026-09-01 07:08:41 +00:00