Add WatchJobs(ctx) to the Client interface and HTTPClient: opens the
apiserver's /xce/v1/events SSE stream and yields JobEvent values until the
context is cancelled or the stream ends. JobEvent carries the event type,
job id, status string, exit code, and a raw job snapshot.
Also fix ListJobs' filter params: the apiserver accepts filters via the
"filter" query param in field:op:value form (e.g. queue_name:eq:normal);
the individual queue_name/submit_user params it ignores with a 400, which
broke every filtered list. Multiple conditions are brace-wrapped {a},{b}.
Verified live on t1: WatchJobs captured job.updated PEND→RUN→DONE for a
submitted job.
Co-Authored-By: Claude <noreply@anthropic.com>
Add a Server-Sent Events endpoint /xce/v1/events streaming job lifecycle
events (job.created/updated/deleted/finished) to subscribers. The
DefaultHandler now publishes every job event to a process-wide EventBroker;
the SSE handler subscribes and streams JSON frames with a keep-alive comment
during idle.
- backend/broker.go: thread-safe fan-out Broker (drops for slow consumers so
the notification path never blocks), Event type with job_status string.
- backend/handler.go: OnJob* publish to DefaultBroker.
- controllers/events.go: SSE handler (gin, text/event-stream, keep-alive).
- router + whitelist + rbac: /xce/v1/events route, event:subscribe
permission granted to mcp_readonly/mcp_operator/cluster_admin.
- Tests: broker fan-out/slow-consumer/unsubscribe, handler publishes, SSE
end-to-end via httptest.
Verified live on t1: SSE stream delivered job.updated PEND→RUN events for a
submitted job.
Co-Authored-By: Claude <noreply@anthropic.com>
Full Agent chain verified on t1: mcp_operator API key → mcp-server
(scheduler_submit/get_job) → apiserver REST → controller → scheduler →
worker, job submit→run→DONE; cancel(kill)→EXIT. Note the apiserver
controller-restart rejoin gap (DataStore stays stale until apiserver
restart).
Co-Authored-By: Claude <noreply@anthropic.com>
bug-1510 (dispatch-fail re-PEND, 6928f74) + bug-1511 (sched_update_workers
in-place update on re-push, 1a2bd3f) together close bug-1494. Update
left_scheduler_work.md and buglog.
Co-Authored-By: Claude <noreply@anthropic.com>
When a worker is re-pushed (e.g. it rejoined with fresh resvals after the
controller/worker had been down), sched_update_workers built a new worker and
sched_add_worker's key_add returned EEXIST — the fresh worker was dropped and
the scheduler kept the stale state (slots=0 from the dead/absent period).
PEND jobs never dispatched even after the worker returned with full capacity
(bug-1494, controller-restart self-heal).
sched_update_workers now updates an already-known worker in place: swap the
fresh resvals/resInUse/status into the existing worker (the stale arrays are
freed when the discarded object is torn down) and adopt its host_eval_ctx.
Also NULL-guard make_sched_worker (it can return NULL).
Verified on the t1 cluster: scheduler started while the worker was dead
(slots=0), worker rejoined (slots=8), previously-PEND jobs dispatched and
completed, no ASAN.
Co-Authored-By: Claude <noreply@anthropic.com>
Document that the scheduler dispatch-fail re-PEND fix (commit 6928f74)
resolves one root cause of the "all jobs PEND" symptom; the
controller-restart self-heal-reconnect aspect of bug-1494 remains open.
Log bug-1510 in buglog.
Co-Authored-By: Claude <noreply@anthropic.com>
The controller rejects a dispatch when the chosen worker is not in a
dispatchable state (UNAVAIL after a worker death, or still joining) and
replies with a jobDispatchFail notify. The scheduler had no handler for that
event, so a job it had just committed (state=RUN) was never returned to PEND
— it was never re-elected and stayed PEND cluster-wide even after the worker
recovered.
Handle NotifyData_data_jobDispatchFail: log the failure and call the new
sched_requeue_pend_job(), which frees the dispatch bookkeeping, resets the
job to PEND, and re-enqueues it in the fcfs pendJobs queue so the next
election session re-dispatches it. White-box test TUs that #include
handler.c get a sched_requeue_pend_job stub.
Co-Authored-By: Claude <noreply@anthropic.com>
Merge the scheduler-join NULL guard from the remote js branch: if the
controller's cluster config is not loaded yet (cold boot / config not
pushed yet), xcesched_push_configs would dereference a NULL cluster_config
and crash the controller (ASan SEGV on scheduler join). Skip the push; the
scheduler receives config on a later join once config is available.
Co-Authored-By: Claude <noreply@anthropic.com>
When a user asks to view cluster status (e.g. "看看机器情况"), the LLM saw
non-OK values in the MCP response (a node UNAVAIL, a job PEND/EXIT) and
treated them as incidents — proposing unsolicited diagnostic/repair actions
and referencing "known bugs". The user only wanted a factual status report.
Mitigation at the MCP layer:
- Add server instructions (returned to the client in the initialize
handshake) stating that the status tools return operational snapshots,
that UNAVAIL/PEND/EXIT/UNKWN are normal state, and that the model must
not speculate or propose actions unless the user explicitly asks.
- Append report-only guidance to the scheduler_list_nodes / list_jobs /
cluster_metrics descriptions.
Verified: initialize response carries the 857-char instructions; tools/list
shows the updated descriptions.
Co-Authored-By: Claude <noreply@anthropic.com>
GetJobByIDs (GET /xce/v1/jobs/:jobids) built its job_id filter via
ConvertToNewFilterArray into QueryOptions.CompConditions, but ExecuteQuery
only ever applied opts.Conditions and silently dropped CompConditions. Every
per-id job query therefore returned the ENTIRE job table, and the MCP
scheduler_get_job tool returned the first job in the store regardless of the
requested id (observed: get_job(73930445842022400) -> "post-clean").
Fix: add FilterCompConditions (per-field AND, OR across ConditionArrays,
main-condition AND OR(sub-conditions) within an array) and run it in
ExecuteQuery after the flat filter; extract the bitmask-aware per-row match
into rowMatchesCondition shared by FilterRows and FilterCompConditions.
Verified via the MCP server: scheduler_get_job now returns the requested job
(mcp-e2e, EXIT), and the full submit->hold->release->cancel->get cycle still
works.
Co-Authored-By: Claude <noreply@anthropic.com>
Two stacked defects in the apiserver job-control link, both of which kept
hold/release/cancel from ever reaching the controller:
1. controllers/jobs.go: validateJobControl and foundJob asserted
jobinfo.Data["array_idx"].(int64), but StoreToOrderMaps produces
array_idx as int (models.JobInfo.ArrayIdx is declared int) -> panic
-> Gin Recovery -> HTTP 500 before backend.ControlJob ran. Add
normalizeJobInt64 / jobArrayIdx helpers and use them in both
functions (job_status already had this fix via normalizeJobStatus;
array_idx was missed).
2. backend/clib.go: backend.ControlJob set xce_jobctrl_t.action to the
models.JOB_* constant (JOB_STOP=1 / JOB_RESUME=2 / JOB_KILL=3 /
JOB_CHKPNT=8), but that field is the capnp JobControl union
discriminant (JobControl_u_siginfo=0 / JobControl_u_ckptoption=1).
The controller's job_ctrl() selected the wrong union branch (or
none, since encode_JobControl drops the union pointer for
non-0/1 discriminants) and rejected every control with -EINVAL.
Add jobCtrlDiscriminant() and use it.
Also: resolveExecUser was defined in the cgo-only convert.go but is
exercised by a non-cgo test, so `make test` (CGO_ENABLED=0) failed to
build the backend package. Move it to convert_pure.go so it compiles in
both modes.
Tests: 3 controller regression tests (validateJobControl allowed /
not-allowed branches, foundJob array-child branch) + 1 backend
jobCtrlDiscriminant test. make test fully green. Cluster E2E verified:
hold -> PSUSP, release -> PEND, cancel -> EXIT on a PEND job, no 500,
no -EINVAL, all daemons healthy.
Co-Authored-By: Claude <noreply@anthropic.com>
Phase 3 (B1 dual-state + B2): a job never lingers as RUN when its worker
is lost.
Dual-state B1 (user-approved): RDHUP (worker TCP dropped = confirmed dead)
marks its RUN jobs EXIT immediately (mark_worker_jobs_exit). Heartbeat
timeout (unknown) keeps them UNKWN+TRANSIENT awaiting reconnect
reconciliation; the TRANSIENT TTL GC is the backstop. handle_worker_leave
now calls mark_worker_jobs_exit (was mark_worker_jobs_unknown in Phase 2).
B2 (worker side): resend_job_status now checks the recovered process is
alive (kill(pid,0)) before re-reporting RUN — a dead process is reported
finished so the controller converges to EXIT instead of a stale RUN. And
resend_job_status runs on EVERY rejoin (checking_connection), not just
startup, so a reconnecting worker's jobs are reconciled (clears UNKWN via
the existing notify_jobstart claim path).
Design: docs/Job-Finish-Robustness-设计文档.md §4.4 (dual-state decision).
Unit test: worker RDHUP marks jobs EXIT (not UNKWN).
Phase 2 of the job-finish robustness design (Part B, B1+B3).
B1 (worker liveness): a worker that stops reporting (heartbeat timeout,
already detected by check_worker_timeout_) or disconnects now has its RUN
jobs marked JOB_STAT_UNKWN|JOB_STAT_TRANSIENT (transient_since set, durable
WAL_JOB_UPDATE) instead of lingering as RUN forever. Marking happens
AFTER the worker lock is released (B5: avoids worker->jobmgr inversion
with the scheduler's jobmgr->worker order).
B3 (restart): persist_recover marks all recovered RUN jobs UNKWN+TRANSIENT
in-memory after replay — a controller restart leaves worker connectivity
unknown. In-memory only (no WAL append) because persist_recover may run
on a future standby that must never append.
Unit tests (red->green): worker-loss marks its RUN jobs unknown, other-host
jobs untouched, restart marks all RUN jobs. test_replay_apply_seam updated:
replay still drives facts to RUN, then B3 adds UNKWN+TRANSIENT.
A worker-reported finish may arrive before the job exists: job
construction is deferred to apply_committed_entry (the submit's WAL
entry), leaving a commit->apply window. on_job_finish dropped such
finishes ('can not find job'), so a fast job that finished in that
window stayed RUN forever (e.g. noexec-test/newcfg-test/verify-fix).
Add an in-memory unknown-finish registry: on_job_finish registers the
finish when the job isn't materialized, and apply_committed_entry's
WAL_JOB_SUBMIT case drains it right after restore_job — appending the
durable WAL_JOB_FINISH and mutating the job to DONE/EXIT immediately so
the scheduler never dispatches a job that already finished.
Design: docs/Job-Finish-Robustness-设计文档.md (Part A of A+B; Part B
worker-liveness + reconnect reconciliation deferred to later phases).
Unit tests (red->green): registry register/take, apply-once-job-
materializes (DONE), exit-code maps to EXIT, drain no-op without
registration.
Test fixtures create WAL/snapshot scratch dirs via mkdtemp
(/tmp/xce_e2e_ckpt_*, /tmp/xce_ucaf61_*, /tmp/xce_ts_*) but their
teardown only frees the persist_t — the dir is never removed. persist_init
preallocates a 64MB WAL segment per init, so a full controller_unit_test
run leaves ~28G in /tmp in under a minute, filling the root filesystem.
Remove only xce_* DIRECTORIES at suite start (stale leftovers) and via
atexit (this run). Files are untouched: the user's /tmp/xce-mcp binary
and the suite's own /tmp/xce_test.log survive the -type d filter.
The apiserver's 30s heartbeat sends a fire-and-forget nop (xce_nop) that
never waits for the reply. The controller replies to every nop, and
xce_handle_reply's default case returned -ENOENT for OPCode_nop, so
vtask_entry called xce_wakeup and left the nop's status sitting in the
single-slot wbuf. The next blocking request (xce_fetchfile / xce_jobsub
/ xce_jobctrl) then consumed that stale status as its own reply:
the fetchfile waiter woke with status=0 but ctx->reply.u.fetchfilereply
still NULL -> 'nil reply' -> job peek 501.
Return 0 for OPCode_nop replies so they never touch wbuf. Control-op
replies (jobctrl/queuectrl/hostctrl) legitimately hit the default case
and still wake.
Regression test: test_handle_reply_nop_does_not_wake (red->green).
bug-1506 (controller): sched_worker_change pointed w->host.resvals_to_sched
at resvalist.data, then xlist_fini freed it — a concurrent reader (the
WORKER_JOIN hook encoding the worker's resvals) hit an ASAN
heap-use-after-free that crashed the controller and failed every job. Take
ownership of the resval data via MOVE_PTR (mirrors collect_resvals_workers);
the prior array is intentionally leaked (freeing it races the same readers).
bug-1507 (worker): the apiserver's bug-1500 execuser fix sent the submit
user ("admin") as execuser, but that user does not exist in the worker
host's /etc/passwd. systemd failed the worker unit with EXIT_USER (217),
so every job died immediately after submit. job_start_local_task now
checks the execuser with getpwnam() and falls back to "nobody" when the
user does not exist.
Verified with a full job lifecycle (submit -> RUN as nobody -> DONE) on
the clean build: all 4 daemons healthy, zero ASAN.
Co-Authored-By: Claude <noreply@anthropic.com>
`xcecli jobs list -a` returned "no job found" even when jobs existed.
FilterRows routes every condition on a bitmask field (job_status /
host_status) to MatchesBitmaskCondition, which only implemented OpEQ/OpNE
bitmask semantics — the `-a` filter compiles to `job_status notnull`
(OpNOTNULL), which fell through to `return false` and filtered out every
job. (The default list's `notin(EXIT,DONE)` is compiled to OpEQ(mask), so
it was unaffected.)
Add OpISNULL/OpNOTNULL branches to MatchesBitmaskCondition that test the
raw field value. New query_test.go: 3 regression tests (notnull matches
all, null matches none, bitmask notin path unaffected), RED->GREEN.
Verified live: `xcecli jobs list -a` now lists all jobs including DONE.
Co-Authored-By: Claude <noreply@anthropic.com>
Job PEND root cause when the scheduler registered before the worker:
xcesched_push_workers only fires on scheduler join (sched.c on_scheduler_join),
so a later worker join's slots (set_host_slots from hconfig->mxj=8) is never
re-pushed to the scheduler -> allocation unsatisfied (slots total 0) -> PEND.
Operational fix (no code change): start controller -> worker -> scheduler so
the scheduler's join-time schedworkers push carries the worker's slots.
Co-Authored-By: Claude <noreply@anthropic.com>
The RUSAGE_UPDATE job hook fires with (cluster_t*, xce_job_rusages_t*) as
its varargs — the apiserver and scheduler subscribers parse exactly that
contract — but persist_job_hook_fun's default case treated the second arg
as a job_t* and read j->job.state at offset 56. A xce_job_rusages_t is 56
bytes and shares only jobid@0 with xce_job_t (which is why the jobid read
coincidentally worked), so the state read landed one byte past the struct:
ASAN heap-buffer-overflow (READ of size 4) in thread T7 whenever a job runs
and the worker pushes rusage updates. This was the recurring controller
crash (term 3/4/5) blocking job-PENDING verification.
Add a dedicated HOOK_DETAIL_JOB_RUSAGE_UPDATE case that reads jobid from
the xce_job_rusages_t* and bakes the job's current state from a jobmgr
key_find on the vararg cluster_t* (race-free: the sole caller
update_job_rusages() holds that job's write lock while firing the
AFTER_EVENT hook; re-locking the job inside the hook would deadlock).
Regression test: test_job_hook_rusage_writes_wal_entry now passes a real
56-byte xce_job_rusages_t instead of a full job_t (which masked the
overflow) — ASAN stack-buffer-overflow under the old code, PASS under the
fix. Full controller unit suite: 458 tests, 0 failures.
Co-Authored-By: Claude <noreply@anthropic.com>
tryConnect built the xce_client_config_t with C.CString(host/from/key), then a
defer freed all three. But xcecli_ctx_init stores the config POINTER
(ctx->conf = config), not a copy — client_failover.c dereferences
config->masters/from/key for the client's ENTIRE lifetime (e.g.
xce_failover_next_target strncpy's a master host on every write-fail/redirect).
Freeing the strings left config->masters[0] dangling → ASAN heap-use-after-free
(strncpy in the background C thread) after a few submits.
Fix: keep the three C strings for the process lifetime (tryConnect runs once at
startup, gCtx is a process-global); free them only on the init-failure path.
Verified: 6 consecutive job submits, apiserver stays up, 0 ASAN errors
(previously crashed after ~2-3 submits).
Co-Authored-By: Claude <noreply@anthropic.com>
modelJobToSubmit built the C job_startinfo_t without setting execuser, so the
controller forwarded a NULL execuser in the startjob request and the worker
ran every job as "nobody". Now the submit carries the resolved exec user: an
explicit exec_user wins, otherwise the job runs as the submitting user.
- add resolveExecUser() (plain-Go, unit-testable) + TestResolveExecUser
- set si.execuser in modelJobToSubmit, free it in freeJobSubmit
Note: deploying this exposed an ASAN heap-use-after-free in the apiserver's
submit/notification path (strncpy reading freed memory); the fix is committed
but needs the memory-lifecycle investigation before it can ship safely.
Co-Authored-By: Claude <noreply@anthropic.com>
bug-1498 (re-dispatch): sched_job_is_ready() was a stub returning 1, so any
job lingering in the fcfs pendJobs queue was always elected. A dispatched job
was never marked non-PEND (the controller defers its schedStartJob notify to
the not-yet-implemented apply pump, controller/job.c), so fcfs_update_job
re-enqueued it on every job update and the election thread re-dispatched the
running job — the worker got conflicting signals and the job was SIGTERM'd.
- sched_job_is_ready: elect only PEND, non-suspended jobs
- sched_commit_decision (dispatch): mark the job JOB_STAT_RUN on dispatch so
it leaves the PEND set and is no longer re-electable
- fcfs_update_job: dedup via ordq_peek before enqueue
bug-1497 (double-free): sched_update's resval paths transferred ownership of
rv->val to sharedInst/w->resvals but left the request pointer set, so
sched_pend_update_killer double-freed it -> use-after-free in
build_host_eval_ctx. Clear rv->val = NULL on transfer.
Tests: new scheduler/testsuite/test_redispatch.c (8 cases): sched_job_is_ready
PEND gating + dispatch-commit marks the job RUN. Full scheduler meson suite
green. Live-verified: a `sleep 60` job is elected exactly once, runs the full
duration and exits 0 (`completed successfully`); before the fix it was
re-dispatched at ~11s and killed (SIGTERM 143).
Co-Authored-By: Claude <noreply@anthropic.com>
put_resvals (resource.c) used `idxs[i] <= 0` to skip "invalid" resource
indexes, but index 0 IS valid — it's the first resource in the table
(e.g. "slots"). For idx 0 the val was never allocated (xvals[i] stayed
NULL) and the write loop then SEGV'd. bug-1495.
Fix: skip only negative indexes (`idxs[i] < 0`), and NULL-guard the
write loop defensively.
Verified end-to-end: a locally-built scheduler now connects to the t1
controller and dispatches all 5 test jobs to node t1 (exec_hosts
populated, jobs run and exit). This was the final blocker for job
scheduling (bug-1494: all jobs PEND).
Also: add the t1 host definition to hosts.yaml (worker join was rejected
with -EINVAL because no matching hostconfig existed for hostname "t1").
Co-Authored-By: Claude <noreply@anthropic.com>
worker/src/resource_sync.c init_builtin_load_indices registers 11 load
indices (r15s/r1m/r15m/ut/pg/ls/it/swp/mem/tmp/io) but resources.yaml only
defined r1m/r15s/mem/tmp. The 7 missing ones (r15m/ut/pg/ls/it/swp/io) caused
controller "update_resval: Success" ERRORs and scheduler "unknown resource in
per-worker resvals update" warnings, leaving the worker resource view
incomplete. Add the 7 definitions matching the existing r1m/r15s pattern.
Docs: log bug-1494 (all jobs PEND, scheduler never dispatches; after this
config fix jobs still PEND — remaining suspect is the scheduler→controller
join path after restart), update design doc + verification manual known-issue
tables and root CLAUDE.md cluster-verification status.
Co-Authored-By: Claude <noreply@anthropic.com>
The apiserver returns list rows in OrderMap columnar form —
{"Fields":["col",...],"Data":{"col":val,...}} — but the read clients
unmarshalled each row as a flat object, so every field came back empty
(counts were right, values were all zero/"").
Fix: decode rows as apiRow{Fields, Data map[string]json.RawMessage} and
pull named fields out of Data via rowStr/rowInt/rowInt64/rowFloat helpers.
ListNodes, ListQueues, and getJobs (ListJobs/GetJob) all use apiRow now.
TDD: 3 new ordermap_test.go tests built from live-captured apiserver
response shapes (red: fields empty → green: parsed). Live-verified against
t1: node name=t1 cores=8, 4 queues active, 3 jobs with real fields.
Co-Authored-By: Claude <noreply@anthropic.com>
New §5 in src/docs/chatbot-verification.md: how to connect Claude Code to
the scheduler-agent MCP server. Covers building the mcp-server binary,
`claude mcp add` CLI registration (verified against installed CLI
v2.1.220), the .mcp.json project config schema, the three scopes
(local/user/project), the first-run approval gate, verification commands,
and stdio/env gotchas. Original §5-§7 renumbered to §6-§8.
Co-Authored-By: Claude <noreply@anthropic.com>
New src/docs/chatbot-verification.md: reproducible steps to deploy the
scheduler-agent stack on the t1/t2/t3 test cluster (AppImage build, store
sync, controller/apiserver/scheduler/worker startup) and verify Phase 0-1
(login, mcp_operator API key, read ops, submit). Documents the vcpus quirk,
setsid launch pattern, conf/current pointer mechanism, and known bugs
(job-control HTTP 500 = bug-1490, HA election deadlock).
Also add the manual to the design doc's related-documents index.
Co-Authored-By: Claude <noreply@anthropic.com>
Three fixes found during scheduler-agent Phase 1 cluster verification:
- whitelist.go: add /xce/v1/apikeys to WhitelistPOSTURL (Phase 0 added the
route but never whitelisted it → 403 "URL is not allowed"). bug-1489.
- jobs.go: fix job_status int64 assertion in job control. store.Jobs returns
status as int/float64; the `.(int64)` assertion broke control with
"job_status is not an int64". Add normalizeJobStatus() accepting int/int32/
int64/float64/json.Number.
- rbac.yaml: mcp_operator role gains queue:control (the job-control handler
also checks queue:control on the job's queue); add rbac_mcp_operator_test.
Co-Authored-By: Claude <noreply@anthropic.com>
Complete design spec for the natural-language scheduler agent:
- Layered architecture (UI/LLM/Agent core/Adapter/Scheduler)
- Unified Resource Model (URM) types and principles
- SchedulerAdapter interface + per-backend strategy
- Language selection strategy (Go primary)
- MCP protocol tool set (read/write/advanced)
- Security model (write-protected by default, API Key auth)
- Event stream design with current-state research
- Implementation roadmap Phase 0-4
- Testing strategy with golden tests
- Design principles + ADR decision records
- Current progress status and blockers
Co-Authored-By: Claude <noreply@anthropic.com>
Two pre-existing failures (2026-05-13, commit 491c255) surfaced by the
controller unit-test suite (1070 Tests 2 Failures), both in the config
version storage path:
1. conf_trim_old_versions sorted cluster.bin.* versions by
second-granularity st_mtime. Versions saved within the same second
(rapid replication, or the trim test's 15ms-spaced saves) tie, glibc's
stable mergesort keeps creation order, and the newest (current-pointer)
entry lands at the protected keep_n index — so nothing is pruned and
keep_n+1 files remain. Store st_mtim (struct timespec) and compare
tv_sec then tv_nsec.
2. cr_setup() built its tmpdir path from (pid, time(NULL)) at second
granularity; a Unity assertion failure longjmps past cr_teardown(),
leaking the dir, and the next test (same pid+second) silently inherited
the leftover conf/current — cascading a second failure
(test_conf_get_saved_tag_no_config read a real tag instead of 0). Add a
per-invocation counter to the path so no two tests share a dir.
Verified: nix develop ./src --command bash src/tools/xce-build.sh all →
meson 9/9 Ok 0 Fail, controller 1070 Tests 0 Failures, xceadmin 172
passed, frontend Go all ok.
Co-Authored-By: Claude <noreply@anthropic.com>
The merged origin/working commit e0c5e8b pins capnpc's source to an
immutable commit directly in src/nixSupport/xce.nix (bug-918 canonical
fix, verified with a full controller_unit_test build: 978 Tests 0
Failures). The flake.nix overlay from 98066db solved the same bug with a
ghfast.top-proxied fetchurl tarball; keeping both is redundant and the
proxied tarball hash differs from GitHub's git-archive. Keep the
fetchFromGitHub pin in xce.nix (idiomatic, no third-party mirror,
scoped to the build), drop the global pkgs overlay.
Co-Authored-By: Claude <noreply@anthropic.com>
- Host admin-state (open/closed/locked) now in scope for the upgrade
snapshot; host and queue admin-state frames migrate to independently-
extensible Cap'n Proto structs (one format-version bump covers both)
- bug-1418 blocker cleared: fixed 2026-07-09 via raft-apply-decoupling,
independently re-verified 2026-07-23 with a clean 10-round chaotic soak
- New Design Invariants section (6 standing rules) and Requirement 7
(regression coverage for host-placeholder/rejoin, overlapping upgrades,
format-mismatch rejection, WAL-trim-under-chaos)
Spec reached GO with 15/15 tasks complete (see prior task-confirmation
commits); these doc artifacts were generated during the design/research
phases but never separately staged.
wal_type_name() never had a case for WAL_CONFIG_MARKER (type 50) -- every
config-change marker entry printed as UNKNOWN instead of a real, defined
type. Added the case plus payload decode (tag/sha256) matching config_repl.c's
config_marker_encode/_decode byte layout, hand-decoded here rather than
linking config_repl.c into xce_test's link graph.
Gap detection also had no way to tell "committed prefix missing!" (real
loss) apart from a normal checkpoint-triggered WAL trim (persist.c's
trim_wal_segments, floor-guarded so it never removes a segment a standby
still needs) -- every long-running node's WAL, once checkpointed, would
false-alarm on every inspection. Cross-checks the sibling data/snap/
directory (snapshot LSN is literally in the filename, snap-<hex>.capnp)
and downgrades the message when a snapshot covers the gap.
Adds waldump_stats_t.leading_gap_explained and 3 new tests plus one
assertion on the existing gap test.
trim_wal_segments + rotate_snapshots run on every checkpoint --
operators don't need to manually purge data/. Documents the
snap_keep_n=2 default and that it's a cluster-config field, not a CLI
flag.
--system-site-packages means pip skips vendoring click/pycapnp/pyyaml/
jsonschema since nix already provides them -- deliberate, to keep
pycapnp's ABI-sensitive build pinned to the exact capnproto C library
nix built it against. Activating the venv outside nix develop (or the
AppImage) hits ModuleNotFoundError immediately; not a packaging bug.
Standby-redirect half is done and correct: OP_upgradeConfig has no
server-side redirect wiring today, matches design.md's own documented
"Out of Boundary" gap, and this test locks in the real current
rejection behavior.
Primary-push success-path half is unreliable (~24/25 live fails in one
investigation round, see bug-1485) -- the elected primary was observed
as a <defunct> zombie shortly after promotion, but a dedicated debug
pass could not reproduce it in 26+ clean runs on an idle host and
captured no crash evidence either way. tasks.md marks 6.3 _Blocked:_
pending re-verification on an idle host with real stderr capture.
30-round chaotic soak reported committed_job_loss=1 at round 25, but the
flagged job was confirmed present on every later verify and the final
sweep -- a real loss never comes back. Root-caused from ctrl-1/ctrl-2/ctrl-3
logs, no controller change needed: a newly-elected primary cannot fold a
prior-term entry into its own committed_lsn merely because it holds it
durably (Raft Figure-8 safety via ha_arm_own_term_gate) -- it needs a fresh
majority ack for its own-term WAL_NOOP first. That normally closes in
<200ms, but chaotic mode's overlapping-restart design let it stretch to
6.232s this run (the peer whose ack was needed was itself mid-catch-up from
its own recent restart). verify_classify ran its expect-job check
immediately on primary detection with no settle/retry, unlike verify_no_loss
(bug-1422) or wait_for_full_convergence, so it landed inside that window.
Added a bounded retry (5 attempts, 2s settle) around the LOSS classification
only, mirroring verify_no_loss's precedent -- a job still missing after that
still counts as a genuine loss. Verified with a fresh 30-round chaotic
re-soak: committed_divergence=0 committed_job_loss=0 result=PASS.
