The controller computed run usage (CPU time, memory, CPU peak and
efficiency, LSF bjobs -l definitions) but the capnp Job struct had no
field for it, so the job pushed to the apiserver never carried it, and
snapshots (the same struct) dropped it: a job restored from a checkpoint
lost its accumulated usage.
- controller.capnp: Job.runUsage @13 (JobRunusage); old entries decode
with no run usage.
- apiserver: map run_usage into JobInfo; max_mem, avg_mem, used_mem,
cpu_time, peak_cpu, peak_cpu_time, peak_cpu_duration, mem_efficiency,
avg_cpu_efficiency, peak_cpu_efficiency become queryable columns.
- xcecli: -o fields MAX_MEM, AVG_MEM, MEM_EFFICIENCY, CPU_PEAK,
CPU_PEAK_DURATION, CPU_EFFICIENCY, CPU_PEAK_EFFICIENCY (CPU_TIME now
renders too), and a MEMORY USAGE / CPU USAGE block in jobs list -l.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Regression from 63fd3d2a (bug-1870), found by the --config-upgrade HA soak
(force-elect: "submit on {ctrl-2,ctrl-1} failed", ~640 standby NACKs;
bisected: 725301dd passes, 88c4310e fails).
load_snapshot() seeded current_lsn = checkpoint_lsn, but current_lsn is
the NEXT lsn to write and the snapshot covers 1..checkpoint_lsn. The
off-by-one was always there, hidden because every checkpoint also wrote a
WAL_CHECKPOINT marker ABOVE the checkpoint, so the WAL scan moved
current_lsn past it. With the marker gone, a standby whose WAL held
nothing above its config-marker checkpoint (LSN 11) recovered with
current_lsn=11: its raft layer counted entry 11 as held (ae_already_have,
not written), and persist_wal_write_replicated() then refused entry 12 as
a gap (ae_append_persist_fail) on every resend -- stuck at 11 forever.
Both of load_snapshot's commit points now set current_lsn =
checkpoint_lsn + 1, matching the apply-side catch-up path (lsn + 1).
New test_recover_from_snapshot_without_wal_resumes_after_it (RED before:
current_lsn 3, not 4). Two snapshot-entity tests asserted the old
current_lsn == checkpoint_lsn and now expect +1. Controller suite 1245/0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The capnp Job struct -- the checkpoint/entity-snapshot form of xce_job_t --
had no exit-info field, so capnp_encode_job() never wrote job.exit_info
and a job restored from a snapshot (instead of WAL replay) came back with
exit_info NULL, for DONE/EXIT and failed jobs alike. bug-1689/bug-1754
made the WAL facts carry the code; anything older than the last
checkpoint's WAL trim still lost it.
- controller.capnp: struct JobExitInfo $C.mapname("job_exitinfo_t")
{ exitcode }, and Job.exitInfo @12 $C.mapname("exit_info"). The
generator now emits encode/decode/free for it inside encode_Job /
decode_Job / free_Job; restore_job()'s create_job() already takes
exit_info via MOVE_PTR. Null stays null (no fabricated exit 0). A new
ordinal, so older snapshots/recordings decode with exit_info NULL.
New test_snapent_checkpoint_exit_info_roundtrip (RED before): exit codes
0 and 137 and a NULL survive both the raw checkpoint frames and a real
persist_load_snapshot restore. Full /xce-build all (asan): 111 meson
suites, 0 failures; all Go packages pass.
Not live-verified: the --force-snapshot HA soak cannot reach its snapshot
step because of a pre-existing checkpoint-marker replication stall,
reproduced on the baseline without this change (bug-1870).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
load_snapshot_entities' queue-frame decode migrates from the old raw
{name_len,name,status} layout to {payload_len, capnp QueueAdminState
bytes}, still calling the unmodified restore_queue_state. Adds a new
host-frame section decode after queues, dispatching to
restore_host_state (worker.c, task 3) with the same bounds-checking
and restore-before-advance discipline as the existing job/queue loops.
Un-ignores test_snapent_load_snapshot_restores_entities_before_advancing_position
(blocked since task 2.2) -- now passes for real. Also fixes a genuine
regression in test_snapshot_recv_out_of_order_chunks_assemble_correctly
(test_config_repl.c): its synthetic checkpoint-body filler landed on
the new host_count field's offset, tripping the cluster==NULL guard --
pinned to a well-formed host_count=0, matching that test's existing
fmt/job_count/queue_count precedent. Task 4 of 19.
Requirements: 1.10
write_snapshot now appends a flat [config_bytes_len][raw bytes]
section after the host frames, reading the locally-held active config
straight off disk (read_active_config_bytes, persist.c) rather than
calling into config_repl.c -- mirrors scan_latest_committed_config_ref's
existing precedent of avoiding a persist.c -> config_repl.c
function-call dependency. Populated on every checkpoint, not only
marker-triggered ones (Req 2.4). No local config yet (early boot)
degrades to an explicit empty section rather than failing the
checkpoint; genuine corruption still fails loud. Task 2.3 of 19.
Requirements: 2.1, 2.3, 2.4, 5.1
collect_queue_frames now encodes each queue as a QueueAdminState capnp
frame (capnp_encode_queue_admin_state, mirroring task 2.1's host-frame
encoder) instead of the raw {name_len,name,status} layout;
write_snapshot's queue trailer writes [payload_len][capnp bytes] per
queue. restore_queue_state and load_snapshot's queue decode are
untouched (task 4's boundary).
Temporarily TEST_IGNORE_MESSAGE's
test_snapent_load_snapshot_restores_entities_before_advancing_position
(m_queues=2 real write+load round trip hits the still-raw loader) --
tracked in tasks.md Implementation Notes, must be re-enabled by task 4.
Matches existing codebase precedent (bug-847) for this pattern. Task
2.2 of 19.
Requirements: 1.9, 5.7
Adds collect_host_frames + capnp_encode_host_admin_state, walking
c->workermgr the same way collect_job_frames/collect_queue_frames do.
Every host currently in the table gets one HostAdminState frame
(status masked to XCE_WORKER_ADMIN_MASK only, no dynamic/liveness
fields), written into write_snapshot after the queue trailer. Task 2.1
of 19.
Requirements: 1.6, 1.7, 1.9
Adds independently-extensible Cap'n Proto frames for the upgrade
snapshot's host/queue admin-state sections (frame-only, not reusing
Worker/QueueConf which bundle dynamic state) and bumps
SNAPSHOT_FORMAT_VERSION 1->2 so a checkpoint written under the prior
raw queue-frame layout is fail-loud rejected rather than
misinterpreted. Task 1 of 19.
Requirements: 1.9, 5.7
Only 21 of 86 test_*.c group functions called their own nested
UNITY_BEGIN()/UNITY_END(). Each nested call reset Unity's global
NumberOfTests/TestFailures counters, so controller_test.c's final
`return UNITY_END();` -- the value meson test actually gates on --
only ever reflected whichever small window of tests ran since the
LAST nested reset, silently discarding pass/fail evidence for
everything before it (measured: 16 of 972 total tests in one run).
The same nested-BEGIN calls also caused PASS/FAIL banners to show
the wrong source file for tests in files that don't reset it
themselves (Unity.TestFile just carries over from whichever group
last called UNITY_BEGIN()).
Fix: replace every nested UNITY_BEGIN() with UnitySetTestFile(__FILE__)
(sets file attribution only, no counter reset) and drop the matching
UNITY_END(); add the same UnitySetTestFile(__FILE__) as the first
statement to the other group functions that had neither. The 2
genuinely standalone binaries (test_job_array_sched_push.c,
test_job_array_apiserver_push.c -- each has its own int main(), not
registered via controller_test.c's TEST() macro) are untouched.
Verified: single consolidated summary now reads
"972 Tests 0 Failures 0 Ignored OK", exactly matching the
ground-truth RUN_TEST call-site count across all 83 registered
group files (976 total minus 4 belonging to the 2 standalone
binaries). Every PASS/FAIL banner now names its own true source
file. Full xce-build.sh all regression pass clean (Ok: 9, Fail: 0).
No production code touched -- test-harness plumbing only.
Extends persist_load_snapshot's public signature with an
out_entities_restored bool*, per design.md's documented contract. The
static load_snapshot function -- the real restore logic -- is completely
untouched (still today's watermark-only implementation, byte-identical
to before this task); only a thin wrapper was added that naively maps
*out_entities_restored = (rc==0), deliberately scaffolding so the 3 new
RED tests fail for genuine reasons rather than accidentally passing.
ha_handle_snapshot_chunk's sole call site gets a mechanical
declare-and-pass update -- confirmed inert, never reads the new
out-param (task 6.1's job).
Three tests against persist_load_snapshot directly (unlike prior tasks'
tests, which deliberately stayed on the raw-byte-parsing side to avoid
this exact function): restore-before-advance (fails at the job-lookup
assertion -- nothing is ever restored today), corrupted-entity-leaves-
position-unchanged (fails at rc!=0 -- today's code ignores the corrupted
section entirely, since load_snapshot never reads past the legacy
header), legacy-file-reports-position-only (fails at entities_restored==
false -- the naive wrapper claims restoration for every rc==0 case,
including legacy files).
This is the exact function/call site that caused task 2.2's rejection
(a live position-advances-before-restore-validates hazard). Independent
review verified load_snapshot's body is byte-identical to commit
d52eb94, the wrapper does nothing beyond the naive mapping, and ha.c's
change never reads the new out-param anywhere -- no repeat of that
mistake here.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
A checkpoint file whose position matches an in-flight peer transfer
(task 4.1's snapshot_inflight_lsn[]) must survive rotate_snapshots even
when snap_keep_n would otherwise delete it. Genuinely fails today (file
removed, access() returns -1) since rotate_snapshots has no awareness of
in-flight transfers yet.
Executed directly (no subagent dispatch) per the hybrid execution-mode
decision -- small, single-file, isolated test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
Queue's (name, status) round-trip through write_snapshot's raw on-disk
bytes -- a plain fixed record (no capnp; queue definitions are
config-owned, only runtime admin status is WAL/snapshot-tracked, per
design.md's corrected queue model). Fails today (Expected 4 Was 0) since
write_snapshot still hardcodes queue_count=0 from task 2.2's placeholder.
Does not call persist_load_snapshot, following task 2.2's established
raw-byte-parsing pattern.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
write_snapshot now Cap'n-Proto-encodes every retained job under nested
table+per-entity RTTI_LOCK_READ (table lock alone does not exclude a
concurrent per-entity writer -- design.md's central correctness finding),
appends length-prefixed job frames after a new format_version marker, and
fixes a previously-unchecked fdatasync return. Any failure anywhere in the
path (encode, write, sync) discards the whole checkpoint -- no partial
file at the final path, proven by a real mid-encode-loop fault-injection
test (forces one specific job's encode to fail among N, asserts nothing
survives).
Also fixed two pre-existing bugs in task 1.1's test fixture: terminal-state
seeded jobs need a non-zero finish_time or job_should_retain's retention
window silently excludes them from any checkpoint; the fixture needs
xstring_table_init or every seeded job's submit/start_info silently
decodes as NULL.
Round 1 of this task also touched load_snapshot/persist.h to make the
round-trip tests pass end-to-end, but review caught a live hazard: it
advanced current_lsn/checkpoint_lsn BEFORE the entity-decode loop ran, so
a decode failure still left the position advanced -- reachable today
through the unmodified ha_handle_snapshot_chunk production path, not
hypothetical. Remediated by fully reverting load_snapshot/persist.h to
byte-identical pre-task-2.2 state and rewriting the round-trip tests to
parse write_snapshot's raw on-disk bytes directly (read_Job/decode_Job)
instead of round-tripping through persist_load_snapshot -- proves the
write side without depending on load_snapshot, which stays task 5.2's
scope entirely, not started here.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
Three tests against the current sentinel-only write_snapshot/load_snapshot:
job content round-trip and concurrent-mutation-not-torn genuinely FAIL
(RED) since no entity content is written today; write-failure-discard
already passes (write_snapshot's open()-failure path already discards
correctly) -- flagged transparently as a GREEN-from-day-one regression
guard rather than faked RED, per real fault injection (directory removal
-> genuine ENOENT).
Carries forward two notes for task 2.2: fdatasync's return value is
currently unchecked in write_snapshot (persist.c:1505), and a mid-encode-
loop fault-injection test should be added once the per-job encode loop
exists.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
Adds the ha-entity-snapshots Kiro spec (requirements, design, research,
tasks) for task 5.10a / raft-ha-core-safety: making the HA checkpoint an
actually-restorable copy of committed job/queue state instead of a
position-only sentinel, closing the residual chaotic-soak divergence gap.
Task 1.1 (foundation): reusable test fixture snapent_make/snapent_teardown
in a new test_persist_snapshot_entities.c, seeding N jobs (varying state)
and M queues (varying status) into a real persist_t+cluster_t for later
write/load/rotation-guard tests to build on. Build clean; new suite 1/1;
regression spot-checks (test_persist_hooks 19/19, test_commit_barrier_reply
22/22) clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
