parse_jobsel() parses c-a-ts-seq with an optional [k] or [a-b] under a
fixed grammar and check order (syntax, field widths, array-identity
alignment, element bound, reversed range), saturating digits so an
oversized number cannot wrap; plus is_array_tag(), jobsel_element_id(),
jobsel_strerror() and JOBID_FIELD_MAX_* constants. The selection-cases
conformance table runs in the libxce suite.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With Nagle on, the small TLS record that ends a burst -- a packet header,
the last short chunk of an HA snapshot, a reply -- waited for the peer's
delayed ACK, ~40 ms on Linux: seen on 5 of 40 snapshot transfers. Set
TCP_NODELAY on every netio TCP socket, outbound (socket_create) and
accepted (socket_accept).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
SSL_get_error() reports SSL_ERROR_SSL whenever the calling thread's
OpenSSL error queue is non-empty. io_worker serves every endpoint from
one thread, so an error left by another connection -- a ctrlstatus
client's EOF queues 'unexpected eof while reading' in OpenSSL 3 -- turned
the next plain would-block SSL_read on a healthy connection into a fatal
error. handle_error() then tore that connection down; mid snapshot
transfer this left a rejoining standby with a truncated snap.part until
its liveness timeout forced a reconnect and a fresh transfer.
Clear the queue before each SSL_read/SSL_write/SSL_do_handshake so the
classification is about that call only.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The dispatcher checked each queued task's endpoint busy flag one item at a
time, while the vcpu that finished the endpoint's previous message cleared
the flag without the taskq lock (a lock there deadlocked, bug-1282). Within
one scan, task N was skipped (endpoint still busy), the endpoint freed up,
and task N+1 of the SAME endpoint was dequeued first. Standby controllers
then handled WAL catch-up entry 258 before 257 and NACKed (log mismatch):
26 reorders -> ~576 NACKs in a 45-min production-like run, both standbys
at once at 21:39:24, primary lost its write quorum and answered
jobdispatch/join with E_XCE_NOT_READY, the scheduler client failed over and
rejoined, and the rejoin JOBS_PUSH carried lagging committed state
("refusing resync state ... existing job is RUN" WARNs).
dataq gains dataq_deq_ctx() (a per-scan matcher context). The scan is now
vtask_deq_eligible(taskq, is_blocked): it remembers every endpoint found
busy and keeps all later tasks of that endpoint ineligible for the rest of
the scan (past 64 distinct busy endpoints, every later IO task waits for
the next scan) -- per-endpoint FIFO without any new lock.
Test lib/testsuite/test_vtask_order.c injects is_blocked to free the
endpoint mid-scan: task N+1 was dequeued before the fix, none after; other
endpoints and FIFO/generic tasks unaffected. Suites: lib 1/1, controller
5/5, worker 10/10, scheduler 87/87, harness 5/5. Chaotic HA soak 2/2 PASS
with 0 WAL reorders (2 and 5 per run before), no ASAN.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
pthread_cond_wait() reacquires its mutex before honoring a cancellation,
but mylock.c installed no cleanup handler to release it again -- a
thread cancelled mid-wait terminated holding the lock forever. vtaskmgr's
pthread_cancel()-based teardown could land there from three angles
(dispatcher cancelled while polling the shared queue, or a vcpu racing
back into it after its own shutdown signal), permanently deadlocking any
other thread that touched the same queue and hanging vtaskmgr_fini()'s
pthread_join() indefinitely.
Reproduced live as xce_test hanging on Ctrl-D exit once a connection
context was entered; gdb confirmed a vcpu thread blocked forever on a
mutex owned by an already-exited thread ID.
Fix: cbuf_put()/cbuf_get()/cbuf_timedget() now unlock via a
pthread_cleanup_push/pop handler around the wait, so a cancelled waiter
always releases the mutex. vcpu_idle()/vcpu_entry() also recognize the
teardown poison pill and exit immediately instead of needlessly
re-announcing into the already-dead shared queue.
New test_vtaskmgr_teardown.c reproduces the race deterministically
(self-pinned to one CPU, since true multi-core parallelism rarely hits
the window) and bounds vtaskmgr_fini() with a timed join so a
regression fails fast instead of hanging the suite. Logged as bug-1810.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds apply_job_task_count() to overwrite a standalone primary resreq's
minTasks/maxTasks with a job's real requested task count, no-op for
COMPOUND/ALTER nested primaries, with a defensive min>max clamp.
Closes the gap that lets scheduler-allocation's multi-host allocator
ever be reached (bug-1770). Wiring at live ingest call sites follows
in subsequent tasks.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds rusage_hash to xce_primary_resreq_t (computed once by
parse_resreq_primary() via a new hash_rusages() helper, hashing
rusage content rather than raw struct bytes since resname is a
non-interned strdup'd pointer), the xce_resreq_fingerprint_t value
type, and resreq_fp on xce_primary_schedobj_t, populated once at real
job-ingest time in core_build_primary_schedobj().
Task 1.2 (scheduler-allocation spec). Requirement 7.1.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Add xce_job_state_query_req()/xce_job_state_report_req() to request.c,
paginating job IDs / job-state entries across OPCode_jobStateQuery /
OPCode_jobStateReport messages via direct Request.body union arms
(mirroring xce_jobdispatch_req/xce_notify_req's direct-send shape, not
xce_push_data_req's XceData-table multiplexing -- these are dedicated
union arms, not multiplexed push/reply-get traffic). 8 round-trip tests
(normal/empty/multi-page for both query and report, covering all 4
JobStateStatus values and both queried states) all pass.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
vtask_enq() published a task onto the shared dispatch queue via
dataq_enq() before finishing its own initialization (task->owner) -- a
concurrent vcpu could dequeue, run, and free the task before the
producer's own write executed, an ASAN-confirmed UAF that crashed 2 of
3 controllers under a heavy burst-array-job HA workload. Moved
task->owner = vtaskmgr before the publish, mirroring the endpoint_ref-
before-publish discipline already established two lines above it
(bug-1298). Adds a stress-test regression guard (test_vtask_enq_race.c).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
make_job_account() allocates account->accounts via calloc(1,
sizeof(xce_host_account_t*)) -- room for exactly 1 host pointer -- but
its own loop writes account->accounts[i] for i up to jalloc->nhosts-1,
overflowing the heap allocation for any job spanning more than one
host. Already reachable from live production dispatch (core_events.c,
schedobj.c, scheduler.c) for any multi-host job -- found while
verifying viability of a proposed scheduler snapshot-allocation-capture
fix, not from a user report.
Sibling function make_host_allocs() has the identical bug shape
(calloc(1, ...) vs a loop writing up to nhosts-1); every current call
site happens to pass nhosts=1 so it has never actually overflowed in
production, but it's the same live footgun in a public exported
helper. Fixed alongside since it's the identical bug class in the same
file.
New test_xcealloc.c drives a 2-host allocation through both functions,
reproducing the exact ASAN heap-buffer-overflow at xcealloc.c:188
pre-fix (RED), passing post-fix (GREEN). Both calloc(1, ...) sites
changed to size the allocation by the actual host count.
Full regression: libxce 303/303 (was 301, +2 new), scheduler 53/54 (1
pre-existing unrelated bug-1560 flake), controller 1211/1213 (2
pre-existing unrelated bug-1561 overlayfs flakes) -- zero new
failures.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
XCECLI_FF_CALL (backing xcecli_heartbeat) can silently rotate+reconnect
the transport entirely inside a single call and still return success, so
heartbeat.go's failure-to-success transition detector could never observe
a reconnect that completes within one heartbeat tick -- the common case,
since elections finish in single-digit seconds against a default 30s
interval. apiserver's application-level rejoin never fired on a real
idle-period failover, and job pushes silently stopped reaching it.
Adds a monotonic reconnect_gen counter to endpoint_t (netio.h), bumped by
xce_ctx_reconnect() (xcectx.c) on every successful reconnect regardless of
trigger path. heartbeat.go now compares this generation every tick instead
of tracking success/failure transitions. Re-validated live against task
5.2's exact scenario (apiserver alone, zero write traffic): rejoin fired
22s after a primary kill, 12s after election.
Reopens and closes out component-failover-adoption's own NO-GO -> GO
validation cycle from 2026-08-15.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Raft-drive-async's reconnect-offload burst (Task 2.1/2.2) exposed a
latent race in io_worker's epoll dispatch loop: ha.c's
on_standby_connected() evicts a stale duplicate peer channel via
iomgr_unmonitor()+endpoint_unref(), but epoll_ctl(DEL) only stops
FUTURE epoll_wait() calls -- it does not invalidate an event for that
same endpoint already sitting in the CURRENT batch's events[] array.
A later event in the same batch then dereferences the freed
endpoint_t/SSL*, producing the ASAN SEGV observed deep inside
OpenSSL's ssl3_read_bytes during raft-drive-async's round-9 soak.
The original "concurrent SSL_do_handshake from two io_worker threads"
hypothesis was wrong and is retracted -- iomgr_start() creates exactly
one io_worker thread per iomgr_t.
Fix: extract io_worker_dispatch_batch(), which takes endpoint_ref() on
every endpoint in the batch before dispatching any of them and
endpoint_unref()s each only after the whole batch completes -- the
same lock-free ref/unref bracket already used for bug-1298 and
INVOKE_MSG_HANDLER, just never applied to io_worker's own loop. Pure
atomics, no locks, no blocking added to the hot path.
Adds test_netio_batch_dispatch_uaf.c, which drives the real dispatch
loop with a synthetic epoll_event[] batch (TDD RED confirmed via ASAN
heap-use-after-free before the fix, GREEN after). Verified further
with two independent clean 10-round live sequential HA soaks (33 and
36 jobs, zero loss) and full ASAN suite runs (xcelib_test 294/0,
controller_unit_test 972/0). raft-drive-async's kiro-validate-impl
re-run flips NO-GO to GO.
restore_job_array()'s bulk-insert pass could fail partway through
(key_add() on realloc growth exhaustion), leaving an already-inserted
prefix permanently visible in c->jobmgr but excluded from
out_new_jobs -- so those elements existed and were individually
addressable but never reached HOOK_DETAIL_JOB_NEW, never scheduled or
pushed, not on this attempt or any later successful retry. A literal
violation of Requirements 2.3 and 9.4, tracked as bug-1443.
Adds kvmgr_reserve()/xzarray_reserve(), new purely-additive primitives
in shared kvmgr.c/zarray.c (following the existing kvmgr_capture_keys()
template -- bare extern function, no RTTI vtable slot, caller-holds-
write-lock contract), that pre-grow c->jobmgr's backing store before
the bulk-insert pass begins, closing the dominant N-scaling failure
trigger structurally.
Found a genuine off-by-one in the existing, unmodified
xzarray_ensure_items(): its strict less-than sufficiency check means
reserving exactly array_size and then performing array_size real
stores still triggers one more growth cycle on the last store.
Reserving array_size + 1 is the exact minimal correct compensation,
independently re-derived and confirmed by review with concrete
arithmetic.
For the now-much-narrower residual case (kvtab_add()'s own small
per-entry allocation, or a content collision -- both non-N-scaling,
honestly documented as still theoretically possible): restore_job_array()
now populates out_new_jobs with whatever prefix was genuinely inserted
before a failure, and apply_committed_entry() fires HOOK_DETAIL_JOB_NEW
for that non-empty partial prefix even when the overall return is
negative, while still propagating the negative rc so the LSN correctly
halts and the Live Apply Resource Fence still engages for the
remainder. A genuinely-inserted element is never left un-pushed again.
Task 7.1 of job-array-single-wal-record (Requirements 2.3, 9.4, 9.6).
- job_init() now checks makeJobsub()/makeString(ckptinfo) return
values and propagates failure instead of storing NULL with a
success return (Req 10.1).
- Root-caused the previously observed makeJobsub() NULL-with-EINVAL
failure to a test-binary-only hazard: multiple controller test
files each called job_obj_init() from their own setUp() with no
cross-file coordination, re-running jobsub_table_init() against an
already-live lock/table. Fixed via atomic_flag-guarded idempotent
job_obj_init()/job_obj_fini(), plus a distinct -ENODEV return from
makeJobsub()/makeJobstartinfo() when the table isn't ready
(previously fell through to undefined behavior) (Req 10.2).
- Verified (test) MOVE_PTR fields are always NULL on a freshly-built
submission payload -- no code change needed (Req 10.3).
- Fixed job_destroy()'s accounts leak via the existing
free_job_account() helper, replacing the stale FIXME (Req 10.4).
- Verified (inspection + test) struct Job's capnp schema has no
ckptinfo field, so decode_Job/free_Job have nothing to leak there
-- no code change needed (Req 10.5).
Independently re-verified: reviewer reproduced a pre-existing,
unrelated rtti-library leak-on-init-failure against a stashed
pre-task copy of jobj.c to confirm it predates this change.
Task 1.3 of job-array-single-wal-record.
Replace the deliberate election-carrier leak with an atomic refcount:
a bias reference held by ha->elect_ctx plus one reference per captured
vote endpoint, released via a new endpoint_t.on_fini teardown-completion
hook (fired once, before ep->data is freed). Both real-election and
PreVote capture sites (raft_cb_send_vote_request,
raft_cb_send_pre_vote_request) increment on capture; both publish sites
(ha_start_election_async, ha_start_prevote_async) now atomic_exchange
the outgoing carrier and release it instead of leaking or freeing
outright, so a still-outstanding vote endpoint keeps the superseded
carrier alive until its own teardown completes.
Verified: 758+ controller tests green, isolated LeakSanitizer run shows
zero ha_elect_ctx_t leaks across many election/PreVote rounds, no
premature-free/UAF under supersession, and a live 10-round HA soak
(36 jobs, zero loss, all 3 nodes served as primary).
vtaskmgr gains vtask_class_t (LIVENESS/LIVE_BLOCK/REQ_BLOCK) + a pool-level
blocking reservation: REQ_BLOCK fast-rejects -XCE_ERR_NOT_READY past
n_vcpus-R; LIVENESS never gated (deadlock-freedom). commit_submit folds its
inflight_waits onto the pool via commit_set_pool; production wired in
cluster.c phase2 (vtaskmgr_set_liveness_reserved(R) + commit_set_pool).
Lock order p->lock->ce->lock->blk_lock. Dormant until 3.1. Commit suites
38/30/22/38 green; lib 282/0, controller 196/0, ASAN clean. Phase 1 done.
socket_connect sets O_NONBLOCK before connect and bounds the attempt with
poll(timeout); new connect_timeout_ms param (0=DEFAULT_CONNECT_TIMEOUT_MS);
all 5 callers updated (ha.c x3, client.c, xcectx.c). Revalidation Trigger:
shared netio helper signature changed. Broad build links all components;
xcelib_test 278/0, controller_unit_test 194/0. 1.4/1.5 unit tests satisfied
by the TDD output of 1.2/1.3 (test_slowlane.c, test_netio_bounded_connect.c).
Add io_classify_io_error and propagate endpoint/epoll errors through
endpoint_update_epoll and cbuf_get instead of swallowing them, so client
failover surfaces the real error rather than hanging. Adds netio arm-failure,
close-propagation, and request send-propagation tests.
Pre-existing uncommitted work, committed to establish a clean baseline before
ha-commit-durability implementation.
Add void (*on_connected)(endpoint_t *ep) field to endpoint_t with
endpoint_set_on_connected() helper. io_handler fires it exactly once
immediately after ssl_handshake_done=1 then nulls the pointer.
Enables ha_start_election_async to send OP_requestVote as soon as
the TLS handshake with a peer completes — no blocking SSL_connect.
6 new unit tests pass; controller suite 1/1 OK.
Worker (P2.5-5):
- Added parse_peers() static helper to src/worker/src/main.c
- Switched getopt to getopt_long to support --peers and --ca-cert long options
- When --peers is given: parses into xce_cluster_config_t, calls
xce_cluster_config_validate(), logs INFO with candidate count/identity/ca_cert
- If validation fails: prints clear error and exits(1)
- WARN logged when candidates_count > 3
- Backward compatible: no --peers falls back to existing single-controller path
Scheduler (P2.5-6):
- Added parse_peers() static helper to src/scheduler/src/main.c
- Extended argp struct with --peers (-P) and --ca-cert (-C) options
- Same validation/logging pattern as worker
- Backward compatible: no --peers falls back to existing single-controller path
Tests (new):
- src/lib/testsuite/test_cluster_parse.c: 8 unit tests for parse_peers logic
(single entry, multiple entries, no-name format, empty/null/missing-port errors,
name-stripped correctly, MAX_HA_CANDIDATES cap respected)
- Registered in test.c and meson.build
- All 8 tests GREEN; 3 pre-existing failures in unrelated GC/CSE tests unchanged
- hook_item_t.fn changed to _Atomic hook_fn_t (C11 stdatomic)
- xce_disable_hook(): atomically stores NULL into the slot fn field;
returns 0 on success, -ENOENT if fn not found, -EINVAL on bad args
- xce_add_hook(): scans for NULL (disabled) slots before allocating;
uses atomic_store_explicit(release) when writing fn pointer
- xce_run_hooks(): atomic_load_explicit(acquire) + NULL guard so
disabled slots are silently skipped without disrupting the loop
- Add src/lib/testsuite/test_hooks.c with 3 TDD tests:
test_disable_hook_stops_invocation
test_disable_hook_returns_enoent_if_not_found
test_add_hook_reuses_null_slot
- Register test_hooks in meson.build and test.c runner
Required by ha_demote_to_standby() to atomically disable persistence
hooks when the controller steps down from primary, preventing unbounded
hook list growth across promote/demote cycles (resolution A6-015,
A7-004, A7-016).
Add kvtab_foreach walk primitive (P0-14) to kv.c/kv.h, then implement
kvmgr_capture_keys in kvmgr.c that deep-copies all key→index mappings
into a malloc'd slot_key_t[] array safe to read after the write lock is
released during checkpoint serialization (Part 3.8).
Key design points:
- Both kvtab_foreach and kvmgr_capture_keys take var (RTTI data pointer)
and use RTTI_DATA_TO_OBJ internally — the RTTI framework's makeInstance
returns the data region pointer, not the meta_obj_t header pointer
- String keys are copied via strncpy into slot_key_t.key[XCE_MAXNAME_LEN]
- Integer keys are formatted as decimal strings via snprintf
- Two-pass approach: count live entries first, allocate, then fill
- Tests cover string kvmgr (3 entries), empty kvmgr, and deep-copy safety
rtti header already present at rtti-0.0.2 with vfreeze_fn_t, vthaw_fn_t,
retain_fn_t, and the two new slots in intf_varray_t. No header modification
needed.
Convert intf_varray_t compound-literal initializers in zarray.c and kvmgr.c
to designated style with explicit .vfreeze = NULL and .vthaw = NULL slots.
kv.c has no intf_varray_t initializer.
Add test_zarray_cow.c with test_freeze_thaw_slots_exist which verifies the
struct layout has the slots accessible and that they are NULL until P0-2
wires the real implementations. Wire test into meson.build and test.c.
Full clean rebuild confirmed: 27/28 targets linked, no new test failures.
Only the 4 pre-existing failures remain.
Add the full call chain for host control operations from the Go API
server through the C library to the controller:
- schema/controller.capnp: HostControl struct + hostctrlmsg @21
in Request.body union
- include/typemap.h: xce_hostctrl_t* hostctrlmsg in body union
(placed after @20 clrpreasonmsg, matching capnp ordinal)
- include/client.h, xceapi.h, request.h: declarations for
xcecli_hostctrl, xce_hostctrl, xce_hostctrl_req
- lib/client.c: xcecli_hostctrl (validate + delegate to xce_hostctrl)
- lib/xceapi.c: xce_hostctrl (validate ctx state + send + wait reply)
- lib/request.c: xce_hostctrl_req (validate hostname/empty/nil,
accept only HOST_OPEN/HOST_CLOSE, reject HOST_TAG/UNTAG)
- lib/helper.c: Request_body_hostctrlmsg cases in
xce_request_body() and xce_request_msg_name()
- controller/cluster.c: OPCode_hostcontrol dispatch
- controller/include/cluster.h: do_hostcontrol_op declaration
- controller/worker.c: do_hostcontrol_op + fix pre-existing
goto-clean bug in worker_ctrl that leaked write lock and
reference on HOST_OPEN/HOST_CLOSE no-op paths
TDD: 8 xce_hostctrl_req validation tests (null ep, null ctrl,
null/empty hostname, HOST_DO_NONE no-op, HOST_TAG/UNTAG reject,
negative action reject, HOST_OPEN/CLOSE accept via xce_request).
116/116 xcelib tests pass. CGO_ENABLED=0 Go backend tests pass.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix issue to xce_get() api cause client side crash
- add parse resreq section
- add rusage parser
- parse resreq with basic support alternative & compound, still miss select
