bug-1453: remove unused client-side CA-certificate plumbing. C-side:
drop ca_cert_path from xce_cluster_config_t, do_connect()'s ca_cert
param, --ca-cert flags on worker/scheduler, and controller's cert
population (TLS is server-side only, SSL_VERIFY_NONE everywhere on
the client side already, verified zero SSL_CTX_load_verify_locations
calls in the C tree). Go-side: delete xcecli's self-signed cert
TOFU-pinning flow (cmd/auth/cert/*), CACert field, SetRootCAs/
loadRootCAs, and every call site.
bug-1454: fix xcecli_ctx_t leak in scheduler.c sched_init() — the
unconditional initial ctx_init was never freed before the HA
candidate loop re-initialized it on the first iteration.
bug-1455: sched_init()'s memset(sched, 0, ...) was silently wiping
sched->cluster_cfg that main.c had populated from --peers just before
calling sched_init(), so --peers was always ignored and the scheduler
fell back to the hardcoded localhost:9990 single-host default.
Restructured sched_init() to take cluster_cfg as an explicit parameter
(assigned after the memset) instead of relying on caller-side
ordering with a separate sched_set_cluster_cfg() setter, which is now
removed.
bug-1456: revert an unsafe stack-local xce_client_config_t introduced
while fixing bug-1455 — xcecli_ctx_init() stores that pointer as
ctx->conf and client_failover.c dereferences it for the ctx's entire
life (runtime reconnect), so a per-loop-iteration stack copy would
dangle the moment sched_init() returned. Back to mutating the
caller-owned, process-lifetime config in place.
Verified: full ASAN build (all components), meson test suite 9/9 OK,
manual scheduler testsuite 15/15, Go build/vet/test all pass.
Dedicated-pthread pool isolated from the vtask pool; non-blocking slowlane_submit
(0 enqueued / -EEXIST deduped / -EAGAIN cap-full-deferred), admission cap on
running tasks, per-key in-flight dedup, run-exactly-once, drain+join fini. Wired
into controller lifecycle (dormant; consumers land in 2.x/3.x). 7 unit tests,
mutation-verified cap+dedup; full suite 194/0 green, 15x ASAN.
cluster_t embeds commit_engine_t; init in cluster_init_phase2 after the
vtaskmgr guard (idempotence-guarded — phase2 can run twice), wiring
repl_set_commit_engine + ha.repl back-pointer before any standby can
register; fini after iomgr/vtask shutdown. persist_wal_sync_append
passes written_lsn; engine-wired path drops -ENODEV and the
committed_lsn store (engine owns advancement); LSN-0 first-entry
routes to legacy per the accepted interim. p2 -ENODEV pin updated to
the Req 6.3 contract. 34/34 + 6/6 green through honest delegation;
census exactly at the 3 validated baselines; production binary links.
- config_repl.c: add handle_repl_config_push — standby handler for
OPCode_replConfigPush. Verifies SHA-256, saves blob to disk via
conf_save_config, reloads via load_primary_config, ACKs primary.
- config_repl.h: declare handle_repl_config_push.
- cluster.c: dispatch OPCode_replConfigPush to handle_repl_config_push;
populate c->myhost_info.name from myconf->name (ctrl-1/ctrl-2/ctrl-3)
instead of the static "controller" placeholder.
- ha.c: fix raft_cb_become_leader push_config_to_standbys call to pass
real config bytes and tag read from disk (was NULL/0 — standbys got
empty push and never saved config).
- typemap.h: add raw_sha256/raw_bytes/raw_bytes_len fields to
repl_config_push_req_t (mirrors upgrade_config_req_t pattern).
- request.c: pre-extract List(UInt8) sha256/bytes from OPCode_replConfigPush
before capn_free() in xce_decode(); free them in xce_free_request().
- helper.c: add all HA/replication opcode names to xce_request_op_name
so "unknown opcode 30" becomes "replConfigPush" in logs.
- cluster_def.h: add embedded myhost_info field to cluster_t struct.
- ha.h: declare ha_start_primary_heartbeat; remove dead
ha_find_and_connect_to_primary declaration.
Add ctrl_tunables struct to cluster_t with per-table fields:
jobmgr_nhash/delta, workermgr_nhash/delta, qmgr_nhash/delta,
resmgr_nhash/delta/inst, xstrings_nhash
init_controller_tunables() sets defaults (jobs=512, others=128)
and is called from cluster_init_phase1 so values are available
before cluster_init_phase2. Phase2 reads from c->tunables instead
of cluster_config or local #defines.
Each table gets its own values since they hold different item counts:
jobs can scale to hundreds, workers/queues/resources are typically
single-digit to tens.
- errnox.h: add E_XCE_NOT_READY and E_XCE_STANDBY error codes with
extern int XCE_ERR_NOT_READY / XCE_ERR_STANDBY runtime variables
- ha.h: declare ha_dispatch_gate(ha_state_t *, int opcode)
- ha.c: define XCE_ERR_* variables; initialize heartbeat_seen semaphore
and primary_ctrl_fd = -1 in ha_init(); sem_destroy in ha_fini();
implement ha_dispatch_gate() with allowlist for CTRL_INIT/CTRL_ELECTING
(only OP_requestVote and OP_probeConfigTag pass) and CTRL_STANDBY
(replication, election, and housekeeping ops pass; primary-only ops
return -XCE_ERR_STANDBY)
- cluster.c: wire ha_dispatch_gate() in myreq_handler() before opcode
switch; gate failure sends error reply and returns early
All 393 unit tests pass. Satisfies requirements 1.3 and 1.4.
When a worker re-joins and already exists in workermgr, makeWorker_()
now overrides the existing slot instead of returning EEXIST. Admin-set
flags (LOCKEDU, CLOSED, HIDDEN, CLOSED_BY_ADMIN) survive the override.
Add XCE_WORKER_CLOSED_BY_ADMIN flag to distinguish admin-closed hosts
from workload-closed ones. worker_ctrl sets/clears it alongside CLOSED.
Refactor worker_init/worker_destroy into copy_worker/kill_worker helpers
shared by both fresh creation and override paths.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three identical static copies (controller, worker, scheduler) replaced
with a single xce_parse_peers() in src/lib/cluster.c.
The libxce version is the canonical implementation: stores the logical
name from 'name=host:port' syntax so daemons on the same dev host can
identify themselves by name rather than IP address.
- xce_candidate_t gains name[] field for logical peer identity
- parse_peers stores name prefix before '=' (empty when omitted)
- cluster_init matches own identity by name first, hostname as fallback
enabling multiple controllers on the same dev host via
--peers ctrl-1=10.0.1.1:7000,ctrl-2=10.0.1.2:7001
Add redirect helpers to ha.c/ha.h so a standby controller never accepts
join requests as if it were the primary:
- ha_should_redirect(): returns 1 when role is STANDBY and primary_host/port
are known — caller should send E_XCE_REDIRECT.
- ha_no_primary_known(): returns 1 when role is STANDBY but primary address
is not yet known — caller should send E_XCE_NOMASTER (retry later).
- Add E_XCE_NOMASTER and E_XCE_REDIRECT to errnox.h.
- Wire both guards into handle_join_() in join.c (before the role switch)
so all join types (worker, scheduler, apiserver, client) are protected.
- Full redirect response sends status=-E_XCE_REDIRECT with nodata body;
the primary host:port are logged so operators can trace misrouted clients.
Full Cap'n Proto RedirectInfo response body is deferred to a follow-up
(schema union would require regenerating capnp bindings).
5 new TDD tests — 110 total, 0 failures.
Add terminal-state constants and GC retention predicates for the
checkpoint xzarray_thaw() GC pass:
- types.h: add JOB_STAT_TRANSIENT (0x400), XCE_WORKER_HIDDEN (0x00200000),
QUEUE_STAT_HIDDEN (0x0100)
- persist.c: implement job_is_terminal(), job_should_retain(),
worker_should_retain() — placed here so the unit test binary
(which links only persist.c) can access them without RTTI deps
- persist.h: declare the three GC predicates; include xce/types.h
- worker.c: set dead_since = time(NULL) in mark_worker_unavail()
when dead_since is not already set
- cluster.c: add TODO P0-8b marker on zarray_del endpoint cleanup
- meson.build: register test_gc_retention.c; raise test timeout 60→120s
- testsuite/test_gc_retention.c: 19 tests covering job_is_terminal,
job_should_retain (TRANSIENT TTL, terminal TTL, non-terminal always
retained), and worker_should_retain (HIDDEN/UNAVAIL/healthy/zero
dead_since); all pass (63 total, 0 failures)
- cluster_def.h: add struct tag `cluster_s` to cluster_t to resolve
conflicting typedef with persist.h forward declaration
- cluster_def.h: add HOOK_DETAIL_JOB_STOP and HOOK_DETAIL_JOB_RESUME enum
values after HOOK_DETAIL_JOB_RUSAGE_UPDATE
- job.c on_job_stop(): replace mark_job_update() call with direct
xce_run_hooks(BEFORE/AFTER, HOOK_DETAIL_JOB_STOP) so WAL writer
records stop events with the correct detail value
- job.c on_job_resume(): same pattern with HOOK_DETAIL_JOB_RESUME
- job.c do_clr_preason(): fix typo — second hook call was HOOK_BEFORE_EVENT
instead of HOOK_AFTER_EVENT (double-before, zero-after bug)
- join.c handle_worker_join(): move xce_reply() to after HOOK_AFTER_EVENT
so client ACK is not sent before WAL write completes (durability fix)
- testsuite/test_hooks.c: new TDD tests for JOB_STOP/RESUME enum values
- testsuite/controller_test.c + meson.build: register test_hooks suite
Reply ordering audit results:
job.c handle_jobsubmit: reply before AFTER_EVENT — intentional (jobid
returned to client before scheduler notification; WAL hook belongs in
BEFORE_EVENT for this path, handled in Task 13)
job.c do_set_preason: reply before set_preason() loop — intentional
fire-and-forget (no AFTER_EVENT hook at top level)
job.c do_clr_preason: reply before loop — same fire-and-forget pattern
job.c do_jobcontrol_op: no AFTER_EVENT hook, reply only — OK
join.c handle_worker_join: FIXED (reply moved after AFTER_EVENT)
worker.c on_worker_update/dead: reply after all hooks — OK
cluster.c: AFTER_EVENT hooks are internal (no reply) — OK
Extends src/schema/controller.capnp with persistence structures needed
for WAL replication and snapshots:
- PersistConfig (8 fields, @0-@7): capnp-serialisable copy of the
persistence configuration; mapped to capnp_persist_cfg_t to avoid
collision with the plain-C persist_config_t in persist.h
- SnapshotHeader (9 fields, @0-@8): snapshot manifest with LSN, term,
timestamp, record counts and checksums; mapped to capnp_snap_hdr_t
- ReplEnvelope (6 fields, @0-@5): replication wire envelope (NOT the
on-disk format); named to avoid collision with wal_entry_hdr_t and
the WALEntry_ internal symbol c-capnproto generates; mapped to
capnp_repl_envelope_t
Also extends existing structs:
- Job.transientSince @11 (UInt64=0, mapname transient_since): set when
job enters JOB_STAT_TRANSIENT; also adds uint64_t transient_since to
xce_job_t in types.h
- Worker.deadSince @6 (UInt64=0, mapname dead_since): mirrors existing
time_t dead_since field in xce_worker_t
OPCode enum extended with 7 replication opcodes @22-@28:
replEntry, replAck, redirect, heartbeat, steppingDown,
replSnapshot, replCatchUpDone
New C typedefs (capnp_persist_cfg_t, capnp_snap_hdr_t,
capnp_repl_envelope_t) added to typemap.h for the encode/decode shims.
TDD: test_persist_schema.c added (6 tests, all GREEN). Controller test
suite remains 1/1 PASS.
