The controller rejects a dispatch when the chosen worker is not in a
dispatchable state (UNAVAIL after a worker death, or still joining) and
replies with a jobDispatchFail notify. The scheduler had no handler for that
event, so a job it had just committed (state=RUN) was never returned to PEND
— it was never re-elected and stayed PEND cluster-wide even after the worker
recovered.
Handle NotifyData_data_jobDispatchFail: log the failure and call the new
sched_requeue_pend_job(), which frees the dispatch bookkeeping, resets the
job to PEND, and re-enqueues it in the fcfs pendJobs queue so the next
election session re-dispatches it. White-box test TUs that #include
handler.c get a sched_requeue_pend_job stub.
Co-Authored-By: Claude <noreply@anthropic.com>
bug-1453: remove unused client-side CA-certificate plumbing. C-side:
drop ca_cert_path from xce_cluster_config_t, do_connect()'s ca_cert
param, --ca-cert flags on worker/scheduler, and controller's cert
population (TLS is server-side only, SSL_VERIFY_NONE everywhere on
the client side already, verified zero SSL_CTX_load_verify_locations
calls in the C tree). Go-side: delete xcecli's self-signed cert
TOFU-pinning flow (cmd/auth/cert/*), CACert field, SetRootCAs/
loadRootCAs, and every call site.
bug-1454: fix xcecli_ctx_t leak in scheduler.c sched_init() — the
unconditional initial ctx_init was never freed before the HA
candidate loop re-initialized it on the first iteration.
bug-1455: sched_init()'s memset(sched, 0, ...) was silently wiping
sched->cluster_cfg that main.c had populated from --peers just before
calling sched_init(), so --peers was always ignored and the scheduler
fell back to the hardcoded localhost:9990 single-host default.
Restructured sched_init() to take cluster_cfg as an explicit parameter
(assigned after the memset) instead of relying on caller-side
ordering with a separate sched_set_cluster_cfg() setter, which is now
removed.
bug-1456: revert an unsafe stack-local xce_client_config_t introduced
while fixing bug-1455 — xcecli_ctx_init() stores that pointer as
ctx->conf and client_failover.c dereferences it for the ctx's entire
life (runtime reconnect), so a per-loop-iteration stack copy would
dangle the moment sched_init() returned. Back to mutating the
caller-owned, process-lifetime config in place.
Verified: full ASAN build (all components), meson test suite 9/9 OK,
manual scheduler testsuite 15/15, Go build/vet/test all pass.
5.5 (Req 6 / G-5): the JOB_RUSAGE notify path called the plugin but never
stored the usage data — xce_sched_job_t had no rusage field (UB-6.1 unmet).
- Add xce_job_rusages_t* rusages to xce_sched_job_t.
- New sched_apply_job_rusage() stores the report ON the job (ownership
transferred), replacing any prior via free_JobRusages_ptr; -ENOENT unknown,
-EINVAL NULL. Dequeue NULLs the request pointer on success; kill_sched_job
releases job->rusages.
5.8 (G-3): sched_pend_update_killer left five new-op payloads on empty
break cases — a per-message leak since ownership is transferred on enqueue.
- Free each with the exact generated deep-freer: free_QueueConf_ptr,
free_Resource_ptr[], free_ResourceVal_ptr[], free_ResValList_ptr,
free_JobRusages_ptr. Safe: handlers nullify on enqueue, makeResource copies
via makeInstance, dequeue copies values (no aliasing).
Tests: sched_job_update_test 7/7 (3 update + 4 rusage), regression
sched_test 15/15, ASAN clean (no bad-free/double-free).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The notify schedUpdateJob path shared XCE_SCHED_OP_UPDATE_JOB with the
dataset bulk-load, so the dequeue routed both to sched_update_jobs()
(create/replace) — a notify update to an existing job corrupted/duplicated
live job state and never returned -ENOENT (Req 3 UB-3.1/UB-3.2 unmet;
sched_update_existing_job was dead code copying only state).
- Add distinct op XCE_SCHED_OP_UPDATE_JOB_FIELDS (+ jobfieldsupdate union),
enqueued only by the notify handler, dequeued to sched_update_existing_job.
- Move sched_update_existing_job to schedobj.c; update all mutable fields
(state, priority, queue, resreq) in place; -ENOENT unknown, -EINVAL NULL.
- Dataset bulk-create path unchanged. Killer frees the new payload.
- New white-box test sched_job_update_test (3/3); regression sched_test 15/15.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire up 6 previously-stubbed message handlers in the scheduler:
resource dataset, resval dataset, job update notify, queue update
notify, resval update notify, and job rusage notify. Remove the
unreachable pushmsg branch (vtask_entry rewrites all pushmsg to
datamsg before the handler fires).
Changes by component:
- helper.h/helper.c: add xce_notify_event_name() for symbolic log names
- scheduler.h: add 5 new xce_sched_op_t values and matching union fields
- plugin.h/plugin.c: add plugin_job_rusage_hook_t and plugin_job_rusage_update()
- schedobj.c: add sched_update_existing_queue() with NULL/-ENOENT guards
- scheduler.c: wire 5 new dequeue cases; add sched_update_existing_job()
and static sched_op_name(); emit WARN log on handler failure
- handler.c: implement sched_handle_resources/resvals (dataset) and
sched_handle_sched_update_{job,queue,resvals}/job_rusage (notify);
remove dead sched_handle_jobpush/workerpush and pushmsg branch;
update TRACE logs to use symbolic name helpers
- test.c: 5 new unit tests covering notify event names, enum values,
and queue update NULL guard (15/15 pass)
Worker (xcecd):
- Add xce_cluster_config_t cluster_cfg field to controller_connector_ctx_t
(controller_connector.h includes xce/cluster.h)
- Populate cc_ctx->cluster_cfg from parsed cluster_cfg after xcecd_start()
sets cc_ctx (main.c, removes the P3 placeholder)
- Replace xcecd_join_cluster() single-shot connect with a candidate-iterating
loop: tries each candidate, calls xcecli_stop() before the next attempt,
falls back to the original single-host path when candidates_count == 0
Scheduler (xce_scheduler):
- Add xce_cluster_config_t cluster_cfg field to xce_scheduler_t
(scheduler.h includes xce/cluster.h)
- Add sched_set_cluster_cfg() setter
- Replace the hardcoded xcecli_start/xcecli_join pair in sched_init() with
the same candidate-iterating pattern; original single-host path preserved
- main.c calls sched_set_cluster_cfg() before sched_init() when --peers given
(removes the P3 placeholder comment)
checking_connection() already calls xcecd_join_cluster() on retry, giving
the worker free reconnect-after-failover behaviour for no additional code.
