lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix: correct controller restart recovery — jobs, workers, and resource ordering

Browse source 
Four bugs fixed:

1. persist.c: array job WAL loop aborted after index [0] when replication
   returned -ENODEV (no standbys). Loop guard now uses the return value of
   persist_wal_sync_append() which is 0 on local-disk success regardless of
   replication status.

2. join.c: WAL-recovered worker slot (status=UNAVAIL) caused add_worker() to
   return -EEXIST on reconnect. New EEXIST path in handle_worker_join() looks
   up the existing slot by name, updates its endpoint, and clears UNAVAIL so
   the normal hook/reply/on_worker_join flow proceeds.

3. xcectx.c: cbuf_get() in xce_hello() blocked indefinitely when the TCP
   connection dropped because no error handler was registered. Added
   handle_disconnect() which sets ctx->state=XCE_CTX_ERR and calls
   cbuf_tryput(-ECONNRESET) to unblock the waiter, allowing
   checking_connection() to detect the failure and rejoin.

4. cluster.c: persist_recover() was called before cluster_build_resources()
   populated resmgr, causing restore_worker() → add_hostattrs_resvals() to
   fail with EEXIST noise. Moved persist_recover() to after HOOK_AFTER_EVENT
   so resmgr is fully populated before WAL replay.
This commit is contained in:
Rongsong Shen 2026-05-09 03:02:39 +00:00
parent b830cecf97
commit 6429ef0bff
4 changed files with 59 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

20
src/controller/cluster.c
View file

@ -609,7 +609,10 @@ int cluster_init(cluster_t* c)
apiserver_init(c);
/* Persistence: derive paths, init, and recover from WAL/snapshot */
/* Persistence: derive paths and init WAL/snapshot directories.
* Recovery is deferred until after cluster_build_resources() runs inside
* the HOOK_AFTER_EVENT below so that resmgr is fully populated before
* restore_worker() tries to register per-host resource values. */
{
const char* workdir = c->myconf->workdir ? c->myconf->workdir
: DEFAULT_WORKDIR;
@ -623,12 +626,6 @@ int cluster_init(cluster_t* c)
ERROR("persist_init failed: %d", rc);
return rc;
}
rc = persist_recover(&c->persist);
if (rc != 0) {
ERROR("persist_recover failed: %d", rc);
persist_fini(&c->persist);
return rc;
}
}
/* HA initialisation */
@ -684,6 +681,15 @@ int cluster_init(cluster_t* c)
xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_CLUSTER, HOOK_AFTER_EVENT,
HOOK_DETAIL_CLUSTER_INIT, c);
/* Recover WAL/snapshot now that resmgr, jobmgr, and workermgr are
* fully populated by cluster_build_resources() above. */
rc = persist_recover(&c->persist);
if (rc != 0) {
ERROR("persist_recover failed: %d", rc);
persist_fini(&c->persist);
return rc;
}
return 0;
}