lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(worker): replicate admin host-control status updates to standbys (bug-1472)

Browse source 
worker_from_capnp()'s idempotency guard returned -EEXIST unconditionally
whenever the target worker already existed, discarding the decoded WAL
payload entirely. Correct for a duplicate WAL_WORKER_CTRL/JOIN replay,
but wrong for HOOK_DETAIL_WORKER_UPDATE (hostctrl -c/-o admin close/open):
the target existing is the normal case there, yet the status change was
silently dropped on every member other than the one that processed the
RPC directly -- a permanent, unconditional, cluster-wide replication gap
for host admin-status, confirmed via byte-precise checkpoint divergence
on a live 3-node soak.

Now decodes the entry's hook_detail before applying the existing-worker
guard: JOIN keeps its no-op short-circuit; UPDATE replaces (not ORs,
since HOST_OPEN must clear closed bits) the existing worker's
XCE_WORKER_ADMIN_MASK bits with the decoded payload's, in place, under
the worker's own write lock. HOOK_DETAIL_WORKER_DEAD is deliberately
left unfixed -- its replication would need fields (resvals) that never
round-trip through the Worker capnp wire schema at all, and this
codebase already treats liveness state as locally-derived rather than
WAL-replicated elsewhere -- filed separately as bug-1472-remainder.

Independently reviewed: RED->GREEN cycle re-reproduced from scratch,
replace-vs-OR semantics and lock discipline verified directly, DEAD
deferral corroborated against an independent design-invariant comment
this fix didn't originally cite. Suite: 1030 -> 1033 tests, 0 failures.
This commit is contained in:
Rongsong Shen 2026-07-26 04:06:42 +00:00
parent 8b3e2ab03c
commit 7547a8f14b
2 changed files with 241 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

45
src/controller/worker.c
View file

@ -823,12 +823,37 @@ int do_hostcontrol_op(cluster_t* c, endpoint_t* ep, xce_request_t* request)
/* Reconstruct a worker from a WAL_WORKER_CTRL Cap'n Proto payload.
* Payload layout: [uint32_t detail][Cap'n Proto xce_worker_t].
* Returns -EEXIST if the worker is already registered (idempotent replay). */
*
* If the worker is already registered:
* - HOOK_DETAIL_WORKER_JOIN: a duplicate JOIN replay is genuinely a
* no-op -- returns -EEXIST, no mutation (unchanged, pre-bug-1472
* behavior).
* - HOOK_DETAIL_WORKER_UPDATE: the target already existing is the
* NORMAL case (worker_ctrl()'s HOST_OPEN/HOST_CLOSE, worker.c:732-794,
* only ever runs against an already-joined host). bug-1472: replace
* (not OR -- worker_ctrl uses assignment-style clear/set, so a
* HOST_OPEN must be able to CLEAR the closed bits) the existing
* worker's admin-masked status bits with the decoded payload's, then
* return 0 so the caller (persist.c's WAL_WORKER_CTRL apply case)
* falls through to fire the HOOK_DETAIL_WORKER_UPDATE notification
* instead of silently discarding the change.
* - HOOK_DETAIL_WORKER_DEAD: left as -EEXIST (unchanged) -- see
* buglog.json bug-1472 for why DEAD's replication needs its own,
* separately-tracked fix: its mutation isn't confined to
* XCE_WORKER_ADMIN_MASK (XCE_WORKER_UNAVAIL + dead_since + hostattrs
* reset), and the live per-host resource-value array
* (xce_worker_t.resvals, a zarray_t) that mark_worker_unavail()
* zeroes via set_host_slots() never round-trips through the Worker
* capnp schema at all (only resvals_to_sched does) -- a naive
* status-replace here would be an incomplete fix. */
int worker_from_capnp(cluster_t* c, const void* capnp_payload, size_t len)
{
if (!capnp_payload || len <= sizeof(uint32_t))
return -EINVAL;
uint32_t hook_detail;
memcpy(&hook_detail, capnp_payload, sizeof(hook_detail));
/* Skip the 4-byte detail discriminator written by persist_worker_hook_fun */
const uint8_t* data = (const uint8_t*)capnp_payload + sizeof(uint32_t);
size_t data_len = len - sizeof(uint32_t);
@ -844,13 +869,27 @@ int worker_from_capnp(cluster_t* c, const void* capnp_payload, size_t len)
decode_Worker(&host, &cw);
capn_free(&c_ctx);
/* Idempotency guard: skip if already present */
/* Idempotency guard: an existing worker means either a duplicate JOIN
* replay (no-op) or an UPDATE targeting an already-joined host (the
* normal case -- must be applied, see bug-1472). */
worker_t* existing = NULL;
lock(c->workermgr, RTTI_LOCK_READ);
int found = key_find(c->workermgr, (void*)host.name, (void**)&existing);
unlock(c->workermgr);
if (found > 0 && existing != NULL)
if (found > 0 && existing != NULL) {
if (hook_detail == HOOK_DETAIL_WORKER_UPDATE) {
lock(existing, RTTI_LOCK_WRITE);
existing->host.status
= (existing->host.status & ~XCE_WORKER_ADMIN_MASK)
| (host.status & XCE_WORKER_ADMIN_MASK);
unlock(existing);
free_Worker(&host);
return 0;
}
return -EEXIST;
}
return restore_worker(c, host);
}