---
id: "bug-1472"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-07-25T16:35:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 2
files: ["src/controller/worker.c worker_from_capnp (~827-856); src/controller/persist.c case WAL_WORKER_CTRL apply path (~3724-3786)"]
tags: ["worker.c", "persist.c", "worker_from_capnp", "WAL_WORKER_CTRL", "host-control-replication", "hostctrl", "admin-status", "standby-divergence", "pre-existing", "out-of-config-upgrade-snapshot-scope", "root-caused"]
related: ["bug-1471", "bug-1472-remainder"]
fix_commits: ["7547a8f1", "45400c3a"]
legacy: true
---

# bug-1472: Host admin-control status changes (hostctrl -c / hostctrl -o, i.e

## Description
Host admin-control status changes (hostctrl -c / hostctrl -o, i.e. HOST_CLOSE/HOST_OPEN via worker_ctrl()) never propagate to standby members' in-memory worker state once the target host has already been joined -- discovered while root-causing bug-1471's single-byte checkpoint divergence, but the defect itself is general and has nothing to do with checkpoints, config upgrades, or the config-upgrade-snapshot spec.

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
worker_from_capnp() (worker.c:827-856) is invoked UNCONDITIONALLY by persist.c:3731 for every WAL_WORKER_CTRL entry, regardless of its hook_detail discriminator (JOIN, UPDATE, or DEAD all share this one apply path -- see persist.c:3766-3783's switch, which only controls which post-apply NOTIFICATION hook fires, not whether the mutation itself is applied). worker_from_capnp's own idempotency guard (worker.c:847-853): if key_find() on c->workermgr finds a worker with the decoded name ALREADY PRESENT, it returns -EEXIST immediately, WITHOUT applying any decoded field (including host.status) to the existing worker object. persist.c:3732-3733 treats any -EEXIST return as 'idempotent re-apply (replay / resync) -- no hook' and silently returns 0. This EEXIST short-circuit is correct for its originally-intended purpose (a duplicate/replayed JOIN entry for a worker that's already been created should indeed be a no-op), but WAL_WORKER_CTRL is also the opcode for HOOK_DETAIL_WORKER_UPDATE (worker.c:732-794's worker_ctrl(), which handles HOST_OPEN/ HOST_CLOSE admin commands, sets w->host.status, then fires exactly this same HOOK_AFTER_EVENT/HOOK_DETAIL_WORKER_UPDATE/WAL_WORKER_CTRL path at worker.c:786-787) and HOOK_DETAIL_WORKER_DEAD. For UPDATE/DEAD events, the target worker EXISTING is the NORMAL, EXPECTED case (you can only update or kill an already-joined worker) -- yet the same 'already exists -> skip entirely' logic applies, so the decoded status change is silently discarded on every member OTHER than the one that directly processed the original RPC. Confirmed via git blame: worker_from_capnp was introduced 2026-05-08 (commit 9b6a91a8, 'feat: implement restore_job/worker_from_capnp for WAL recovery'), predating config-upgrade-snapshot's spec work by ~2.5 months; no config-upgrade-snapshot task touches this function or the WAL_WORKER_CTRL apply case. Empirically confirmed via byte-precise capnp decode of two independent live-soak checkpoint sets (/tmp/xce-ha-configupgrade{,-v2}): ctrl-1 (which processed a `hostctrl -c` against a previously-joined synthetic host) has host.status=0x00500000 (XCE_WORKER_CLOSED|XCE_WORKER_CLOSED_BY_ADMIN) in its checkpoint's HostAdminState frame; ctrl-2/ctrl-3 (standbys) both show host.status=0x00000000 for the identically-named host entry -- proving the close never reached their live c->workermgr entries. Because a standby's OWN restart+full-WAL-replay hits the identical code path (persist.c's replay call site also routes through apply_committed_entry's same WAL_WORKER_CTRL case), a member that fully replays from LSN 1 would ALSO reconstruct the JOIN first (creating the worker) and then skip the later UPDATE (already exists) -- so this is not even a restart-repairable staleness; the divergence appears to be permanent for that member until directly patched or until some other mechanism re-applies the admin action. General severity: this affects ANY admin host-control action (`hostctrl -c`/`hostctrl -o`) against an already-joined host, unconditionally, cluster-wide -- not just this soak's synthetic cfgupgrade-host scenario.

## Fix
FIXED 2026-07-25 (TDD: RED test first, confirmed failing against the pre-fix code, then GREEN). worker_from_capnp() (src/controller/worker.c) now decodes the WAL_WORKER_CTRL entry's hook_detail discriminator (the same 4 leading bytes it previously skipped without inspecting) BEFORE applying its existing-worker idempotency guard. When the target worker already exists: HOOK_DETAIL_WORKER_JOIN keeps the original -EEXIST no-op (a duplicate JOIN replay is genuinely idempotent -- unchanged, regression-guarded). HOOK_DETAIL_WORKER_UPDATE now REPLACES (not ORs -- worker_ctrl()'s HOST_OPEN uses assignment-style &= ~(...) to CLEAR admin bits, so OR would leave a re-opened worker stuck closed) the existing worker's XCE_WORKER_ADMIN_MASK bits with the decoded payload's admin-masked bits, under the worker's own write lock (mirrors restore_host_state's masking discipline, worker.c:487, with replace instead of OR semantics), then returns 0 so persist.c's WAL_WORKER_CTRL apply case (unchanged -- its existing '0 on any successful apply, -EEXIST only for genuine JOIN no-op' contract already handled this generically) falls through to fire the correct HOOK_DETAIL_WORKER_UPDATE notification instead of silently discarding the status change. HOOK_DETAIL_WORKER_DEAD is DELIBERATELY LEFT UNFIXED (still -EEXIST/no-op) -- investigated, not skipped: mark_worker_unavail()'s mutation (worker.c:661-679) is NOT confined to XCE_WORKER_ADMIN_MASK (sets XCE_WORKER_UNAVAIL + dead_since + resets hostattrs_t to 'unknown' + zeroes resource-value slots via set_host_slots), and critically, the live per-host resource-value array xce_worker_t.resvals (a zarray_t, what set_host_slots actually mutates and what scheduling reads) is NOT part of the Worker capnp wire schema at all -- only the separate resvals_to_sched/nresvals_to_sched fields are (confirmed by reading the generated decode_Worker/encode_Worker in controller.capnp.c and the schema in controller.capnp). A naive status-replace for DEAD would be an incomplete fix (would not zero standbys'/replayers' resource slots on a dead-worker event, a real correctness gap for POST-PROMOTION scheduling since the newly-promoted primary would read stale, non-zeroed slots) and needs its own investigation into whether/how to extend the wire schema -- filed separately as bug-1472-remainder below rather than rushed. Fix verified via 2 new + regression tests in test_apply_committed_entry.c (test_ace_worker_ctrl_update_applies_to_existing_worker, test_ace_worker_ctrl_update_replaces_not_ors_admin_bits, test_ace_worker_ctrl_join_duplicate_existing_is_noop -- the last one a regression guard proving JOIN's existing no-op behavior is unchanged) plus the full controller_unit_test suite: baseline (this worktree, immediately pre-fix) 1030 Tests 0 Failures, post-fix 1033 Tests 0 Failures (net +3, exactly the new tests, zero regressions). persist.c was NOT touched -- confirmed its WAL_WORKER_CTRL apply case (persist.c:3724-3786) already treats any non-negative worker_from_capnp() return as a successful apply generically (re-looks-up the worker by name and fires the hook_detail-keyed notification unconditionally), so returning 0 from the new UPDATE branch required no caller-side change.

[UPDATE 2026-08-16] Tag hygiene: this entry's own UPDATE-case fix landed 2026-07-25 (commit 7547a8f1) but its tags were never updated off 'not-fixed'/'needs-owner'. The DEAD-case gap this entry deferred is bug-1472-remainder, separately FIXED and landed today in commit 45400c3a. Both halves of bug-1472 are now fixed and committed; closing out.

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-1471, bug-1472-remainder