--- id: "bug-1472-remainder" status: "fixed" status_updated: "2026-09-24" status_note: "" component: "controller" severity: "unknown" timestamp: "2026-07-25T20:15:00Z" last_seen: "2026-09-19T09:00:00Z" occurrences: 1 files: ["src/controller/worker.c worker_from_capnp (~827-895", "post-bug-1472-fix); mark_worker_unavail (~661-679); src/schema/controller.capnp struct Worker (~136-144)"] tags: ["worker.c", "worker_from_capnp", "mark_worker_unavail", "WAL_WORKER_CTRL", "HOOK_DETAIL_WORKER_DEAD", "capnp-schema-gap", "resource-value-replication", "standby-divergence", "post-promotion-scheduling-risk", "needs-owner", "needs-design-decision", "deferred"] related: ["bug-1472"] legacy: true --- # bug-1472-remainder: Follow-up split out of bug-1472: HOOK_DETAIL_WORKER_DEAD entries applied via WAL_WORKER_CTRL… ## Description Follow-up split out of bug-1472: HOOK_DETAIL_WORKER_DEAD entries applied via WAL_WORKER_CTRL against an already-existing worker still return -EEXIST (no-op) -- a dead-worker transition detected by one member (mark_worker_unavail(), worker.c:661-679) still never replicates its state to other members (standbys, or this member's own WAL replay), unconditionally. NOT a regression: this was already true before bug-1472's investigation; bug-1472's fix deliberately scoped itself to the byte-precise-confirmed HOOK_DETAIL_WORKER_UPDATE case only and left DEAD as -EEXIST/no-op, unchanged. ## Environment _Not recorded (migrated from buglog.json)._ ## Reproduction _Not recorded (migrated from buglog.json)._ ## Root cause mark_worker_unavail()'s mutation is NOT confined to XCE_WORKER_ADMIN_MASK the way worker_ctrl()'s is -- it sets XCE_WORKER_UNAVAIL (a non-admin bit), dead_since, resets hostattrs_t to 'unknown' sentinel strings, AND zeroes the live per-host resource-value slots via set_host_slots()/update_resvals(), which mutates xce_worker_t.resvals (a zarray_t of live xce_val_t entries). Critically, w->host.resvals is NOT part of the Worker capnp wire schema at all (confirmed by reading src/schema/controller.capnp's struct Worker and the generated decode_Worker/encode_Worker in controller.capnp.c) -- only the separate resvals_to_sched/nresvals_to_sched fields (a different, scheduler-push-facing concept) are encoded. So even applying a simple 'replace admin-masked bits' fix analogous to bug-1472's UPDATE fix would NOT fully replicate a DEAD transition's effects: the resource-value slot zeroing would still be silently lost on every other member, which matters materially at PROMOTION time (a newly-promoted primary would read stale, non-zeroed slots for a worker that died before promotion, potentially over-scheduling a dead host). A correct fix needs either (a) a schema change adding a wire-encodable resource-value-slots field so the DEAD payload can carry it, or (b) a different replication strategy for DEAD (e.g. re-deriving 'this worker's connection is gone locally' independently on each member rather than replicating a mutation), which is a design decision beyond a same-task patch. ## Fix FIXED (TDD RED->GREEN), design direction chosen by user: wire-encode the resource-value-slot zeroing rather than a non-replicated per-member re-derivation. Investigation revised the original design framing: NO new capnp schema field was actually needed. capnp_encode_worker (persist_worker_hook_fun) already serializes the FULL xce_worker_t.host -- including `attrs` (HostAttrs) -- on every WAL_WORKER_CTRL entry, and mark_worker_unavail() calls reset_hostattrs_unknown() BEFORE firing the hook that triggers that encode, so a DEAD entry's wire payload already carries the post-reset 'unknown'/0 attrs state. The actual gap was narrower: worker_from_capnp() had no DEAD branch at all (fell through to the JOIN-duplicate -EEXIST no-op), so nothing ever APPLIED what was already on the wire. Fixed by adding a DEAD branch (worker.c) mirroring the existing UPDATE branch (bug-1472's own fix): sets XCE_WORKER_UNAVAIL, takes dead_since from the decoded payload's deadSince field (the PRIMARY's own death timestamp -- genuinely wire-driven, not locally re-derived, honoring the chosen design direction), and calls reset_hostattrs_unknown()+add_hostattrs_resvals()+set_host_slots(0) locally to re-derive the resvals from the now-reset attrs -- these three calls are deterministic/data-independent (same effect mark_worker_unavail() itself produces on the primary), so xce_worker_t.resvals (the zarray_t the original investigation worried about) never needs to cross the wire at all. Regression test (test_apply_committed_entry.c's test_ace_worker_ctrl_dead_applies_to_existing_worker): pre-existing worker with live (non-'unknown') attrs receives a DEAD WAL entry with a specific non-zero deadSince -- RED under the pre-fix code (rc=-EEXIST, no mutation, no notification, the exact silent-discard this entry describes), GREEN under the fix (XCE_WORKER_UNAVAIL set, dead_since matches the payload exactly, hostattrs reset to 'unknown'/0, one notification fires). Full controller_unit_test suite: 1101 tests, 2 pre-existing unrelated failures (bug-1561), no regressions. [Original entry] NOT FIXED -- deliberately deferred, investigated not skipped (see bug-1472's own fix note for the full investigation). Needs its own design decision on how (or whether) to wire-encode the resource-value-slot zeroing, or an alternative non-replicated detection strategy, before a fix can be written safely. ## Verification _Not recorded (migrated from buglog.json)._ ## References - Related: bug-1472