add kiro skills

2026-05-21 03:41:57 +00:00 · 2026-05-21 03:41:57 +00:00 · 86fe35e671
commit 86fe35e671
parent e5fde7b55f
53 changed files with 8406 additions and 166 deletions
--- a/.kiro/specs/worker-join-refactor/design.md
+++ b/.kiro/specs/worker-join-refactor/design.md
@ -0,0 +1,450 @@
+# Design Document: worker-join-refactor
+
+## Overview
+
+This refactor restructures the worker join path in the XCE controller to make the check → create-or-update → status/resource-update sequence explicit and readable at the `handle_worker_join` call site.
+
+The current implementation delegates the create-or-update decision to `add_worker`, which in turn delegates it to `makeWorker` in `wobj.c`. This hides the branching and introduces a latent deadlock: `handle_worker_join` holds the `workermgr` write lock while calling `add_worker → makeWorker`, which attempts to re-acquire the same lock. Since `my_rwlock_t` is `pthread_rwlock_t` (non-reentrant), this deadlocks. The refactor eliminates the outer lock, makes existence-checking visible at the join handler, and introduces two narrowly-scoped helpers for the create and rejoin paths.
+
+### Goals
+
+- Make the check → create-or-update → status/resource-update control flow visible in `handle_worker_join`
+- Eliminate the double workermgr write lock acquisition (latent deadlock fix)
+- Preserve all observable behavior: hooks, `on_worker_join`, admin-bit preservation
+
+### Non-Goals
+
+- Changes to `makeWorker` or any other function in `src/lib/wobj.c`
+- Changes to other join handlers (`handle_client_join`, `handle_scheduler_join`, etc.)
+- Job dispatch, WAL, or persist subsystem behavior changes
+
+---
+
+## Boundary Commitments
+
+### This Spec Owns
+
+- `handle_worker_join` in `src/controller/join.c` — rewritten to expose explicit branching
+- `add_worker` in `src/controller/worker.c` — replaced by `init_worker`, `rejoin_worker`, and `apply_join_state`
+- `src/controller/include/worker.h` — updated exported declarations
+
+### Out of Boundary
+
+- `makeWorker` / `makeWorker_` / `override_worker` in `src/lib/wobj.c` — must not be modified
+- Other join handler functions in `join.c`
+- Persist/WAL hook internals
+
+### Allowed Dependencies
+
+- `key_find` from `kvmgr` (via RTTI) for existence check
+- `makeWorker` from `wobj.c` for first-time worker creation
+- `find_matched_hostconfig`, `add_hostattrs_resvals`, `set_host_slots`, `mark_host_res` from `worker.c` (already present)
+- `xce_run_hooks` for BEFORE/AFTER event hooks
+- `on_worker_join` for post-join config push
+
+### Revalidation Triggers
+
+- Changes to `makeWorker` behavior (create vs update path) require re-checking `init_worker`
+- Changes to `XCE_WORKER_ADMIN_MASK` definition require re-checking the admin-bit preservation in `rejoin_worker`
+- Adding new hconfig fields requires updating `apply_join_state`
+
+---
+
+## Architecture
+
+### Existing Architecture Analysis
+
+**Current flow in `handle_worker_join` (join.c)**:
+
+```
+lock(workermgr, WRITE)
+    run_hooks(BEFORE_EVENT)
+    add_worker()                     ← delegates create-or-update decision
+        makeWorker()
+            lock(workermgr, WRITE)   ← DEADLOCK: already held by caller
+            key_find + create-or-override
+            unlock(workermgr)
+        per-worker lock
+            clear UNAVAIL, set ep, apply hconfig
+        per-worker unlock
+    vget(workermgr, idx, &w)
+    run_hooks(AFTER_EVENT)
+    xce_reply(...)
+unlock(workermgr)
+on_worker_join(...)
+```
+
+**Problems**:
+1. Double write lock on `workermgr` deadlocks (pthread_rwlock_t is non-reentrant)
+2. Create-or-update branch hidden inside `makeWorker_` — invisible at call site
+3. `xce_reply` called while holding the write lock (unnecessary)
+
+### Architecture Pattern & Boundary Map
+
+```mermaid
+graph TB
+    HWJ[handle_worker_join]
+    FHC[find_matched_hostconfig]
+    HooksBefore[run_hooks BEFORE]
+    KeyFind[key_find workermgr READ lock]
+    InitW[init_worker]
+    RejoinW[rejoin_worker]
+    AJS[apply_join_state]
+    MakeW[makeWorker wobj.c]
+    HooksAfter[run_hooks AFTER]
+    OWJ[on_worker_join]
+
+    HWJ --> FHC
+    HWJ --> HooksBefore
+    HWJ --> KeyFind
+    KeyFind -->|ENOENT| InitW
+    KeyFind -->|found| RejoinW
+    InitW --> MakeW
+    InitW --> AJS
+    RejoinW --> AJS
+    HWJ --> HooksAfter
+    HWJ --> OWJ
+```
+
+**Refactored flow**:
+- `handle_worker_join` owns the explicit `key_find` branch decision (Req 1)
+- `init_worker` owns first-time registration (Req 2); delegates creation locking to `makeWorker`
+- `rejoin_worker` owns in-place attribute/endpoint update (Req 3); uses per-worker write lock
+- `apply_join_state` is a static shared helper for status and resource update (Req 4)
+- No outer `workermgr` write lock in `handle_worker_join` — eliminates deadlock
+
+**Locking model**:
+
+| Scope | Lock | Holder | Purpose |
+|-------|------|--------|---------|
+| Existence check | workermgr READ | `handle_worker_join` (brief) | Safe key_find |
+| Worker creation | workermgr WRITE | `makeWorker` (internal) | key_add atomicity |
+| Worker update | per-worker WRITE | `rejoin_worker` / `apply_join_state` | Field mutation |
+
+### Technology Stack
+
+| Layer | Choice | Role | Notes |
+|-------|--------|------|-------|
+| Language | C99 | Implementation | Existing codebase standard |
+| Locking | pthread_rwlock_t (my_rwlock_t) | Concurrency | Existing primitive; non-reentrant |
+| Object management | RTTI (key_find, key_add, makeInstance) | Worker lifecycle | wobj.c / kvmgr.c |
+| Hooks | xce_run_hooks | Persistence integration | Unchanged hook fire points |
+
+---
+
+## File Structure Plan
+
+### Modified Files
+
+- `src/controller/join.c` — `handle_worker_join`: remove outer workermgr write lock; add explicit `key_find` check; call `init_worker` or `rejoin_worker` based on result
+- `src/controller/worker.c` — remove `add_worker`; add `apply_join_state` (static), `init_worker`, `rejoin_worker`
+- `src/controller/include/worker.h` — replace `add_worker` declaration with `init_worker` and `rejoin_worker`
+
+---
+
+## System Flows
+
+```mermaid
+sequenceDiagram
+    participant JC as handle_worker_join
+    participant WC as worker.c helpers
+    participant WO as makeWorker (wobj.c)
+    participant HS as hooksys
+
+    JC->>JC: find_matched_hostconfig (fail fast, no lock)
+    JC->>HS: run_hooks BEFORE_EVENT WORKER_JOIN
+    JC->>JC: lock(workermgr READ)
+    JC->>JC: key_find(workermgr, name, &w)
+    JC->>JC: unlock(workermgr READ)
+
+    alt host not found (ENOENT)
+        JC->>WC: init_worker(c, ep, name, attrs, hconfig)
+        WC->>WO: makeWorker(workermgr, &tmp_host)
+        Note over WO: lock WRITE, create entry, unlock WRITE
+        WC->>WC: apply_join_state under per-worker WRITE lock
+        WC-->>JC: idx
+    else host found
+        JC->>WC: rejoin_worker(c, w, ep, attrs, hconfig)
+        WC->>WC: per-worker WRITE lock
+        WC->>WC: update hostattrs + ep (preserve admin bits)
+        WC->>WC: apply_join_state (status, resources)
+        WC->>WC: unlock per-worker
+        WC-->>JC: idx
+    else key_find error (not ENOENT)
+        JC-->>JC: reject join, return error
+    end
+
+    JC->>JC: set priv->u.idx = idx
+    JC->>JC: xce_reply (outside any lock)
+    JC->>HS: run_hooks AFTER_EVENT WORKER_JOIN
+    JC->>WC: on_worker_join(c, ep)
+```
+
+**Key decisions**:
+- `find_matched_hostconfig` fails fast before any lock (Req 4.4 — reject if NULL)
+- Hooks fire outside the workermgr lock, bracketing the create-or-update step
+- `xce_reply` and `on_worker_join` execute after all locks are released
+
+---
+
+## Requirements Traceability
+
+| Requirement | Summary | Component | Interface | Flow |
+|-------------|---------|-----------|-----------|------|
+| 1.1 | key_find before create-or-update | `handle_worker_join` | — | Sequence: key_find step |
+| 1.2 | Branch on key_find result | `handle_worker_join` | — | alt/else branches |
+| 1.3 | Reject on non-ENOENT error | `handle_worker_join` | — | else-error branch |
+| 2.1 | Create via makeWorker | `init_worker` | `init_worker` signature | create branch |
+| 2.2 | Log failure, set reply status, no on_worker_join | `handle_worker_join` | — | error exit path |
+| 2.3 | Assign priv->u.idx | `handle_worker_join` | — | after successful join |
+| 3.1 | Update hostattrs on rejoin | `rejoin_worker` | `rejoin_worker` signature | update branch |
+| 3.2 | Update ep on rejoin | `rejoin_worker` | — | update branch |
+| 3.3 | Preserve admin bits | `rejoin_worker` | — | ADMIN_MASK pattern |
+| 4.1 | Clear XCE_WORKER_UNAVAIL | `apply_join_state` | — | shared path |
+| 4.2 | Apply hconfig overrides | `apply_join_state` | — | shared path |
+| 4.3 | Update host-attr resource values | `apply_join_state` | — | shared path |
+| 4.4 | Reject if find_matched_hostconfig NULL | `handle_worker_join` | — | fail-fast before lock |
+| 5.1 | BEFORE/AFTER hook order preserved | `handle_worker_join` | — | hook call sites |
+| 5.2 | on_worker_join still called | `handle_worker_join` | — | post-join step |
+| 5.3 | Critical section as short as current | Locking model | — | per-lock scoping |
+| 5.4 | Existing tests pass | All | — | behavioral preservation |
+
+---
+
+## Components and Interfaces
+
+### Summary
+
+| Component | Layer | Intent | Req Coverage | Key Dependencies |
+|-----------|-------|--------|--------------|-----------------|
+| `handle_worker_join` | join.c | Explicit branch controller | 1.1–1.3, 2.2–2.3, 4.4, 5.1–5.3 | `key_find` (P0), `init_worker` (P0), `rejoin_worker` (P0) |
+| `init_worker` | worker.c | First-time registration | 2.1–2.3 | `makeWorker` (P0), `apply_join_state` (P0) |
+| `rejoin_worker` | worker.c | Reconnect update | 3.1–3.3 | `apply_join_state` (P0), per-worker lock (P0) |
+| `apply_join_state` | worker.c (static) | Status + resource update | 4.1–4.3 | `add_hostattrs_resvals` (P0), `set_host_slots` (P0) |
+
+---
+
+### join.c
+
+#### `handle_worker_join` (rewritten)
+
+| Field | Detail |
+|-------|--------|
+| Intent | Explicit check → create-or-update → status/resource sequence for worker join |
+| Requirements | 1.1, 1.2, 1.3, 2.2, 2.3, 4.4, 5.1, 5.2, 5.3 |
+
+**Responsibilities & Constraints**
+- Calls `find_matched_hostconfig` first; rejects join with appropriate error code if NULL (Req 4.4)
+- Fires BEFORE_EVENT hook before the key_find step
+- Acquires a brief workermgr READ lock for `key_find`; releases before any write path
+- Branches explicitly on key_find result: `init_worker` or `rejoin_worker`
+- Fires AFTER_EVENT hook after the create/update completes, outside any lock
+- Calls `xce_reply` and `on_worker_join` outside any lock
+- Does NOT hold the workermgr write lock at any point (prevents double-lock deadlock)
+
+**Dependencies**
+- Inbound: `handle_join_` dispatches to this function (P0)
+- Outbound: `find_matched_hostconfig` — hostconfig lookup (P0)
+- Outbound: `xce_run_hooks` — BEFORE/AFTER_EVENT (P0)
+- Outbound: `key_find(c->workermgr, ...)` — existence check (P0)
+- Outbound: `init_worker` — first-time join path (P0)
+- Outbound: `rejoin_worker` — rejoin path (P0)
+- Outbound: `on_worker_join` — config push post-join (P0)
+
+**Contracts**: Service [x] / State [x]
+
+##### Service Interface (C)
+
+```c
+/* Unchanged external signature */
+static int handle_worker_join(cluster_t* c, endpoint_t* ep,
+                              xce_request_t* request);
+```
+
+- Preconditions: `request->body.joinmsg` non-NULL; `ep->data` non-NULL
+- Postconditions: on success, `priv->u.idx` set; `on_worker_join` invoked; BEFORE/AFTER hooks fired
+- Error return: negative errno; reply status set to error code; `on_worker_join` not called
+
+**Implementation Notes**
+- Integration: The outer `lock(c->workermgr, RTTI_LOCK_WRITE)` present in the current code is **removed**. The `vget` lookup for the AFTER_EVENT hook (currently inside the lock) moves to after the create/update call.
+- Validation: All early-exit paths (hostconfig NULL, key_find non-ENOENT error, init/rejoin failure) must set `reply.status` and call `xce_reply` before returning.
+- Risks: `priv->u.idx` must be set before `on_worker_join` and after the lock is released; the existing order is preserved.
+
+---
+
+### worker.c
+
+#### `apply_join_state` (static)
+
+| Field | Detail |
+|-------|--------|
+| Intent | Shared status and resource update applied to both new and rejoining workers |
+| Requirements | 4.1, 4.2, 4.3 |
+
+**Responsibilities & Constraints**
+- Called under the per-worker write lock (caller must hold it)
+- Clears `XCE_WORKER_UNAVAIL` from `w->host.status` (Req 4.1)
+- Sets `w->ep = ep`
+- Applies hconfig hosttype/model overrides (dereference + makeString pattern)
+- Calls `add_hostattrs_resvals` (Req 4.3)
+- Computes mxj slots and calls `set_host_slots` (Req 4.2)
+- Replaces `dispatch_window_config` and `thresholds` from hconfig
+- Loops over `hconfig->nresources` calling `mark_host_res` (bool resources)
+
+**Contracts**: Service [x]
+
+##### Service Interface (C)
+
+```c
+/* static — internal to worker.c */
+static int apply_join_state(cluster_t* c, worker_t* w, endpoint_t* ep,
+                            xce_hostconfig_t* hconfig);
+```
+
+- Preconditions: caller holds per-worker WRITE lock; `w`, `hconfig` non-NULL
+- Postconditions: worker status, ep, and all resource values updated
+- Returns: 0 on success; negative errno on resource update failure
+
+---
+
+#### `init_worker`
+
+| Field | Detail |
+|-------|--------|
+| Intent | Register a new worker entry for a first-time joining host |
+| Requirements | 2.1, 2.2, 2.3 |
+
+**Responsibilities & Constraints**
+- Builds a temporary `xce_worker_t` with name and attrs
+- Calls `makeWorker(c->workermgr, &tmp)` — delegating write-lock acquisition and allocation to wobj.c
+- On NULL return: logs failure, returns `-errno` (caller sets reply status)
+- On success: acquires per-worker write lock, calls `apply_join_state`, unlocks
+- Returns the worker index (`w->idx`)
+
+**Dependencies**
+- Outbound: `makeWorker` (wobj.c) — allocation + key_add (P0)
+- Outbound: `apply_join_state` — status/resource setup (P0)
+
+**Contracts**: Service [x]
+
+##### Service Interface (C)
+
+```c
+/* exported — declared in worker.h */
+int init_worker(cluster_t* c, endpoint_t* ep, const char* host,
+                hostattrs_t* attrs, xce_hostconfig_t* hconfig);
+```
+
+- Preconditions: caller does NOT hold workermgr write lock; `host` non-NULL
+- Postconditions: new worker registered in workermgr; status and resources initialized
+- Returns: worker index (≥ 0) on success; negative errno on failure
+- Error cases: `makeWorker` allocation failure → return `-errno` without calling `on_worker_join`
+
+---
+
+#### `rejoin_worker`
+
+| Field | Detail |
+|-------|--------|
+| Intent | Update an existing worker entry when the same host reconnects |
+| Requirements | 3.1, 3.2, 3.3 |
+
+**Responsibilities & Constraints**
+- Accepts a `worker_t*` found by the caller's `key_find`
+- Acquires per-worker write lock
+- Saves admin status bits: `admin_status = w->host.status & XCE_WORKER_ADMIN_MASK`
+- Frees existing `hosttype` and `hostmodel` strings; copies new ones from `attrs`
+- Copies numeric attrs fields (`cpufactor`, `maxcpus`, `maxmem`, `maxswap`, `maxtmp`, `ndisks`)
+- Restores `w->host.status` admin bits (Req 3.3)
+- Calls `apply_join_state` (clears UNAVAIL, sets ep, applies hconfig overrides, updates resources)
+- Releases per-worker write lock
+- Returns `w->idx`
+
+**Dependencies**
+- Outbound: `apply_join_state` — status/resource setup (P0)
+- Outbound: `dereference` / `makeString` — string memory management (P0)
+
+**Contracts**: Service [x]
+
+##### Service Interface (C)
+
+```c
+/* exported — declared in worker.h */
+int rejoin_worker(cluster_t* c, worker_t* w, endpoint_t* ep,
+                  hostattrs_t* attrs, xce_hostconfig_t* hconfig);
+```
+
+- Preconditions: caller holds a reference on `w`; does NOT hold per-worker write lock
+- Postconditions: `w->host.attrs` updated; `w->ep` set to `ep`; admin bits preserved; resources updated
+- Returns: `w->idx` (≥ 0); negative errno on `apply_join_state` failure
+- Invariant: `w->host.status & XCE_WORKER_ADMIN_MASK` unchanged from pre-call value
+
+```c
+#define XCE_WORKER_ADMIN_MASK \
+    (XCE_WORKER_LOCKEDU | XCE_WORKER_CLOSED | XCE_WORKER_HIDDEN \
+     | XCE_WORKER_CLOSED_BY_ADMIN)
+```
+
+---
+
+### worker.h
+
+#### Exported Declaration Changes
+
+Remove:
+```c
+extern int add_worker(void*, endpoint_t*, const char*, hostattrs_t*);
+```
+
+Add:
+```c
+extern int init_worker(cluster_t*, endpoint_t*, const char*, hostattrs_t*,
+                       xce_hostconfig_t*);
+extern int rejoin_worker(cluster_t*, worker_t*, endpoint_t*, hostattrs_t*,
+                         xce_hostconfig_t*);
+```
+
+---
+
+## Error Handling
+
+### Error Strategy
+
+Fail fast, explicit error propagation, no silent fallbacks.
+
+### Error Categories
+
+| Error | Source | Handler | Reply Status |
+|-------|--------|---------|-------------|
+| `find_matched_hostconfig` returns NULL | No matching hostconfig | Reject join; set `reply.status = -errno` | `-errno` |
+| `key_find` returns error other than -ENOENT | workermgr internal error | Reject join (Req 1.3) | key_find return code |
+| `init_worker` returns error | `makeWorker` allocation failure | Log, reject join (Req 2.2) | error code |
+| `rejoin_worker` returns error | `apply_join_state` failure | Log, reject join | error code |
+| `apply_join_state` partial failure | Resource update failure | Log with PERROR; continue (matches existing behavior) | 0 |
+
+### Monitoring
+
+- Existing `ERROR` / `TRACE` macros used throughout — no changes to log verbosity
+- `PERROR("update_resval")` in `update_resvals` preserved for resource update failures
+
+---
+
+## Testing Strategy
+
+### Unit Tests (src/controller/testsuite/)
+
+1. **First-time join creates worker**: call `init_worker` with a new hostname; verify `key_find` succeeds afterwards, `XCE_WORKER_UNAVAIL` is clear, and resource values are set.
+2. **Rejoin updates attrs and ep**: call `init_worker`, then call `rejoin_worker` with different attrs; verify updated attrs, updated ep, and original admin bits preserved.
+3. **Admin bits preserved on rejoin**: set `XCE_WORKER_CLOSED` before rejoin; verify flag still set after `rejoin_worker`.
+4. **NULL hostconfig rejects join**: configure cluster with no matching host config; verify `handle_worker_join` path returns error without modifying workermgr.
+5. **Non-ENOENT key_find error rejected**: simulate `key_find` returning `-EIO`; verify join rejected and `on_worker_join` not called.
+
+### Integration Tests
+
+1. **BEFORE/AFTER hook order**: verify hooks fire in the correct order relative to workermgr state changes using existing `test_e2e_hooks.c` patterns.
+2. **on_worker_join called after successful join**: verify `worker_push_config` executes on both first-time and rejoin paths.
+3. **Existing testsuite passes without modification**: `make test` in `src/frontend/` and testsuite build must remain green.
+
+### Behavioral Preservation
+
+The existing unit tests in `src/controller/testsuite/` exercise `add_worker` indirectly through cluster setup helpers. Test fixtures that call `add_worker` directly (if any) must be updated to call `init_worker` with the added `hconfig` parameter. All observable behaviors — hook ordering, resource initialization, admin-bit preservation, `on_worker_join` invocation — are explicitly covered by the traceability table and unit tests above.
--- a/.kiro/specs/worker-join-refactor/requirements.md
+++ b/.kiro/specs/worker-join-refactor/requirements.md
@ -0,0 +1,71 @@
+# Requirements Document
+
+## Project Description (Input)
+Simplify the code of worker join in the controller by first checking whether the host is in `workermgr`; if it does not exist, create the host first; otherwise find the host in `workermgr` and update the host attrs and endpoint. Then update host status and resource information.
+
+## Introduction
+
+The controller's `handle_worker_join` function currently delegates the entire create-or-update decision to `add_worker`, which in turn delegates to `makeWorker` in `wobj.c`. The internal branching is invisible at the call site and the locking structure is confusing (the outer join handler holds the `workermgr` write lock while `makeWorker` re-acquires the same lock). This refactor restructures the worker join path so the check → create-or-update → status/resource-update sequence is explicit and readable.
+
+## Boundary Context
+
+- **In scope**: `handle_worker_join` in `src/controller/join.c` and `add_worker` in `src/controller/worker.c`.
+- **Out of scope**: `makeWorker` in `src/lib/wobj.c`, other join handlers (`handle_client_join`, `handle_scheduler_join`, etc.), job dispatch, and the persist/WAL subsystem behavior.
+- **Adjacent expectations**: Persist hooks (`XCE_CONTROLLER_HOOK_WORKER`) must still fire at the same event points (BEFORE_EVENT / AFTER_EVENT for HOOK_DETAIL_WORKER_JOIN). The `on_worker_join` callback that pushes configuration to the worker must still be called after a successful join.
+
+## Requirements
+
+### Requirement 1: Explicit Host Existence Check
+
+**Objective:** As a controller maintainer, I want the worker join code to make an explicit lookup before deciding whether to create or update, so that the control flow is readable without tracing into `makeWorker`.
+
+#### Acceptance Criteria
+1. When a worker join request is received, the Controller shall look up the joining host by name in `workermgr` using `key_find` before taking any create-or-update action.
+2. The Controller shall branch on the result of that lookup: one branch for a host that does not yet exist, one branch for a host that already exists.
+3. If `key_find` returns an error other than `-ENOENT`, the Controller shall reject the join and return that error code.
+
+---
+
+### Requirement 2: Host Creation for First-Time Workers
+
+**Objective:** As a controller maintainer, I want a new host entry to be created when a worker joins for the first time, so that the cluster knows about this node.
+
+#### Acceptance Criteria
+1. When the host is not found in `workermgr`, the Controller shall allocate and register a new worker entry (via the existing `makeWorker` or equivalent creation path) using the joining worker's name and host attributes.
+2. If creation fails, the Controller shall log the failure, set the reply status to the error code, and return without calling `on_worker_join`.
+3. The Controller shall assign the new entry's index to `priv->u.idx` so subsequent disconnect handling can find the worker.
+
+---
+
+### Requirement 3: Attribute and Endpoint Update for Rejoining Workers
+
+**Objective:** As a controller maintainer, I want an existing worker entry to be updated in-place when the same host rejoins, so that stale attributes and the old disconnected endpoint are replaced.
+
+#### Acceptance Criteria
+1. When the host already exists in `workermgr`, the Controller shall update the existing worker's host attributes (`hostattrs`) with the values supplied in the join message.
+2. When the host already exists in `workermgr`, the Controller shall update the existing worker's endpoint (`ep`) to the new connection's endpoint.
+3. The Controller shall preserve any admin-set status bits (`XCE_WORKER_LOCKEDU`, `XCE_WORKER_CLOSED`, `XCE_WORKER_HIDDEN`, `XCE_WORKER_CLOSED_BY_ADMIN`) when updating an existing entry.
+
+---
+
+### Requirement 4: Status and Resource Information Update
+
+**Objective:** As a controller maintainer, I want status and resource data to be updated after every successful join (both new and rejoining), so that the scheduler sees accurate host availability and capacity.
+
+#### Acceptance Criteria
+1. After a successful create or update, the Controller shall clear the `XCE_WORKER_UNAVAIL` flag on the worker's status.
+2. After a successful create or update, the Controller shall apply host-config overrides (hosttype, hostmodel, mxj slots, dispatch window, thresholds, and boolean resources) from the matched `xce_hostconfig_t`.
+3. After a successful create or update, the Controller shall update all host-attribute-mapped resource values (`type`, `model`, `cpuf`, `maxcpus`, `maxmem`, `maxswap`, `maxtmp`, `ndisks`, `slots`).
+4. If `find_matched_hostconfig` returns `NULL`, the Controller shall reject the join with an appropriate error code and not modify `workermgr`.
+
+---
+
+### Requirement 5: Behavioral Preservation
+
+**Objective:** As a developer, I want the refactored code to preserve all existing observable behavior so that no regressions are introduced.
+
+#### Acceptance Criteria
+1. The Controller shall fire `XCE_CONTROLLER_HOOK_WORKER / HOOK_BEFORE_EVENT / HOOK_DETAIL_WORKER_JOIN` before modifying the worker entry, and `HOOK_AFTER_EVENT / HOOK_DETAIL_WORKER_JOIN` after modification, matching the existing hook invocation order.
+2. The Controller shall call `on_worker_join` after a successful join (both new and rejoin) to push configuration to the worker.
+3. While the `workermgr` write lock is held during the join operation, the Controller shall keep the critical section as short as the current implementation.
+4. The existing unit tests in `src/controller/testsuite/` shall pass without modification.
--- a/.kiro/specs/worker-join-refactor/research.md
+++ b/.kiro/specs/worker-join-refactor/research.md
@ -0,0 +1,192 @@
+# Research Log: worker-join-refactor
+
+## Summary
+
+**Discovery scope**: Light discovery (extension of existing C subsystem).
+
+**Key findings**:
+1. **Latent deadlock in current code**: `handle_worker_join` acquires `lock(c->workermgr, WRITE)` then calls `add_worker → makeWorker`, which also calls `lock(workermgr, WRITE)`. `my_rwlock_t` is `pthread_rwlock_t` (non-reentrant), so this double-acquisition deadlocks. The outer lock was added for persistence-hook work on this branch and is the root cause of the "confusing locking structure."
+2. **`makeWorker` in wobj.c is out of scope but must still be called**: It manages the create-or-update decision internally (`makeWorker_`). For the create path, the refactored design must NOT hold the `workermgr` write lock when calling `makeWorker`.
+3. **`override_worker` already preserves admin bits**: The `XCE_WORKER_ADMIN_MASK` (`LOCKEDU | CLOSED | HIDDEN | CLOSED_BY_ADMIN`) is preserved by `override_worker` in wobj.c. The rejoin update path must replicate this guarantee when updating attrs in-place, bypassing `makeWorker`.
+
+---
+
+## Research Log
+
+### Topic 1: Lock Reentrancy
+
+**Question**: Is `my_rwlock_t` reentrant? Is the existing double-lock intentional?
+
+**Investigation**: `src/include/mylock.h` defines `my_rwlock_t` as `pthread_rwlock_t`. POSIX `pthread_rwlock_wrlock` on an already-held write lock by the same thread is undefined behavior (most implementations deadlock). The outer write lock in `handle_worker_join` was introduced on branch `persistent_ha` for the AFTER_EVENT persist hook (the `vget` lookup comment confirms this). This is a newly introduced bug.
+
+**Implication**: The refactored design must eliminate the outer `workermgr` write lock in `handle_worker_join` before any path reaches `makeWorker`.
+
+---
+
+### Topic 2: Existence Check and Locking Strategy
+
+**Question**: How should the explicit `key_find` check be locked?
+
+**Investigation**: `worker_ctrl` uses `lock(workermgr, READ)` → `key_find` → `reference(w)` → `unlock`. Other paths use read locks for existence checks and upgrade to per-object write locks for mutation. The workermgr write lock is only needed for structural changes (insert/delete of entries), not for updating existing worker fields.
+
+**Decision**: Use a brief `READ` lock for the `key_find` check. For the create path, delegate write locking to `makeWorker`. For the rejoin path, use a per-worker write lock.
+
+**TOCTOU note**: A brief window exists between the read-lock check and `makeWorker`'s write lock. This is benign: `makeWorker_` calls `key_find` internally under the write lock and, if the worker was created by another thread between our check and our call, will call `override_worker` (update) instead of `makeInstance` (create). The behavior remains correct.
+
+---
+
+### Topic 3: Rejoin Update — Replicating `override_worker` Semantics
+
+**Question**: For the rejoin path (explicit update), how should attrs be updated in-place without calling `override_worker` (which is static in wobj.c)?
+
+**Investigation**: `override_worker` in wobj.c:
+1. Saves admin bits (`status & XCE_WORKER_ADMIN_MASK`)
+2. Calls `kill_worker` (frees name, attrs, dispatch_window, thresholds, resvals)
+3. Calls `copy_worker` (re-allocates from new src)
+4. Restores admin bits and idx
+
+For the rejoin path in worker.c, we can replicate this semantic without calling `override_worker` directly. The update needs to: free old `hostattrs` fields (hosttype/model strings), copy new attrs, update `ep`, and preserve admin bits on `host.status`. The existing `dereference/makeString` pattern is already used in `add_worker` for hosttype/model override, so the pattern is established.
+
+**Decision**: Implement `rejoin_worker` in worker.c that replicates the admin-bit-preserving update logic inline, avoiding the static `override_worker` dependency.
+
+---
+
+### Topic 4: Resource and Status Update — Shared Path
+
+**Question**: Can the status/resource update logic in `add_worker` (req 4) be shared between create and rejoin paths?
+
+**Investigation**: The current `add_worker` applies the following after `makeWorker` returns, under a per-worker write lock:
+- `w->host.status &= ~XCE_WORKER_UNAVAIL`
+- `w->ep = ep`
+- hconfig hosttype/model override
+- `add_hostattrs_resvals` (resource vals for type, model, cpuf, etc.)
+- `set_host_slots` (mxj or maxcpus)
+- `dispatch_window_config` copy
+- `thresholds` copy
+- bool resource marking loop
+
+This block is identical for both create and rejoin. Extracting it into `static int apply_join_state(cluster_t*, worker_t*, endpoint_t*, xce_hostconfig_t*)` eliminates duplication.
+
+---
+
+### Topic 5: Hook Firing and Lock Relationship
+
+**Question**: Must BEFORE_EVENT and AFTER_EVENT hooks fire while the workermgr write lock is held?
+
+**Investigation**: The current code fires both hooks under the outer workermgr write lock. However, `mark_worker_unavail` and `mark_worker_back` fire hooks while holding only the per-worker write lock (not the workermgr lock). The persist hook signature takes `(c, w)` — it reads from `w` directly, not through the workermgr index. There is no architectural requirement for the workermgr write lock to be held during hook dispatch.
+
+**Decision**: Hooks fire in `handle_worker_join` outside any workermgr lock. BEFORE_EVENT fires before `key_find`. AFTER_EVENT fires after the create/update completes. This matches the behavioral contract from req 5.1.
+
+---
+
+## Architecture Decision: Removing `add_worker`, Adding `init_worker` + `rejoin_worker`
+
+The current `add_worker` is a monolithic function that internally calls `makeWorker` (which hides the create-or-update branch). The refactored design replaces it with:
+
+- `static int apply_join_state(cluster_t*, worker_t*, endpoint_t*, xce_hostconfig_t*)` — shared status/resource update logic (per-worker write lock)
+- `int init_worker(cluster_t*, endpoint_t*, const char*, hostattrs_t*, xce_hostconfig_t*)` — first-time join: calls `makeWorker` + `apply_join_state`
+- `int rejoin_worker(cluster_t*, worker_t*, endpoint_t*, hostattrs_t*, xce_hostconfig_t*)` — rejoin: updates attrs/ep in-place + `apply_join_state`
+
+`add_worker` declaration in `worker.h` is replaced with `init_worker` and `rejoin_worker`.
+
+---
+
+## Risk Assessment
+
+| Risk | Likelihood | Mitigation |
+|------|-----------|-----------|
+| TOCTOU between key_find and makeWorker | Very low (same host reconnecting twice simultaneously) | makeWorker_ handles gracefully via override |
+| Admin bits not preserved in rejoin path | Medium if not explicit | Explicit mask applied in rejoin_worker |
+| Hook ordering regression | Low | Hooks remain in handle_worker_join, same relative order |
+| Existing test breakage | Low | Tests exercise add_worker indirectly; signature change requires test fixture update |
+
+---
+
+# Gap Analysis (2026-05-20)
+
+## Analysis Summary
+
+- **Scope**: Narrow brownfield refactor — one C file (`worker.c`) with three missing function bodies. All surrounding scaffolding is already in place.
+- **Key challenge**: None blocking; the design and test contract are locked. Only implementation is missing.
+- **Recommendation**: Option A (extend existing `worker.c`) is the only appropriate approach.
+- **Effort**: S (1–3 days). All design decisions and lock semantics resolved; code to write is a mechanical extraction from the existing `add_worker` body.
+- **Risk**: Low. Existing `add_worker` contains the exact logic to extract. Tests in `test_worker_join.c` specify every assertion.
+
+## Current Implementation State
+
+| Component | File | Status |
+|-----------|------|--------|
+| `handle_worker_join` (rewritten) | `join.c` | ✅ Done — explicit `key_find` branch, calls `init_worker`/`rejoin_worker` |
+| `worker.h` declarations | `include/worker.h` | ✅ Done — `init_worker`, `rejoin_worker` exported; `add_worker` removed |
+| TDD tests | `testsuite/test_worker_join.c` | ✅ Done — 7 tests for `apply_join_state`, `init_worker`, `rejoin_worker` |
+| Test registered | `testsuite/controller_test.c` | ✅ Done — `test_worker_join` registered |
+| `apply_join_state` (static helper) | `worker.c` | ❌ Missing — body not yet extracted |
+| `init_worker` | `worker.c` | ❌ Missing — stub only; function body not yet written |
+| `rejoin_worker` | `worker.c` | ❌ Missing — stub only; function body not yet written |
+| `add_worker` removal | `worker.c` | ❌ Pending — old function body still present (unused by join.c) |
+
+## Requirement-to-Asset Map
+
+| Requirement | Asset Needed | Status | Gap Tag |
+|-------------|-------------|--------|---------|
+| 1.1 Explicit key_find lookup | `handle_worker_join` explicit key_find | ✅ Present in join.c | — |
+| 1.2 Branch on key_find result | `init_worker`/`rejoin_worker` dispatch in join.c | ✅ Present in join.c | — |
+| 1.3 Reject non-ENOENT errors | Error branch in join.c | ✅ Present in join.c | — |
+| 2.1 Create via makeWorker | `init_worker` calling `makeWorker` | ❌ Missing implementation | Missing |
+| 2.2 Log failure, no on_worker_join | Error path in join.c | ✅ Present in join.c | — |
+| 2.3 Assign priv->u.idx | join.c post-join assignment | ✅ Present in join.c | — |
+| 3.1 Update hostattrs on rejoin | `rejoin_worker` attrs copy | ❌ Missing implementation | Missing |
+| 3.2 Update ep on rejoin | `rejoin_worker` ep assignment | ❌ Missing implementation | Missing |
+| 3.3 Preserve admin bits | `XCE_WORKER_ADMIN_MASK` in `rejoin_worker` | ❌ Missing implementation | Missing |
+| 4.1 Clear UNAVAIL | `apply_join_state` status clear | ❌ Missing implementation | Missing |
+| 4.2 Apply hconfig overrides | `apply_join_state` hosttype/model/slots | ❌ Missing implementation | Missing |
+| 4.3 Update host-attr resources | `apply_join_state` calling `add_hostattrs_resvals` | ❌ Missing implementation | Missing |
+| 4.4 Reject if hostconfig NULL | join.c fail-fast before lock | ✅ Present in join.c | — |
+| 5.1 BEFORE/AFTER hook order | join.c hook call sites | ✅ Present in join.c | — |
+| 5.2 on_worker_join still called | join.c post-join call | ✅ Present in join.c | — |
+| 5.3 Critical section brevity | Locking model in join.c | ✅ Present in join.c | — |
+| 5.4 Existing tests pass | Test suite build | ❌ Pending — add_worker still present, init/rejoin stubs fail tests | Constraint |
+
+## Implementation Approach Options
+
+### Option A: Extend Existing worker.c (Recommended)
+
+**Rationale**: All design decisions are locked. The logic to implement already exists verbatim in `add_worker`. The task is extraction and reorganization:
+
+1. Add `apply_join_state` as a static helper — extract the `~XCE_WORKER_UNAVAIL` + ep + hconfig override + `add_hostattrs_resvals` + `set_host_slots` + `dispatch_window_config` + `thresholds` + bool resources loop from the bottom half of `add_worker`.
+2. Implement `init_worker` — build a `xce_worker_t` from name/attrs, call `makeWorker`, then call `apply_join_state` under per-worker write lock.
+3. Implement `rejoin_worker` — save admin bits, update attrs/ep in-place, call `apply_join_state`, restore admin bits.
+4. Remove `add_worker` (now dead code).
+
+**Files changed**: `src/controller/worker.c` only.
+
+Trade-offs:
+- ✅ Minimal surface — one file, pure extraction
+- ✅ Tests already written and specify exact postconditions
+- ✅ Lock semantics fully resolved in design.md
+- ❌ None — this is the obvious approach
+
+### Option B: New file (not appropriate)
+
+These are not new responsibilities — they replace `add_worker` in the same file. Creating a new file adds navigation overhead with no benefit.
+
+### Option C: Hybrid (not applicable)
+
+No external integration or phased rollout needed. Pure in-file refactor.
+
+## Implementation Complexity & Risk
+
+- **Effort**: S (1–3 days) — logic is a mechanical extraction from `add_worker`; all design decisions resolved
+- **Risk**: Low — no new patterns, no new dependencies, no new locking decisions; tests specify exact assertions
+
+## Recommendations for Design Phase
+
+Design phase is **already approved**. Proceed directly to implementation:
+
+1. Implement `apply_join_state` first (tasks 2 / task 1.1 green)
+2. Implement `init_worker` (task 3.1 / task 1.2 green)
+3. Implement `rejoin_worker` (task 3.2 / task 1.3 green)
+4. Remove `add_worker` (task 4.1 cleanup)
+5. Run `make test` in controller testsuite (task 5 validation)
+
+**No further research needed.** All unknowns from the original research log are resolved in `design.md`.