# Phase 6 — Scheduler Integration Implementation Plan

> **Status: ✓ COMPLETE — 2026-04-30 — 10/10 scheduler tests pass** | branch: `merge_api_server`

**Goal:** Wire the select-expression AST evaluator (`eval_select_simple`) into the scheduler so that `sched_meet_host_selection` filters hosts by actual job select expressions instead of always returning 1.

**Architecture:** Export GC mark primitives from `libxce` to break the circular dependency; add `_Atomic` GC-predicate fields to `xce_scheduler_t`; implement `build_host_eval_ctx` to project `xce_sched_worker_t.resvals` into `host_eval_ctx_t`; call `eval_select_simple` in the stub replacement; drive the full mark→sweep GC from `sched_end_session`.

**Tech Stack:** C99, Unity test framework, Meson/Ninja build, `nix develop` environment, `libxce` (`src/lib`), scheduler (`src/scheduler`)

---

## File Map

| File | Change |
|------|--------|
| `src/lib/xce_select.c` | Export `gc_mark_select_tree`; add `gc_clear_marks`; remove clear from `gc_sweep`; remove Phase 6 guard; add `sel_alloc_thread_id` + thread checks |
| `src/include/resreq.h` | Declare `gc_mark_select_tree`, `gc_clear_marks`, `sel_register_alloc_thread` |
| `src/lib/testsuite/test_select.c` | Prefix every `gc_sweep_select_nodes(NULL,0)` call with `gc_clear_marks()` |
| `src/scheduler/include/scheduler.h` | Add `_Atomic int session_active/eval_in_progress/staging_merge_complete` to `xce_scheduler_t`; declare `sched_session_active/sched_eval_in_progress/staging_merge_complete` |
| `src/scheduler/src/scheduler.c` | Add predicate static helpers; wire `sel_optimize_set(1)` in `sched_init`; drive GC from `sched_end_session` with atomic fence |
| `src/scheduler/src/resource.c` | Add `build_host_eval_ctx`; replace `sched_meet_host_selection` stub |
| `src/scheduler/testsuite/meson.build` | Link `resource.c` + `scheduler.c` for integration tests |
| `src/scheduler/testsuite/test.c` | Add `test_meet_host_selection` integration test |

---

## Task 1: Pre-audit — confirm single-threaded invariant and stub locations (6-1, 6-2)

**Files:**
- Read: `src/scheduler/src/resource.c`
- Read: `src/scheduler/src/main.c`

- [x] **Step 1: Run the blocking-gate grep**

```bash
cd /workspace
grep -rn 'sched_meet_host_selection\|meet_host_selection' src/scheduler/ src/lib/
grep -rn 'eval_select\|alloc_thread\|sched_session' src/scheduler/src/
```

Expected: `sched_meet_host_selection` appears in exactly 3 locations — `scheduler.h` (decl), `resource.c` (stub def), `resource.c` (call site at `sched_try_alloc_on_host_`). No other call sites.

- [x] **Step 2: Confirm allocator-thread-only invariant**

```bash
grep -n 'sched_begin_session\|sched_end_session\|sched_go' src/scheduler/src/main.c
```

Expected: all three called from the same `while(1)` loop in `main`. Document result: "confirmed single-threaded allocator loop in main.c lines ~119-168".

- [x] **Step 3: Confirm `eval_select_simple` signature**

```bash
grep -n 'eval_select_simple' src/include/resreq.h src/lib/xce_select.c
```

Expected: `int eval_select_simple(const xce_select_node_t* root, const host_eval_ctx_t* host)` — returns 1 (match), 0 (no match), negative (error).

- [x] **Step 4: Commit audit result**

```bash
git add -p   # no code changes
git commit -m "chore: phase 6 pre-audit — allocator single-threaded confirmed, stub at resource.c:sched_meet_host_selection"
```

---

## Task 2: Export GC mark primitives — split gc_sweep_select_nodes (6-8 prerequisite)

The circular dependency between `xce_select.c` (libxce) and `sched/types.h` (scheduler) means the GC mark pass cannot live inside `gc_sweep_select_nodes`. Solution: export `gc_clear_marks` and `gc_mark_select_tree` so the **scheduler** owns the mark phase.

`gc_sweep_select_nodes` currently clears marks internally before sweeping. After this task it only sweeps — callers must call `gc_clear_marks()` first. The Phase 6 guard (`if (live_jobs != NULL && njobs > 0) return;`) is removed.

**Files:**
- Modify: `src/lib/xce_select.c`
- Modify: `src/include/resreq.h`
- Modify: `src/lib/testsuite/test_select.c`

- [x] **Step 1: Write the failing test**

In `src/lib/testsuite/test_select.c`, add after the existing GC tests (around line 800):

```c
static void
test_gc_clear_marks_exported(void)
{
    /* After this call, all nodes in the intern table must have gc_mark == 0.
     * We verify by immediately calling gc_sweep (which only sweeps unmarked
     * nodes), expecting no crash and consistent node count. */
    sel_optimize_test_reset();
    sel_optimize_set(1);
    xstring_table_init(64 * 1024);
    sel_node_table_init();

    /* intern two nodes */
    xce_select_node_t* n1 = makeSelectNode(SEL_NUM, (uintptr_t)1.0, 0);
    xce_select_node_t* n2 = makeSelectNode(SEL_NUM, (uintptr_t)2.0, 0);
    TEST_ASSERT_NOT_NULL(n1);
    TEST_ASSERT_NOT_NULL(n2);

    /* mark n1 manually */
    n1->gc_mark = 1;

    /* clear should reset n1->gc_mark */
    gc_clear_marks();
    TEST_ASSERT_EQUAL_INT(0, n1->gc_mark);

    /* mark n2 so sweep preserves it; n1 (unmarked) is swept */
    gc_mark_select_tree(n2);
    TEST_ASSERT_EQUAL_INT(1, n2->gc_mark);

    gc_sweep_select_nodes(NULL, 0);
    /* n1 swept — n2 still alive */
    TEST_ASSERT_EQUAL_INT(1, n2->gc_mark);  /* mark survives sweep */

    sel_node_table_fini();
    sel_optimize_test_reset();
}
```

In the `main` function of `test_select.c`, register the test:

```c
TEST(test_gc_clear_marks_exported);
```

- [x] **Step 2: Run test to verify it fails**

```bash
nix develop --command bash -c "cd /workspace/src && meson setup build --wipe 2>/dev/null || true && ninja -C build lib/testsuite/xce_test 2>&1 | tail -30"
```

Expected: compile error — `gc_clear_marks` and `gc_mark_select_tree` undeclared.

- [x] **Step 3: Add declarations to resreq.h**

In `src/include/resreq.h`, after the `gc_sweep_select_nodes` declaration (line ~121):

```c
/* GC mark primitives — called by the scheduler to implement the mark phase.
 * Call order: gc_clear_marks() → gc_mark_select_tree(root) per live node
 *             → gc_sweep_select_nodes(NULL, 0) */
extern void gc_clear_marks(void);
extern void gc_mark_select_tree(xce_select_node_t* node);
```

- [x] **Step 4: Implement gc_clear_marks and gc_mark_select_tree in xce_select.c**

In `src/lib/xce_select.c`, locate the `gc_mark_tree` function at line ~805:

```c
static void __attribute__((unused)) gc_mark_tree(xce_select_node_t* node)
```

Replace with (non-static, renamed):

```c
void gc_mark_select_tree(xce_select_node_t* node)
{
    if (node == NULL || node->gc_mark) return;
    node->gc_mark = 1;
    switch (node->type) {
    case SEL_RESNAME: case SEL_NUM: case SEL_STR:
    case SEL_ANY:     case SEL_TRUE: case SEL_FALSE:
        break;
    case SEL_NOT:
        gc_mark_select_tree(node->u.operand);
        break;
    default:
        gc_mark_select_tree(node->u.binop.left);
        gc_mark_select_tree(node->u.binop.right);
        break;
    }
}
```

Above that, add `gc_clear_marks`:

```c
void gc_clear_marks(void)
{
    if (sel_node_table.table == NULL) return;
    int caps = sel_node_table.nodes.caps;
    for (int i = 0; i < caps; i++) {
        xce_select_node_t* node =
            (xce_select_node_t*)zarray_get(&sel_node_table.nodes, i);
        if (node == NULL) continue;
        node->gc_mark = 0;
    }
}
```

- [x] **Step 5: Modify gc_sweep_select_nodes — remove clear loop and Phase 6 guard**

In `src/lib/xce_select.c`, in `gc_sweep_select_nodes` (line ~826):

Remove the Phase 6 guard block:
```c
    if (live_jobs != NULL && njobs > 0) {
        return;
    }
```

Remove the clear loop (step 1, lines ~841-852):
```c
    /* 1. Clear all gc_mark bits ... */
    for (int i = 0; i < caps; i++) {
        xce_select_node_t* node =
            (xce_select_node_t*)zarray_get(&sel_node_table.nodes, i);
        if (node == NULL) continue;
        node->gc_mark = 0;
    }
```

The function now starts directly with the `int caps = sel_node_table.nodes.caps;` line and the sweep loop.

- [x] **Step 6: Update test_select.c — prefix all gc_sweep_select_nodes(NULL,0) calls**

```bash
grep -n 'gc_sweep_select_nodes(NULL, 0)' src/lib/testsuite/test_select.c
```

For every occurrence, prepend `gc_clear_marks();` on the line before. There should be 4 occurrences. Example — each:

```c
/* before */
gc_sweep_select_nodes(NULL, 0);

/* after */
gc_clear_marks();
gc_sweep_select_nodes(NULL, 0);
```

- [x] **Step 7: Build and run tests**

```bash
nix develop --command bash -c "cd /workspace/src && ninja -C build && build/lib/testsuite/xce_test -t test_gc_clear_marks_exported"
```

Expected: `test_gc_clear_marks_exported` PASS. Run full suite:

```bash
nix develop --command bash -c "cd /workspace/src && build/lib/testsuite/xce_test"
```

Expected: all GC tests PASS.

- [x] **Step 8: Commit**

```bash
git add src/lib/xce_select.c src/include/resreq.h src/lib/testsuite/test_select.c
git commit -m "feat: export gc_clear_marks + gc_mark_select_tree; remove clear from gc_sweep"
```

---

## Task 3: Add _Atomic GC-predicate fields to xce_scheduler_t (6-3)

**Files:**
- Modify: `src/scheduler/include/scheduler.h`

- [x] **Step 1: Write the failing test**

In `src/scheduler/testsuite/test.c`, add:

```c
#include <stdatomic.h>
#include "scheduler.h"

static void
test_scheduler_t_has_atomic_gc_fields(void)
{
    xce_scheduler_t sched;
    memset(&sched, 0, sizeof(sched));

    /* Verify atomic fields exist and are zero-initialized */
    TEST_ASSERT_EQUAL_INT(0, atomic_load(&sched.session_active));
    TEST_ASSERT_EQUAL_INT(0, atomic_load(&sched.eval_in_progress));
    TEST_ASSERT_EQUAL_INT(0, atomic_load(&sched.staging_merge_complete));
}
```

In `main`, add: `TEST(test_scheduler_t_has_atomic_gc_fields);`

- [x] **Step 2: Run to verify it fails**

```bash
nix develop --command bash -c "cd /workspace/src && ninja -C build scheduler/testsuite/sched_test 2>&1 | tail -20"
```

Expected: compile error — `session_active` not a member of `xce_scheduler_t`.

- [x] **Step 3: Add fields to xce_scheduler_t**

In `src/scheduler/include/scheduler.h`, add three fields at the end of `xce_scheduler_t` (before the closing `}`, after `cbuf_t decisions;`):

```c
    _Atomic int session_active;         /* 1 during scheduling session */
    _Atomic int eval_in_progress;       /* 1 while eval_select loop runs */
    _Atomic int staging_merge_complete; /* 1 after staging merge completes */
```

Also add `#include <stdatomic.h>` near the top of `scheduler.h` (after `#include <time.h>`).

- [x] **Step 4: Build and run**

```bash
nix develop --command bash -c "cd /workspace/src && ninja -C build && build/scheduler/testsuite/sched_test"
```

Expected: `test_scheduler_t_has_atomic_gc_fields` PASS.

- [x] **Step 5: Commit**

```bash
git add src/scheduler/include/scheduler.h src/scheduler/testsuite/test.c
git commit -m "feat: add _Atomic gc-predicate fields to xce_scheduler_t (6-3)"
```

---

## Task 4: Add GC-predicate statics in scheduler.c + wire sel_optimize_set (6-4, 6-9)

**Files:**
- Modify: `src/scheduler/src/scheduler.c`
- Modify: `src/scheduler/include/scheduler.h`

The predicates are scheduler-internal — `static` helpers in `scheduler.c`. They are called at the GC call site (Task 7). The `fake_scheduler_t` stubs in `test_select.c` remain (unit tests still use them).

- [x] **Step 1: Write failing test**

In `src/scheduler/testsuite/test.c`, add:

```c
static void
test_sel_optimize_level_set_at_init(void)
{
    /* After sched_init, sel_get_optimize_level() must return 1. */
    /* We can check it directly since it's a library call. */
    /* We can't call sched_init (it starts threads) so check indirectly:
     * the field is set by sel_optimize_set — test the setter directly. */
    sel_optimize_set(1);
    TEST_ASSERT_EQUAL_INT(1, sel_get_optimize_level());
    sel_optimize_set(0);  /* reset for other tests */
}
```

Add `#include "resreq.h"` to test includes.

In `main`, add: `TEST(test_sel_optimize_level_set_at_init);`

- [x] **Step 2: Run to verify it fails**

```bash
nix develop --command bash -c "cd /workspace/src && ninja -C build scheduler/testsuite/sched_test 2>&1 | tail -20"
```

Expected: compile error — `sel_optimize_set` undeclared (not linked yet — Task 8 will fix meson.build; for now just verify the include path works).

Note: If it fails on `sel_optimize_set` not linked, proceed to Step 3 and come back.

- [x] **Step 3: Add static predicate helpers to scheduler.c**

In `src/scheduler/src/scheduler.c`, after the `#include` block (around line 20), add:

```c
#include "resreq.h"

static int sched_session_active(xce_scheduler_t* sched)
{
    return atomic_load_explicit(&sched->session_active, memory_order_acquire);
}

static int sched_eval_in_progress(xce_scheduler_t* sched)
{
    return atomic_load_explicit(&sched->eval_in_progress, memory_order_acquire);
}

static int sched_staging_merge_complete(xce_scheduler_t* sched)
{
    return atomic_load_explicit(&sched->staging_merge_complete,
                                memory_order_acquire);
}
```

- [x] **Step 4: Wire sel_optimize_set(1) in sched_init**

In `src/scheduler/src/scheduler.c`, in `sched_init` (line ~318), after `memset(sched, 0x0, sizeof(*sched));` (line ~357), add:

```c
    sel_optimize_set(1);
```

This must come before any `makeSelectNode` calls (which happen during job parsing). The write-once guard in `sel_optimize_set` ensures it can only be set once per process.

- [x] **Step 5: Build**

```bash
nix develop --command bash -c "cd /workspace/src && ninja -C build 2>&1 | grep -E 'error:|warning:|FAILED|OK' | head -20"
```

Expected: builds cleanly (the test executable link will be fixed in Task 9 meson update).

- [x] **Step 6: Commit**

```bash
git add src/scheduler/src/scheduler.c
git commit -m "feat: add sched GC predicate statics; wire sel_optimize_set(1) in sched_init (6-4, 6-9)"
```

---

## Task 5: Implement build_host_eval_ctx (6-7)

`build_host_eval_ctx` flattens `xce_sched_worker_t.resvals` (a `zarray_t` of `xce_val_t*` indexed by `res_idx`) into the `numvals`/`strvals` arrays expected by `host_eval_ctx_t`.

Capnp constants (from `controller.capnp.h`): `Value_u_fvalue = 5` (float), `Value_u_svalue = 4` (string), `Value_u_bvalue = 6` (bool → stored as 0.0/1.0 in `numvals`).

**Files:**
- Modify: `src/scheduler/src/resource.c`

- [x] **Step 1: Write the failing test**

In `src/scheduler/testsuite/test.c`, add:

```c
#include "sched/types.h"
#include "resreq.h"
#include "zarray.h"
#include "controller.capnp.h"

/* build_host_eval_ctx is static — declare prototype for test access */
static host_eval_ctx_t* build_host_eval_ctx(const xce_sched_worker_t* w,
                                             int num_res);

static void
test_build_host_eval_ctx_float(void)
{
    /* Build a worker with one float resource at res_idx=2 (value=4.0) */
    xce_sched_worker_t w;
    memset(&w, 0, sizeof(w));
    zarray_init(&w.resvals, 8, NULL);

    xce_val_t v;
    memset(&v, 0, sizeof(v));
    v.which      = Value_u_fvalue;
    v.u.fvalue   = 4.0f;
    xce_val_t* vp = &v;

    /* slot 0, 1 = NULL; slot 2 = &v */
    zarray_set(&w.resvals, 0, NULL);
    zarray_set(&w.resvals, 1, NULL);
    zarray_set(&w.resvals, 2, &vp);

    host_eval_ctx_t* ctx = build_host_eval_ctx(&w, 3);
    TEST_ASSERT_NOT_NULL(ctx);
    TEST_ASSERT_EQUAL_DOUBLE(4.0, ctx->numvals[2]);
    TEST_ASSERT_NULL(ctx->strvals[2]);

    free((void*)ctx->numvals);
    free((void*)ctx->strvals);
    free(ctx);
    zarray_fini(&w.resvals);
}
```

In `main`, add: `TEST(test_build_host_eval_ctx_float);`

- [x] **Step 2: Run to verify it fails**

```bash
nix develop --command bash -c "cd /workspace/src && ninja -C build scheduler/testsuite/sched_test 2>&1 | tail -20"
```

Expected: compile error — `build_host_eval_ctx` not yet defined.

- [x] **Step 3: Implement build_host_eval_ctx in resource.c**

In `src/scheduler/src/resource.c`, add `#include "controller.capnp.h"` to the include block, then add before `sched_meet_host_selection`:

```c
/* Build a host_eval_ctx_t from a scheduler worker's resource-value zarray.
 * Allocates numvals and strvals on the heap; caller frees all three.
 * Returns NULL on OOM. */
static host_eval_ctx_t*
build_host_eval_ctx(const xce_sched_worker_t* w, int num_res)
{
    host_eval_ctx_t* ctx = calloc(1, sizeof(*ctx));
    if (ctx == NULL) return NULL;

    double*      numvals = calloc(num_res, sizeof(double));
    const char** strvals = calloc(num_res, sizeof(char*));
    if (numvals == NULL || strvals == NULL) {
        free(numvals);
        free(strvals);
        free(ctx);
        return NULL;
    }

    for (int i = 0; i < num_res; i++) {
        xce_val_t** slot = (xce_val_t**)zarray_get(&w->resvals, i);
        if (slot == NULL || *slot == NULL) continue;
        xce_val_t* v = *slot;
        switch (v->which) {
        case Value_u_fvalue:
            numvals[i] = (double)v->u.fvalue;
            break;
        case Value_u_bvalue:
            numvals[i] = v->u.bvalue ? 1.0 : 0.0;
            break;
        case Value_u_svalue:
            strvals[i] = v->u.svalue;
            break;
        }
    }

    ctx->num_res = num_res;
    ctx->numvals = numvals;
    ctx->strvals = strvals;
    return ctx;
}

static void
free_host_eval_ctx(host_eval_ctx_t* ctx)
{
    if (ctx == NULL) return;
    free((void*)ctx->numvals);
    free((void*)ctx->strvals);
    free(ctx);
}
```

- [x] **Step 4: Check xce_val_t layout**

```bash
grep -n 'xce_val_t\|fvalue\|svalue\|bvalue' src/include/xce/types.h | head -20
grep -n 'Value_u_fvalue\|Value_u_svalue\|Value_u_bvalue' src/scheduler/src/resource.c || \
  grep -rn 'Value_u_fvalue' src/controller/ | head -3
```

Verify the `u.fvalue`, `u.svalue`, `u.bvalue` field names match `xce_val_t` in `controller.capnp.h`.

- [x] **Step 5: Build and run test**

```bash
nix develop --command bash -c "cd /workspace/src && ninja -C build && build/scheduler/testsuite/sched_test -t test_build_host_eval_ctx_float"
```

Expected: PASS.

- [x] **Step 6: Commit**

```bash
git add src/scheduler/src/resource.c
git commit -m "feat: implement build_host_eval_ctx — project resvals zarray to host_eval_ctx_t (6-7)"
```

---

## Task 6: Implement sched_meet_host_selection (6-5)

Replace the stub with `eval_select_simple`. This is the pragmatic Phase 6 approach (no session restructuring — deferred to a later phase). The `int` return value and `add_job_preason` call in the caller remain unchanged.

**Files:**
- Modify: `src/scheduler/src/resource.c`

- [x] **Step 1: Write the failing test**

In `src/scheduler/testsuite/test.c`, add:

```c
#include "kvmgr.h"

static void
test_meet_host_selection_null_select_passes(void)
{
    /* A NULL select pointer means no expression — always pass */
    xce_scheduler_t sched;
    memset(&sched, 0, sizeof(sched));
    xce_sched_worker_t w;
    memset(&w, 0, sizeof(w));
    zarray_init(&w.resvals, 4, NULL);

    int rc = sched_meet_host_selection(&sched, &w, NULL);
    TEST_ASSERT_EQUAL_INT(1, rc);

    zarray_fini(&w.resvals);
}

static void
test_meet_host_selection_always_true_passes(void)
{
    xce_scheduler_t sched;
    memset(&sched, 0, sizeof(sched));
    sel_optimize_set(1);
    sel_node_table_init();

    xce_sched_worker_t w;
    memset(&w, 0, sizeof(w));
    zarray_init(&w.resvals, 4, NULL);

    /* SEL_TRUE root — must pass any host */
    xce_select_node_t* root = makeSelectNode(SEL_TRUE, 0, 0);
    TEST_ASSERT_NOT_NULL(root);

    xce_select_t sel = { .root = root };
    int rc = sched_meet_host_selection(&sched, &w, &sel);
    TEST_ASSERT_EQUAL_INT(1, rc);

    zarray_fini(&w.resvals);
    gc_clear_marks();
    gc_sweep_select_nodes(NULL, 0);
    sel_node_table_fini();
    sel_optimize_test_reset();
}

static void
test_meet_host_selection_always_false_fails(void)
{
    xce_scheduler_t sched;
    memset(&sched, 0, sizeof(sched));
    sel_optimize_set(1);
    sel_node_table_init();

    xce_sched_worker_t w;
    memset(&w, 0, sizeof(w));
    zarray_init(&w.resvals, 4, NULL);

    /* SEL_FALSE root — must fail every host */
    xce_select_node_t* root = makeSelectNode(SEL_FALSE, 0, 0);
    TEST_ASSERT_NOT_NULL(root);

    xce_select_t sel = { .root = root };
    int rc = sched_meet_host_selection(&sched, &w, &sel);
    TEST_ASSERT_EQUAL_INT(0, rc);

    zarray_fini(&w.resvals);
    gc_clear_marks();
    gc_sweep_select_nodes(NULL, 0);
    sel_node_table_fini();
    sel_optimize_test_reset();
}
```

In `main`, register all three tests.

- [x] **Step 2: Run to verify failures**

```bash
nix develop --command bash -c "cd /workspace/src && ninja -C build scheduler/testsuite/sched_test 2>&1 | tail -20"
```

Expected: link errors (scheduler sources not yet in meson) OR `test_meet_host_selection_always_false_fails` FAIL (stub returns 1 for SEL_FALSE).

Proceed to Step 3.

- [x] **Step 3: Replace the stub in resource.c**

In `src/scheduler/src/resource.c`, replace the `sched_meet_host_selection` function body (currently lines ~415-422):

```c
int sched_meet_host_selection(xce_scheduler_t* sched, xce_sched_worker_t* w,
                               xce_select_t* s)
{
    if (s == NULL || s->root == NULL) return 1;

    /* Count resources: use resvals.caps as upper bound on res_idx */
    int num_res = (int)w->resvals.caps;

    host_eval_ctx_t* hctx = build_host_eval_ctx(w, num_res);
    if (hctx == NULL) {
        /* OOM: conservatively pass host — scheduler will reject on resource
         * quantities anyway */
        return 1;
    }

    int rc = eval_select_simple(s->root, hctx);
    free_host_eval_ctx(hctx);

    /* eval_select_simple: 1=match, 0=no-match, <0=error (treat as no-match) */
    return (rc > 0) ? 1 : 0;
}
```

Add `#include "resreq.h"` to `resource.c` includes if not already present.

- [x] **Step 4: Build and run**

```bash
nix develop --command bash -c "cd /workspace/src && ninja -C build 2>&1 | grep -E 'error:|FAILED' | head -10"
```

Expected: builds cleanly.

- [x] **Step 5: Commit**

```bash
git add src/scheduler/src/resource.c
git commit -m "feat: implement sched_meet_host_selection using eval_select_simple (6-5)"
```

---

## Task 7: Drive GC from sched_end_session with live_jobs + atomic fence (6-8, 6-4, 6-12)

`sched_end_session` collects all live jobs, performs the mark phase, inserts the seq_cst fence, then sweeps. The GC idle invariant (6-4) is enforced by the three predicate guards.

**Files:**
- Modify: `src/scheduler/src/scheduler.c`

- [x] **Step 1: Write the failing test**

In `src/scheduler/testsuite/test.c`, add:

```c
static void
test_gc_predicates_idle_invariant(void)
{
    /* session_active=0, eval_in_progress=0, staging_merge_complete=0 after
     * memset — GC call site guard must NOT log_fatal. We can only test
     * indirectly by calling sched_end_session on an empty scheduler. */
    xce_scheduler_t sched;
    memset(&sched, 0, sizeof(sched));
    /* staging_merge_complete must be 1 for GC to proceed */
    atomic_store(&sched.staging_merge_complete, 1);

    sel_optimize_set(1);
    sel_node_table_init();

    /* sched_end_session with no jobs — should call gc with empty live_jobs */
    /* We can't call sched_end_session directly (it calls plugin_end_session
     * which requires full init). Test the predicate reads directly. */
    TEST_ASSERT_EQUAL_INT(0, atomic_load(&sched.session_active));
    TEST_ASSERT_EQUAL_INT(0, atomic_load(&sched.eval_in_progress));
    TEST_ASSERT_EQUAL_INT(1, atomic_load(&sched.staging_merge_complete));

    gc_clear_marks();
    gc_sweep_select_nodes(NULL, 0);

    sel_node_table_fini();
    sel_optimize_test_reset();
}
```

In `main`, add: `TEST(test_gc_predicates_idle_invariant);`

- [x] **Step 2: Run to verify**

```bash
nix develop --command bash -c "cd /workspace/src && ninja -C build scheduler/testsuite/sched_test 2>&1 | tail -20"
```

- [x] **Step 3: Add live_jobs collection and GC call to sched_end_session**

In `src/scheduler/src/scheduler.c`, replace `sched_end_session` (line ~752):

```c
void sched_end_session(xce_scheduler_t* sched)
{
    plugin_end_session(sched);

    /* GC idle invariant guard — any violation is a use-after-free risk */
    if (sched_session_active(sched) || sched_eval_in_progress(sched)
        || !sched_staging_merge_complete(sched)) {
        log_fatal("sched_end_session: GC idle invariant violated: "
                  "session_active=%d eval_in_progress=%d staging_merge=%d",
                  sched_session_active(sched),
                  sched_eval_in_progress(sched),
                  sched_staging_merge_complete(sched));
        return;
    }

    /* Collect live jobs into a flat array for GC mark phase */
    rtti_iter_t      it;
    xce_sched_job_t* job       = NULL;
    int              njobs     = 0;
    int              cap       = 64;
    xce_sched_job_t** live_jobs = calloc(cap, sizeof(*live_jobs));
    if (live_jobs == NULL) goto sweep;

    iter_init(&it);
    lock(sched->jobs, RTTI_LOCK_READ);
    while (iter_next(&it, sched->jobs, (void**)&job) == RTTI_ITER_OK) {
        if (njobs >= cap) {
            int newcap = cap * 2;
            xce_sched_job_t** tmp = realloc(live_jobs,
                                            newcap * sizeof(*live_jobs));
            if (tmp == NULL) {
                unlock(sched->jobs);
                iter_destroy(&it);
                goto sweep_with_jobs;
            }
            live_jobs = tmp;
            cap       = newcap;
        }
        live_jobs[njobs++] = job;
    }
    unlock(sched->jobs);
    iter_destroy(&it);

sweep_with_jobs:
    /* Mark phase: clear all gc_marks, then mark every select root reachable
     * from live job resreq trees */
    gc_clear_marks();
    for (int i = 0; i < njobs; i++) {
        xce_resreq_t* rr = live_jobs[i]->resreq;
        if (rr == NULL) continue;
        /* dispatch on resreq tag to reach all select roots */
        if (rr->tag == XCE_RESREQ_PRIMARY) {
            if (rr->u.primary && rr->u.primary->select)
                gc_mark_select_tree(rr->u.primary->select->root);
        } else if (rr->tag == XCE_RESREQ_COMPOUND) {
            if (rr->u.compound) {
                xlist_iter_t li;
                xce_primary_resreq_t* pr;
                xlist_for_each(&rr->u.compound->primaries, &li,
                               (void**)&pr) {
                    if (pr && pr->select)
                        gc_mark_select_tree(pr->select->root);
                }
            }
        } else if (rr->tag == XCE_RESREQ_ALTER) {
            if (rr->u.alter) {
                xlist_iter_t li;
                xce_compound_resreq_t* cp;
                xlist_for_each(&rr->u.alter->compounds, &li,
                               (void**)&cp) {
                    if (cp == NULL) continue;
                    xlist_iter_t li2;
                    xce_primary_resreq_t* pr;
                    xlist_for_each(&cp->primaries, &li2, (void**)&pr) {
                        if (pr && pr->select)
                            gc_mark_select_tree(pr->select->root);
                    }
                }
            }
        }
    }

    /* Fence: ensure all gc_mark writes are visible before the sweep reads
     * them.  Also ensures NULL-stores to unlinked roots (if any) are visible
     * before gc_mark_select_tree reads root pointers. */
    atomic_thread_fence(memory_order_seq_cst);

sweep:
    gc_sweep_select_nodes(NULL, 0);
    free(live_jobs);
}
```

Add `#include <stdatomic.h>` to `scheduler.c` includes, and ensure `#include "resreq.h"` is present.

Check `xlist_for_each` macro — if it doesn't exist, use `xlist_iter_t` loop pattern from existing code in the file:

```bash
grep -n 'xlist_for_each\|xlist_iter' src/scheduler/src/scheduler.c | head -5
```

If `xlist_for_each` is not available, replace with the pattern used elsewhere in `scheduler.c`.

- [x] **Step 4: Set/clear session_active and staging_merge_complete at session boundaries**

In `sched_begin_session` (line ~740), at the START:

```c
    atomic_store_explicit(&sched->session_active, 1, memory_order_release);
    atomic_store_explicit(&sched->staging_merge_complete, 0, memory_order_release);
```

In `sched_end_session`, at the END (after the GC sweep), add:

```c
    atomic_store_explicit(&sched->staging_merge_complete, 1, memory_order_release);
```

And in `sched_end_session` at the VERY START (before the guard), add:

```c
    atomic_store_explicit(&sched->session_active, 0, memory_order_release);
```

- [x] **Step 5: Build**

```bash
nix develop --command bash -c "cd /workspace/src && ninja -C build 2>&1 | grep -E 'error:|FAILED' | head -20"
```

Expected: builds cleanly.

- [x] **Step 6: Commit**

```bash
git add src/scheduler/src/scheduler.c
git commit -m "feat: drive GC from sched_end_session — live_jobs mark, seq_cst fence, gc_sweep (6-8, 6-12)"
```

---

## Task 8: Add sel_alloc_thread_id thread-ownership checks (6-10)

**Files:**
- Modify: `src/lib/xce_select.c`
- Modify: `src/include/resreq.h`
- Modify: `src/scheduler/src/scheduler.c` (register allocator thread)

- [x] **Step 1: Write the failing test**

In `src/lib/testsuite/test_select.c`, add:

```c
static void
test_sel_register_alloc_thread(void)
{
    /* sel_alloc_thread_id is 0 (unregistered) before registration —
     * thread checks must be skipped.  After registration they must
     * match pthread_self(). */
    sel_optimize_test_reset();   /* resets sel_alloc_thread_id to 0 */
    sel_optimize_set(1);
    sel_node_table_init();

    /* Should not crash when unregistered */
    xce_select_node_t* n = makeSelectNode(SEL_TRUE, 0, 0);
    TEST_ASSERT_NOT_NULL(n);

    /* Register current thread */
    sel_register_alloc_thread();

    /* makeSelectNode from same thread must still work */
    xce_select_node_t* n2 = makeSelectNode(SEL_FALSE, 0, 0);
    TEST_ASSERT_NOT_NULL(n2);

    gc_clear_marks();
    gc_sweep_select_nodes(NULL, 0);
    sel_node_table_fini();
    sel_optimize_test_reset();
}
```

In `main`, add: `TEST(test_sel_register_alloc_thread);`

- [x] **Step 2: Run to verify it fails**

```bash
nix develop --command bash -c "cd /workspace/src && ninja -C build lib/testsuite/xce_test 2>&1 | tail -15"
```

Expected: compile error — `sel_register_alloc_thread` undeclared.

- [x] **Step 3: Add sel_alloc_thread_id to xce_select.c**

In `src/lib/xce_select.c`, add `#include <pthread.h>` to the include block.

After the `sel_optimize_level` static variable declaration, add:

```c
static pthread_t sel_alloc_thread_id; /* 0 = unregistered, skip checks */
```

Add the registration function before `eval_select`:

```c
void sel_register_alloc_thread(void)
{
    sel_alloc_thread_id = pthread_self();
}
```

Extend `sel_optimize_test_reset` to also reset `sel_alloc_thread_id`:

```c
void sel_optimize_test_reset(void)
{
    atomic_store_explicit(&sel_optimize_level, -1, memory_order_seq_cst);
    sel_alloc_thread_id = 0;
}
```

- [x] **Step 4: Add thread checks in eval_select and gc_sweep_select_nodes**

In `eval_select` (line ~633), after the `sel_get_optimize_level()` guard, add:

```c
    if (sel_alloc_thread_id != 0 && pthread_self() != sel_alloc_thread_id) {
        ERROR("eval_select called from non-allocator thread");
        return -1;
    }
```

In `gc_sweep_select_nodes` (line ~826), at the top after the `table == NULL` check, add:

```c
    if (sel_alloc_thread_id != 0 && pthread_self() != sel_alloc_thread_id) {
        ERROR("gc_sweep_select_nodes called from non-allocator thread");
        return;
    }
```

- [x] **Step 5: Declare sel_register_alloc_thread in resreq.h**

In `src/include/resreq.h`, after `sel_optimize_set`/`sel_get_optimize_level`:

```c
/* Register the current thread as the sole allocator thread.
 * Must be called once before any eval_select or gc_sweep_select_nodes call.
 * A value of 0 (default) disables the check (unit-test mode). */
extern void sel_register_alloc_thread(void);
```

- [x] **Step 6: Wire registration in scheduler main loop**

In `src/scheduler/src/main.c`, in the allocator thread's first `while(1)` iteration (line ~119, before `sched_begin_session`), add a one-time call:

```c
    /* Register this thread as the sole eval/GC thread (write-once) */
    static int sel_thread_registered = 0;
    if (!sel_thread_registered) {
        sel_register_alloc_thread();
        sel_thread_registered = 1;
    }
```

Add `#include "resreq.h"` to `main.c` includes.

- [x] **Step 7: Build and run**

```bash
nix develop --command bash -c "cd /workspace/src && ninja -C build && build/lib/testsuite/xce_test -t test_sel_register_alloc_thread"
```

Expected: PASS. Run full lib suite:

```bash
nix develop --command bash -c "cd /workspace/src && build/lib/testsuite/xce_test"
```

Expected: all tests PASS.

- [x] **Step 8: Commit**

```bash
git add src/lib/xce_select.c src/include/resreq.h src/scheduler/src/main.c
git commit -m "feat: add sel_alloc_thread_id ownership check in eval_select and gc_sweep (6-10)"
```

---

## Task 9: Scheduler testsuite — link resource.c + integration tests (6-11)

**Files:**
- Modify: `src/scheduler/testsuite/meson.build`
- Modify: `src/scheduler/testsuite/test.c`

- [x] **Step 1: Update meson.build to link scheduler source files**

Replace `src/scheduler/testsuite/meson.build` with:

```meson
scheduler_test_src = [
  'test.c',
  '../src/resource.c',
]

scheduler_test_deps = [libxce_deps, librtti_dep, unity_dep]

incdir = include_directories('../include', '../../include', '.')

scheduler_test = executable('sched_test',
                            scheduler_test_src,
                            include_directories:[incdir],
                            dependencies: [scheduler_test_deps],
                            install: false)
```

Note: Only `resource.c` is needed — it implements the functions under test. `scheduler.c` (which calls `sched_init` and starts threads) is intentionally excluded to keep tests free of thread/network side effects.

- [x] **Step 2: Write integration test with a known-good resreq/host pair**

In `src/scheduler/testsuite/test.c`, add:

```c
#include "xstring.h"

static void
test_meet_host_selection_numeric_match(void)
{
    /* Job requires: ncpus >= 4.
     * Worker has ncpus=8 at res_idx=0 → must match (rc=1).
     * Worker has ncpus=2 at res_idx=0 → must not match (rc=0). */

    sel_optimize_set(1);
    xstring_table_init(64 * 1024);
    sel_node_table_init();

    /* Build resreq AST: ncpus >= 4
     * SEL_GE(SEL_RESNAME(res_idx=0, RES_TYPE_NUMERIC), SEL_NUM(4.0)) */
    xce_select_node_t* resname = makeSelectNode(SEL_RESNAME, 0,
                                                RES_TYPE_NUMERIC);
    xce_select_node_t* num4    = makeSelectNode(SEL_NUM,
                                                (uintptr_t)(double)4.0, 0);
    xce_select_node_t* ge      = makeSelectNode(SEL_GE,
                                                (uintptr_t)resname,
                                                (uintptr_t)num4);
    TEST_ASSERT_NOT_NULL(ge);

    xce_select_t sel = { .root = ge };

    /* --- host with 8 cpus --- */
    xce_sched_worker_t w8;
    memset(&w8, 0, sizeof(w8));
    zarray_init(&w8.resvals, 2, NULL);

    xce_val_t v8;
    memset(&v8, 0, sizeof(v8));
    v8.which    = Value_u_fvalue;
    v8.u.fvalue = 8.0f;
    xce_val_t* v8p = &v8;
    zarray_set(&w8.resvals, 0, &v8p);

    xce_scheduler_t sched;
    memset(&sched, 0, sizeof(sched));

    TEST_ASSERT_EQUAL_INT(1, sched_meet_host_selection(&sched, &w8, &sel));

    /* --- host with 2 cpus --- */
    xce_sched_worker_t w2;
    memset(&w2, 0, sizeof(w2));
    zarray_init(&w2.resvals, 2, NULL);

    xce_val_t v2;
    memset(&v2, 0, sizeof(v2));
    v2.which    = Value_u_fvalue;
    v2.u.fvalue = 2.0f;
    xce_val_t* v2p = &v2;
    zarray_set(&w2.resvals, 0, &v2p);

    TEST_ASSERT_EQUAL_INT(0, sched_meet_host_selection(&sched, &w2, &sel));

    zarray_fini(&w8.resvals);
    zarray_fini(&w2.resvals);
    gc_clear_marks();
    gc_sweep_select_nodes(NULL, 0);
    sel_node_table_fini();
    sel_optimize_test_reset();
}

static void
test_meet_host_selection_string_match(void)
{
    /* Job requires: hosttype == "gpu".
     * Worker has hosttype="gpu" at res_idx=1 → must match.
     * Worker has hosttype="cpu" at res_idx=1 → must not match. */

    sel_optimize_set(1);
    xstring_table_init(64 * 1024);
    sel_node_table_init();

    /* Build resreq AST: hosttype == "gpu" */
    xce_select_node_t* resname = makeSelectNode(SEL_RESNAME, 1,
                                                RES_TYPE_STRING);
    const char* gpu_str = xstring_get("gpu", 3);
    xce_select_node_t* str_gpu = makeSelectNode(SEL_STR,
                                                (uintptr_t)gpu_str, 0);
    xce_select_node_t* eq = makeSelectNode(SEL_EQ,
                                           (uintptr_t)resname,
                                           (uintptr_t)str_gpu);
    TEST_ASSERT_NOT_NULL(eq);

    xce_select_t sel = { .root = eq };

    /* helper to build worker with string at res_idx=1 */
    xce_scheduler_t sched;
    memset(&sched, 0, sizeof(sched));

    xce_sched_worker_t wgpu;
    memset(&wgpu, 0, sizeof(wgpu));
    zarray_init(&wgpu.resvals, 2, NULL);

    xce_val_t vgpu;
    memset(&vgpu, 0, sizeof(vgpu));
    vgpu.which    = Value_u_svalue;
    vgpu.u.svalue = "gpu";
    xce_val_t* vgpup = &vgpu;
    zarray_set(&wgpu.resvals, 1, &vgpup);

    TEST_ASSERT_EQUAL_INT(1, sched_meet_host_selection(&sched, &wgpu, &sel));

    xce_sched_worker_t wcpu;
    memset(&wcpu, 0, sizeof(wcpu));
    zarray_init(&wcpu.resvals, 2, NULL);

    xce_val_t vcpu;
    memset(&vcpu, 0, sizeof(vcpu));
    vcpu.which    = Value_u_svalue;
    vcpu.u.svalue = "cpu";
    xce_val_t* vcpup = &vcpu;
    zarray_set(&wcpu.resvals, 1, &vcpup);

    TEST_ASSERT_EQUAL_INT(0, sched_meet_host_selection(&sched, &wcpu, &sel));

    zarray_fini(&wgpu.resvals);
    zarray_fini(&wcpu.resvals);
    gc_clear_marks();
    gc_sweep_select_nodes(NULL, 0);
    sel_node_table_fini();
    sel_optimize_test_reset();
}
```

In `main`, register:
```c
    TEST(test_meet_host_selection_null_select_passes);
    TEST(test_meet_host_selection_always_true_passes);
    TEST(test_meet_host_selection_always_false_fails);
    TEST(test_build_host_eval_ctx_float);
    TEST(test_meet_host_selection_numeric_match);
    TEST(test_meet_host_selection_string_match);
    TEST(test_scheduler_t_has_atomic_gc_fields);
    TEST(test_gc_predicates_idle_invariant);
    TEST(test_sel_optimize_level_set_at_init);
```

- [x] **Step 3: Check xce_val_t field names**

```bash
grep -n 'svalue\|fvalue\|bvalue\|xce_val_t' src/include/xce/types.h | head -20
```

Adjust `v.u.fvalue`, `v.u.svalue`, `v.u.bvalue` in the tests if the generated capnp struct uses different field names.

- [x] **Step 4: Build and run all scheduler tests**

```bash
nix develop --command bash -c "cd /workspace/src && ninja -C build && build/scheduler/testsuite/sched_test"
```

Expected: all tests PASS.

- [x] **Step 5: Run full build + lib testsuite to verify no regressions**

```bash
nix develop --command bash -c "cd /workspace/src && ninja -C build && build/lib/testsuite/xce_test"
```

Expected: all tests PASS.

- [x] **Step 6: Commit**

```bash
git add src/scheduler/testsuite/meson.build src/scheduler/testsuite/test.c
git commit -m "test: scheduler integration tests for sched_meet_host_selection (6-11)"
```

---

## Self-Review Checklist

| Spec item | Task |
|-----------|------|
| 6-1 pre-audit | Task 1 |
| 6-2 blocking gate grep | Task 1 |
| 6-3 _Atomic fields | Task 3 |
| 6-4 GC predicate wrappers (real scheduler) | Task 4 |
| 6-5 implement sched_meet_host_selection | Task 6 |
| 6-6 (deferred — void return + session restructure) | — |
| 6-7 build_host_eval_ctx | Task 5 |
| 6-8 live_jobs flat array + mark phase | Task 7 |
| 6-9 sel_optimize_set(1) | Task 4 |
| 6-10 sel_alloc_thread_id | Task 8 |
| 6-11 integration tests | Task 9 |
| 6-12 atomic_thread_fence | Task 7 |

### Post-review fixes (final code review, 2026-04-30)

| # | Severity | Finding | Fix | Commit |
|---|----------|---------|-----|--------|
| R1 | **CRITICAL** | `staging_merge_complete` bootstrap deadlock — `sched_begin_session` reset it to 0 but never set it back to 1; `sched_end_session`'s guard `(!staging_merge_complete)` always fired → GC never ran, select-node memory leaked indefinitely | Added `atomic_store_explicit(&sched->staging_merge_complete, 1, memory_order_release)` at end of `sched_begin_session` after `sched_account_jobs` | `d438742` |
| R2 | **HIGH** | `recompact_node_ids()` missing from GC protocol in `sched_end_session` — without recompaction `sel_node_next_id` grows monotonically; `select_eval_ctx_t` arrays (sized by max node_id) grow without bound across GC cycles | Added `recompact_node_ids()` call after `gc_sweep_select_nodes(NULL, 0)` | `d438742` |

### Known deferred items

- **6-6** (outer-host/inner-job session restructuring + `void` return): The current implementation uses `eval_select_simple` which re-evaluates the expression for every (host, job) call independently. The design doc describes a future optimisation where a single `select_eval_ctx_t` is reused per host across all jobs in a session. Deferred to a follow-on plan.

- **`eval_in_progress` guard**: The `sched_eval_in_progress` check in `sched_end_session` is always false (field never set to 1). Correct under the current single-threaded model; wiring is deferred to when concurrent eval is introduced.