Rongsong Shen 177cf45af5 docs: mark Phase 6 complete in phase6-scheduler-integration-plan

2026-04-30 07:17:32 +00:00

41 KiB

Raw Blame History

Phase 6 — Scheduler Integration Implementation Plan

Status: ✓ COMPLETE — 2026-04-30 — 10/10 scheduler tests pass | branch: merge_api_server

Goal: Wire the select-expression AST evaluator (eval_select_simple) into the scheduler so that sched_meet_host_selection filters hosts by actual job select expressions instead of always returning 1.

Architecture: Export GC mark primitives from libxce to break the circular dependency; add _Atomic GC-predicate fields to xce_scheduler_t; implement build_host_eval_ctx to project xce_sched_worker_t.resvals into host_eval_ctx_t; call eval_select_simple in the stub replacement; drive the full mark→sweep GC from sched_end_session.

Tech Stack: C99, Unity test framework, Meson/Ninja build, nix develop environment, libxce (src/lib), scheduler (src/scheduler)

File Map

File	Change
`src/lib/xce_select.c`	Export `gc_mark_select_tree`; add `gc_clear_marks`; remove clear from `gc_sweep`; remove Phase 6 guard; add `sel_alloc_thread_id` + thread checks
`src/include/resreq.h`	Declare `gc_mark_select_tree`, `gc_clear_marks`, `sel_register_alloc_thread`
`src/lib/testsuite/test_select.c`	Prefix every `gc_sweep_select_nodes(NULL,0)` call with `gc_clear_marks()`
`src/scheduler/include/scheduler.h`	Add `_Atomic int session_active/eval_in_progress/staging_merge_complete` to `xce_scheduler_t`; declare `sched_session_active/sched_eval_in_progress/staging_merge_complete`
`src/scheduler/src/scheduler.c`	Add predicate static helpers; wire `sel_optimize_set(1)` in `sched_init`; drive GC from `sched_end_session` with atomic fence
`src/scheduler/src/resource.c`	Add `build_host_eval_ctx`; replace `sched_meet_host_selection` stub
`src/scheduler/testsuite/meson.build`	Link `resource.c` + `scheduler.c` for integration tests
`src/scheduler/testsuite/test.c`	Add `test_meet_host_selection` integration test

Task 1: Pre-audit — confirm single-threaded invariant and stub locations (6-1, 6-2)

Files:

Read: src/scheduler/src/resource.c
Read: src/scheduler/src/main.c
Step 1: Run the blocking-gate grep

cd /workspace
grep -rn 'sched_meet_host_selection\|meet_host_selection' src/scheduler/ src/lib/
grep -rn 'eval_select\|alloc_thread\|sched_session' src/scheduler/src/

Expected: sched_meet_host_selection appears in exactly 3 locations — scheduler.h (decl), resource.c (stub def), resource.c (call site at sched_try_alloc_on_host_). No other call sites.

Step 2: Confirm allocator-thread-only invariant

grep -n 'sched_begin_session\|sched_end_session\|sched_go' src/scheduler/src/main.c

Expected: all three called from the same while(1) loop in main. Document result: "confirmed single-threaded allocator loop in main.c lines ~119-168".

Step 3: Confirm eval_select_simple signature

grep -n 'eval_select_simple' src/include/resreq.h src/lib/xce_select.c

Expected: int eval_select_simple(const xce_select_node_t* root, const host_eval_ctx_t* host) — returns 1 (match), 0 (no match), negative (error).

Step 4: Commit audit result

git add -p   # no code changes
git commit -m "chore: phase 6 pre-audit — allocator single-threaded confirmed, stub at resource.c:sched_meet_host_selection"

Task 2: Export GC mark primitives — split gc_sweep_select_nodes (6-8 prerequisite)

The circular dependency between xce_select.c (libxce) and sched/types.h (scheduler) means the GC mark pass cannot live inside gc_sweep_select_nodes. Solution: export gc_clear_marks and gc_mark_select_tree so the scheduler owns the mark phase.

gc_sweep_select_nodes currently clears marks internally before sweeping. After this task it only sweeps — callers must call gc_clear_marks() first. The Phase 6 guard (if (live_jobs != NULL && njobs > 0) return;) is removed.

Files:

Modify: src/lib/xce_select.c
Modify: src/include/resreq.h
Modify: src/lib/testsuite/test_select.c
Step 1: Write the failing test

In src/lib/testsuite/test_select.c, add after the existing GC tests (around line 800):

static void
test_gc_clear_marks_exported(void)
{
    /* After this call, all nodes in the intern table must have gc_mark == 0.
     * We verify by immediately calling gc_sweep (which only sweeps unmarked
     * nodes), expecting no crash and consistent node count. */
    sel_optimize_test_reset();
    sel_optimize_set(1);
    xstring_table_init(64 * 1024);
    sel_node_table_init();

    /* intern two nodes */
    xce_select_node_t* n1 = makeSelectNode(SEL_NUM, (uintptr_t)1.0, 0);
    xce_select_node_t* n2 = makeSelectNode(SEL_NUM, (uintptr_t)2.0, 0);
    TEST_ASSERT_NOT_NULL(n1);
    TEST_ASSERT_NOT_NULL(n2);

    /* mark n1 manually */
    n1->gc_mark = 1;

    /* clear should reset n1->gc_mark */
    gc_clear_marks();
    TEST_ASSERT_EQUAL_INT(0, n1->gc_mark);

    /* mark n2 so sweep preserves it; n1 (unmarked) is swept */
    gc_mark_select_tree(n2);
    TEST_ASSERT_EQUAL_INT(1, n2->gc_mark);

    gc_sweep_select_nodes(NULL, 0);
    /* n1 swept — n2 still alive */
    TEST_ASSERT_EQUAL_INT(1, n2->gc_mark);  /* mark survives sweep */

    sel_node_table_fini();
    sel_optimize_test_reset();
}

In the main function of test_select.c, register the test:

TEST(test_gc_clear_marks_exported);

Step 2: Run test to verify it fails

nix develop --command bash -c "cd /workspace/src && meson setup build --wipe 2>/dev/null || true && ninja -C build lib/testsuite/xce_test 2>&1 | tail -30"

Expected: compile error — gc_clear_marks and gc_mark_select_tree undeclared.

Step 3: Add declarations to resreq.h

In src/include/resreq.h, after the gc_sweep_select_nodes declaration (line ~121):

/* GC mark primitives — called by the scheduler to implement the mark phase.
 * Call order: gc_clear_marks() → gc_mark_select_tree(root) per live node
 *             → gc_sweep_select_nodes(NULL, 0) */
extern void gc_clear_marks(void);
extern void gc_mark_select_tree(xce_select_node_t* node);

Step 4: Implement gc_clear_marks and gc_mark_select_tree in xce_select.c

In src/lib/xce_select.c, locate the gc_mark_tree function at line ~805:

static void __attribute__((unused)) gc_mark_tree(xce_select_node_t* node)

Replace with (non-static, renamed):

void gc_mark_select_tree(xce_select_node_t* node)
{
    if (node == NULL || node->gc_mark) return;
    node->gc_mark = 1;
    switch (node->type) {
    case SEL_RESNAME: case SEL_NUM: case SEL_STR:
    case SEL_ANY:     case SEL_TRUE: case SEL_FALSE:
        break;
    case SEL_NOT:
        gc_mark_select_tree(node->u.operand);
        break;
    default:
        gc_mark_select_tree(node->u.binop.left);
        gc_mark_select_tree(node->u.binop.right);
        break;
    }
}

Above that, add gc_clear_marks:

void gc_clear_marks(void)
{
    if (sel_node_table.table == NULL) return;
    int caps = sel_node_table.nodes.caps;
    for (int i = 0; i < caps; i++) {
        xce_select_node_t* node =
            (xce_select_node_t*)zarray_get(&sel_node_table.nodes, i);
        if (node == NULL) continue;
        node->gc_mark = 0;
    }
}

Step 5: Modify gc_sweep_select_nodes — remove clear loop and Phase 6 guard

In src/lib/xce_select.c, in gc_sweep_select_nodes (line ~826):

Remove the Phase 6 guard block:

    if (live_jobs != NULL && njobs > 0) {
        return;
    }

Remove the clear loop (step 1, lines ~841-852):

    /* 1. Clear all gc_mark bits ... */
    for (int i = 0; i < caps; i++) {
        xce_select_node_t* node =
            (xce_select_node_t*)zarray_get(&sel_node_table.nodes, i);
        if (node == NULL) continue;
        node->gc_mark = 0;
    }

The function now starts directly with the int caps = sel_node_table.nodes.caps; line and the sweep loop.

Step 6: Update test_select.c — prefix all gc_sweep_select_nodes(NULL,0) calls

grep -n 'gc_sweep_select_nodes(NULL, 0)' src/lib/testsuite/test_select.c

For every occurrence, prepend gc_clear_marks(); on the line before. There should be 4 occurrences. Example — each:

/* before */
gc_sweep_select_nodes(NULL, 0);

/* after */
gc_clear_marks();
gc_sweep_select_nodes(NULL, 0);

Step 7: Build and run tests

nix develop --command bash -c "cd /workspace/src && ninja -C build && build/lib/testsuite/xce_test -t test_gc_clear_marks_exported"

Expected: test_gc_clear_marks_exported PASS. Run full suite:

nix develop --command bash -c "cd /workspace/src && build/lib/testsuite/xce_test"

Expected: all GC tests PASS.

Step 8: Commit

git add src/lib/xce_select.c src/include/resreq.h src/lib/testsuite/test_select.c
git commit -m "feat: export gc_clear_marks + gc_mark_select_tree; remove clear from gc_sweep"

Task 3: Add _Atomic GC-predicate fields to xce_scheduler_t (6-3)

Files:

Modify: src/scheduler/include/scheduler.h
Step 1: Write the failing test

In src/scheduler/testsuite/test.c, add:

#include <stdatomic.h>
#include "scheduler.h"

static void
test_scheduler_t_has_atomic_gc_fields(void)
{
    xce_scheduler_t sched;
    memset(&sched, 0, sizeof(sched));

    /* Verify atomic fields exist and are zero-initialized */
    TEST_ASSERT_EQUAL_INT(0, atomic_load(&sched.session_active));
    TEST_ASSERT_EQUAL_INT(0, atomic_load(&sched.eval_in_progress));
    TEST_ASSERT_EQUAL_INT(0, atomic_load(&sched.staging_merge_complete));
}

In main, add: TEST(test_scheduler_t_has_atomic_gc_fields);

Step 2: Run to verify it fails

nix develop --command bash -c "cd /workspace/src && ninja -C build scheduler/testsuite/sched_test 2>&1 | tail -20"

Expected: compile error — session_active not a member of xce_scheduler_t.

Step 3: Add fields to xce_scheduler_t

In src/scheduler/include/scheduler.h, add three fields at the end of xce_scheduler_t (before the closing }, after cbuf_t decisions;):

    _Atomic int session_active;         /* 1 during scheduling session */
    _Atomic int eval_in_progress;       /* 1 while eval_select loop runs */
    _Atomic int staging_merge_complete; /* 1 after staging merge completes */

Also add #include <stdatomic.h> near the top of scheduler.h (after #include <time.h>).

Step 4: Build and run

nix develop --command bash -c "cd /workspace/src && ninja -C build && build/scheduler/testsuite/sched_test"

Expected: test_scheduler_t_has_atomic_gc_fields PASS.

Step 5: Commit

git add src/scheduler/include/scheduler.h src/scheduler/testsuite/test.c
git commit -m "feat: add _Atomic gc-predicate fields to xce_scheduler_t (6-3)"

Task 4: Add GC-predicate statics in scheduler.c + wire sel_optimize_set (6-4, 6-9)

Files:

Modify: src/scheduler/src/scheduler.c
Modify: src/scheduler/include/scheduler.h

The predicates are scheduler-internal — static helpers in scheduler.c. They are called at the GC call site (Task 7). The fake_scheduler_t stubs in test_select.c remain (unit tests still use them).

Step 1: Write failing test

In src/scheduler/testsuite/test.c, add:

static void
test_sel_optimize_level_set_at_init(void)
{
    /* After sched_init, sel_get_optimize_level() must return 1. */
    /* We can check it directly since it's a library call. */
    /* We can't call sched_init (it starts threads) so check indirectly:
     * the field is set by sel_optimize_set — test the setter directly. */
    sel_optimize_set(1);
    TEST_ASSERT_EQUAL_INT(1, sel_get_optimize_level());
    sel_optimize_set(0);  /* reset for other tests */
}

Add #include "resreq.h" to test includes.

In main, add: TEST(test_sel_optimize_level_set_at_init);

Step 2: Run to verify it fails

nix develop --command bash -c "cd /workspace/src && ninja -C build scheduler/testsuite/sched_test 2>&1 | tail -20"

Expected: compile error — sel_optimize_set undeclared (not linked yet — Task 8 will fix meson.build; for now just verify the include path works).

Note: If it fails on sel_optimize_set not linked, proceed to Step 3 and come back.

Step 3: Add static predicate helpers to scheduler.c

In src/scheduler/src/scheduler.c, after the #include block (around line 20), add:

#include "resreq.h"

static int sched_session_active(xce_scheduler_t* sched)
{
    return atomic_load_explicit(&sched->session_active, memory_order_acquire);
}

static int sched_eval_in_progress(xce_scheduler_t* sched)
{
    return atomic_load_explicit(&sched->eval_in_progress, memory_order_acquire);
}

static int sched_staging_merge_complete(xce_scheduler_t* sched)
{
    return atomic_load_explicit(&sched->staging_merge_complete,
                                memory_order_acquire);
}

Step 4: Wire sel_optimize_set(1) in sched_init

In src/scheduler/src/scheduler.c, in sched_init (line ~318), after memset(sched, 0x0, sizeof(*sched)); (line ~357), add:

    sel_optimize_set(1);

This must come before any makeSelectNode calls (which happen during job parsing). The write-once guard in sel_optimize_set ensures it can only be set once per process.

Step 5: Build

nix develop --command bash -c "cd /workspace/src && ninja -C build 2>&1 | grep -E 'error:|warning:|FAILED|OK' | head -20"

Expected: builds cleanly (the test executable link will be fixed in Task 9 meson update).

Step 6: Commit

git add src/scheduler/src/scheduler.c
git commit -m "feat: add sched GC predicate statics; wire sel_optimize_set(1) in sched_init (6-4, 6-9)"

Task 5: Implement build_host_eval_ctx (6-7)

build_host_eval_ctx flattens xce_sched_worker_t.resvals (a zarray_t of xce_val_t* indexed by res_idx) into the numvals/strvals arrays expected by host_eval_ctx_t.

Capnp constants (from controller.capnp.h): Value_u_fvalue = 5 (float), Value_u_svalue = 4 (string), Value_u_bvalue = 6 (bool → stored as 0.0/1.0 in numvals).

Files:

Modify: src/scheduler/src/resource.c
Step 1: Write the failing test

In src/scheduler/testsuite/test.c, add:

#include "sched/types.h"
#include "resreq.h"
#include "zarray.h"
#include "controller.capnp.h"

/* build_host_eval_ctx is static — declare prototype for test access */
static host_eval_ctx_t* build_host_eval_ctx(const xce_sched_worker_t* w,
                                             int num_res);

static void
test_build_host_eval_ctx_float(void)
{
    /* Build a worker with one float resource at res_idx=2 (value=4.0) */
    xce_sched_worker_t w;
    memset(&w, 0, sizeof(w));
    zarray_init(&w.resvals, 8, NULL);

    xce_val_t v;
    memset(&v, 0, sizeof(v));
    v.which      = Value_u_fvalue;
    v.u.fvalue   = 4.0f;
    xce_val_t* vp = &v;

    /* slot 0, 1 = NULL; slot 2 = &v */
    zarray_set(&w.resvals, 0, NULL);
    zarray_set(&w.resvals, 1, NULL);
    zarray_set(&w.resvals, 2, &vp);

    host_eval_ctx_t* ctx = build_host_eval_ctx(&w, 3);
    TEST_ASSERT_NOT_NULL(ctx);
    TEST_ASSERT_EQUAL_DOUBLE(4.0, ctx->numvals[2]);
    TEST_ASSERT_NULL(ctx->strvals[2]);

    free((void*)ctx->numvals);
    free((void*)ctx->strvals);
    free(ctx);
    zarray_fini(&w.resvals);
}

In main, add: TEST(test_build_host_eval_ctx_float);

Step 2: Run to verify it fails

nix develop --command bash -c "cd /workspace/src && ninja -C build scheduler/testsuite/sched_test 2>&1 | tail -20"

Expected: compile error — build_host_eval_ctx not yet defined.

Step 3: Implement build_host_eval_ctx in resource.c

In src/scheduler/src/resource.c, add #include "controller.capnp.h" to the include block, then add before sched_meet_host_selection:

/* Build a host_eval_ctx_t from a scheduler worker's resource-value zarray.
 * Allocates numvals and strvals on the heap; caller frees all three.
 * Returns NULL on OOM. */
static host_eval_ctx_t*
build_host_eval_ctx(const xce_sched_worker_t* w, int num_res)
{
    host_eval_ctx_t* ctx = calloc(1, sizeof(*ctx));
    if (ctx == NULL) return NULL;

    double*      numvals = calloc(num_res, sizeof(double));
    const char** strvals = calloc(num_res, sizeof(char*));
    if (numvals == NULL || strvals == NULL) {
        free(numvals);
        free(strvals);
        free(ctx);
        return NULL;
    }

    for (int i = 0; i < num_res; i++) {
        xce_val_t** slot = (xce_val_t**)zarray_get(&w->resvals, i);
        if (slot == NULL || *slot == NULL) continue;
        xce_val_t* v = *slot;
        switch (v->which) {
        case Value_u_fvalue:
            numvals[i] = (double)v->u.fvalue;
            break;
        case Value_u_bvalue:
            numvals[i] = v->u.bvalue ? 1.0 : 0.0;
            break;
        case Value_u_svalue:
            strvals[i] = v->u.svalue;
            break;
        }
    }

    ctx->num_res = num_res;
    ctx->numvals = numvals;
    ctx->strvals = strvals;
    return ctx;
}

static void
free_host_eval_ctx(host_eval_ctx_t* ctx)
{
    if (ctx == NULL) return;
    free((void*)ctx->numvals);
    free((void*)ctx->strvals);
    free(ctx);
}

Step 4: Check xce_val_t layout

grep -n 'xce_val_t\|fvalue\|svalue\|bvalue' src/include/xce/types.h | head -20
grep -n 'Value_u_fvalue\|Value_u_svalue\|Value_u_bvalue' src/scheduler/src/resource.c || \
  grep -rn 'Value_u_fvalue' src/controller/ | head -3

Verify the u.fvalue, u.svalue, u.bvalue field names match xce_val_t in controller.capnp.h.

Step 5: Build and run test

nix develop --command bash -c "cd /workspace/src && ninja -C build && build/scheduler/testsuite/sched_test -t test_build_host_eval_ctx_float"

Expected: PASS.

Step 6: Commit

git add src/scheduler/src/resource.c
git commit -m "feat: implement build_host_eval_ctx — project resvals zarray to host_eval_ctx_t (6-7)"

Task 6: Implement sched_meet_host_selection (6-5)

Replace the stub with eval_select_simple. This is the pragmatic Phase 6 approach (no session restructuring — deferred to a later phase). The int return value and add_job_preason call in the caller remain unchanged.

Files:

Modify: src/scheduler/src/resource.c
Step 1: Write the failing test

In src/scheduler/testsuite/test.c, add:

#include "kvmgr.h"

static void
test_meet_host_selection_null_select_passes(void)
{
    /* A NULL select pointer means no expression — always pass */
    xce_scheduler_t sched;
    memset(&sched, 0, sizeof(sched));
    xce_sched_worker_t w;
    memset(&w, 0, sizeof(w));
    zarray_init(&w.resvals, 4, NULL);

    int rc = sched_meet_host_selection(&sched, &w, NULL);
    TEST_ASSERT_EQUAL_INT(1, rc);

    zarray_fini(&w.resvals);
}

static void
test_meet_host_selection_always_true_passes(void)
{
    xce_scheduler_t sched;
    memset(&sched, 0, sizeof(sched));
    sel_optimize_set(1);
    sel_node_table_init();

    xce_sched_worker_t w;
    memset(&w, 0, sizeof(w));
    zarray_init(&w.resvals, 4, NULL);

    /* SEL_TRUE root — must pass any host */
    xce_select_node_t* root = makeSelectNode(SEL_TRUE, 0, 0);
    TEST_ASSERT_NOT_NULL(root);

    xce_select_t sel = { .root = root };
    int rc = sched_meet_host_selection(&sched, &w, &sel);
    TEST_ASSERT_EQUAL_INT(1, rc);

    zarray_fini(&w.resvals);
    gc_clear_marks();
    gc_sweep_select_nodes(NULL, 0);
    sel_node_table_fini();
    sel_optimize_test_reset();
}

static void
test_meet_host_selection_always_false_fails(void)
{
    xce_scheduler_t sched;
    memset(&sched, 0, sizeof(sched));
    sel_optimize_set(1);
    sel_node_table_init();

    xce_sched_worker_t w;
    memset(&w, 0, sizeof(w));
    zarray_init(&w.resvals, 4, NULL);

    /* SEL_FALSE root — must fail every host */
    xce_select_node_t* root = makeSelectNode(SEL_FALSE, 0, 0);
    TEST_ASSERT_NOT_NULL(root);

    xce_select_t sel = { .root = root };
    int rc = sched_meet_host_selection(&sched, &w, &sel);
    TEST_ASSERT_EQUAL_INT(0, rc);

    zarray_fini(&w.resvals);
    gc_clear_marks();
    gc_sweep_select_nodes(NULL, 0);
    sel_node_table_fini();
    sel_optimize_test_reset();
}

In main, register all three tests.

Step 2: Run to verify failures

nix develop --command bash -c "cd /workspace/src && ninja -C build scheduler/testsuite/sched_test 2>&1 | tail -20"

Expected: link errors (scheduler sources not yet in meson) OR test_meet_host_selection_always_false_fails FAIL (stub returns 1 for SEL_FALSE).

Proceed to Step 3.

Step 3: Replace the stub in resource.c

In src/scheduler/src/resource.c, replace the sched_meet_host_selection function body (currently lines ~415-422):

int sched_meet_host_selection(xce_scheduler_t* sched, xce_sched_worker_t* w,
                               xce_select_t* s)
{
    if (s == NULL || s->root == NULL) return 1;

    /* Count resources: use resvals.caps as upper bound on res_idx */
    int num_res = (int)w->resvals.caps;

    host_eval_ctx_t* hctx = build_host_eval_ctx(w, num_res);
    if (hctx == NULL) {
        /* OOM: conservatively pass host — scheduler will reject on resource
         * quantities anyway */
        return 1;
    }

    int rc = eval_select_simple(s->root, hctx);
    free_host_eval_ctx(hctx);

    /* eval_select_simple: 1=match, 0=no-match, <0=error (treat as no-match) */
    return (rc > 0) ? 1 : 0;
}

Add #include "resreq.h" to resource.c includes if not already present.

Step 4: Build and run

nix develop --command bash -c "cd /workspace/src && ninja -C build 2>&1 | grep -E 'error:|FAILED' | head -10"

Expected: builds cleanly.

Step 5: Commit

git add src/scheduler/src/resource.c
git commit -m "feat: implement sched_meet_host_selection using eval_select_simple (6-5)"

Task 7: Drive GC from sched_end_session with live_jobs + atomic fence (6-8, 6-4, 6-12)

sched_end_session collects all live jobs, performs the mark phase, inserts the seq_cst fence, then sweeps. The GC idle invariant (6-4) is enforced by the three predicate guards.

Files:

Modify: src/scheduler/src/scheduler.c
Step 1: Write the failing test

In src/scheduler/testsuite/test.c, add:

static void
test_gc_predicates_idle_invariant(void)
{
    /* session_active=0, eval_in_progress=0, staging_merge_complete=0 after
     * memset — GC call site guard must NOT log_fatal. We can only test
     * indirectly by calling sched_end_session on an empty scheduler. */
    xce_scheduler_t sched;
    memset(&sched, 0, sizeof(sched));
    /* staging_merge_complete must be 1 for GC to proceed */
    atomic_store(&sched.staging_merge_complete, 1);

    sel_optimize_set(1);
    sel_node_table_init();

    /* sched_end_session with no jobs — should call gc with empty live_jobs */
    /* We can't call sched_end_session directly (it calls plugin_end_session
     * which requires full init). Test the predicate reads directly. */
    TEST_ASSERT_EQUAL_INT(0, atomic_load(&sched.session_active));
    TEST_ASSERT_EQUAL_INT(0, atomic_load(&sched.eval_in_progress));
    TEST_ASSERT_EQUAL_INT(1, atomic_load(&sched.staging_merge_complete));

    gc_clear_marks();
    gc_sweep_select_nodes(NULL, 0);

    sel_node_table_fini();
    sel_optimize_test_reset();
}

In main, add: TEST(test_gc_predicates_idle_invariant);

Step 2: Run to verify

nix develop --command bash -c "cd /workspace/src && ninja -C build scheduler/testsuite/sched_test 2>&1 | tail -20"

Step 3: Add live_jobs collection and GC call to sched_end_session

In src/scheduler/src/scheduler.c, replace sched_end_session (line ~752):

void sched_end_session(xce_scheduler_t* sched)
{
    plugin_end_session(sched);

    /* GC idle invariant guard — any violation is a use-after-free risk */
    if (sched_session_active(sched) || sched_eval_in_progress(sched)
        || !sched_staging_merge_complete(sched)) {
        log_fatal("sched_end_session: GC idle invariant violated: "
                  "session_active=%d eval_in_progress=%d staging_merge=%d",
                  sched_session_active(sched),
                  sched_eval_in_progress(sched),
                  sched_staging_merge_complete(sched));
        return;
    }

    /* Collect live jobs into a flat array for GC mark phase */
    rtti_iter_t      it;
    xce_sched_job_t* job       = NULL;
    int              njobs     = 0;
    int              cap       = 64;
    xce_sched_job_t** live_jobs = calloc(cap, sizeof(*live_jobs));
    if (live_jobs == NULL) goto sweep;

    iter_init(&it);
    lock(sched->jobs, RTTI_LOCK_READ);
    while (iter_next(&it, sched->jobs, (void**)&job) == RTTI_ITER_OK) {
        if (njobs >= cap) {
            int newcap = cap * 2;
            xce_sched_job_t** tmp = realloc(live_jobs,
                                            newcap * sizeof(*live_jobs));
            if (tmp == NULL) {
                unlock(sched->jobs);
                iter_destroy(&it);
                goto sweep_with_jobs;
            }
            live_jobs = tmp;
            cap       = newcap;
        }
        live_jobs[njobs++] = job;
    }
    unlock(sched->jobs);
    iter_destroy(&it);

sweep_with_jobs:
    /* Mark phase: clear all gc_marks, then mark every select root reachable
     * from live job resreq trees */
    gc_clear_marks();
    for (int i = 0; i < njobs; i++) {
        xce_resreq_t* rr = live_jobs[i]->resreq;
        if (rr == NULL) continue;
        /* dispatch on resreq tag to reach all select roots */
        if (rr->tag == XCE_RESREQ_PRIMARY) {
            if (rr->u.primary && rr->u.primary->select)
                gc_mark_select_tree(rr->u.primary->select->root);
        } else if (rr->tag == XCE_RESREQ_COMPOUND) {
            if (rr->u.compound) {
                xlist_iter_t li;
                xce_primary_resreq_t* pr;
                xlist_for_each(&rr->u.compound->primaries, &li,
                               (void**)&pr) {
                    if (pr && pr->select)
                        gc_mark_select_tree(pr->select->root);
                }
            }
        } else if (rr->tag == XCE_RESREQ_ALTER) {
            if (rr->u.alter) {
                xlist_iter_t li;
                xce_compound_resreq_t* cp;
                xlist_for_each(&rr->u.alter->compounds, &li,
                               (void**)&cp) {
                    if (cp == NULL) continue;
                    xlist_iter_t li2;
                    xce_primary_resreq_t* pr;
                    xlist_for_each(&cp->primaries, &li2, (void**)&pr) {
                        if (pr && pr->select)
                            gc_mark_select_tree(pr->select->root);
                    }
                }
            }
        }
    }

    /* Fence: ensure all gc_mark writes are visible before the sweep reads
     * them.  Also ensures NULL-stores to unlinked roots (if any) are visible
     * before gc_mark_select_tree reads root pointers. */
    atomic_thread_fence(memory_order_seq_cst);

sweep:
    gc_sweep_select_nodes(NULL, 0);
    free(live_jobs);
}

Add #include <stdatomic.h> to scheduler.c includes, and ensure #include "resreq.h" is present.

Check xlist_for_each macro — if it doesn't exist, use xlist_iter_t loop pattern from existing code in the file:

grep -n 'xlist_for_each\|xlist_iter' src/scheduler/src/scheduler.c | head -5

If xlist_for_each is not available, replace with the pattern used elsewhere in scheduler.c.

Step 4: Set/clear session_active and staging_merge_complete at session boundaries

In sched_begin_session (line ~740), at the START:

    atomic_store_explicit(&sched->session_active, 1, memory_order_release);
    atomic_store_explicit(&sched->staging_merge_complete, 0, memory_order_release);

In sched_end_session, at the END (after the GC sweep), add:

    atomic_store_explicit(&sched->staging_merge_complete, 1, memory_order_release);

And in sched_end_session at the VERY START (before the guard), add:

    atomic_store_explicit(&sched->session_active, 0, memory_order_release);

Step 5: Build

nix develop --command bash -c "cd /workspace/src && ninja -C build 2>&1 | grep -E 'error:|FAILED' | head -20"

Expected: builds cleanly.

Step 6: Commit

git add src/scheduler/src/scheduler.c
git commit -m "feat: drive GC from sched_end_session — live_jobs mark, seq_cst fence, gc_sweep (6-8, 6-12)"

Task 8: Add sel_alloc_thread_id thread-ownership checks (6-10)

Files:

Modify: src/lib/xce_select.c
Modify: src/include/resreq.h
Modify: src/scheduler/src/scheduler.c (register allocator thread)
Step 1: Write the failing test

In src/lib/testsuite/test_select.c, add:

static void
test_sel_register_alloc_thread(void)
{
    /* sel_alloc_thread_id is 0 (unregistered) before registration —
     * thread checks must be skipped.  After registration they must
     * match pthread_self(). */
    sel_optimize_test_reset();   /* resets sel_alloc_thread_id to 0 */
    sel_optimize_set(1);
    sel_node_table_init();

    /* Should not crash when unregistered */
    xce_select_node_t* n = makeSelectNode(SEL_TRUE, 0, 0);
    TEST_ASSERT_NOT_NULL(n);

    /* Register current thread */
    sel_register_alloc_thread();

    /* makeSelectNode from same thread must still work */
    xce_select_node_t* n2 = makeSelectNode(SEL_FALSE, 0, 0);
    TEST_ASSERT_NOT_NULL(n2);

    gc_clear_marks();
    gc_sweep_select_nodes(NULL, 0);
    sel_node_table_fini();
    sel_optimize_test_reset();
}

In main, add: TEST(test_sel_register_alloc_thread);

Step 2: Run to verify it fails

nix develop --command bash -c "cd /workspace/src && ninja -C build lib/testsuite/xce_test 2>&1 | tail -15"

Expected: compile error — sel_register_alloc_thread undeclared.

Step 3: Add sel_alloc_thread_id to xce_select.c

In src/lib/xce_select.c, add #include <pthread.h> to the include block.

After the sel_optimize_level static variable declaration, add:

static pthread_t sel_alloc_thread_id; /* 0 = unregistered, skip checks */

Add the registration function before eval_select:

void sel_register_alloc_thread(void)
{
    sel_alloc_thread_id = pthread_self();
}

Extend sel_optimize_test_reset to also reset sel_alloc_thread_id:

void sel_optimize_test_reset(void)
{
    atomic_store_explicit(&sel_optimize_level, -1, memory_order_seq_cst);
    sel_alloc_thread_id = 0;
}

Step 4: Add thread checks in eval_select and gc_sweep_select_nodes

In eval_select (line ~633), after the sel_get_optimize_level() guard, add:

    if (sel_alloc_thread_id != 0 && pthread_self() != sel_alloc_thread_id) {
        ERROR("eval_select called from non-allocator thread");
        return -1;
    }

In gc_sweep_select_nodes (line ~826), at the top after the table == NULL check, add:

    if (sel_alloc_thread_id != 0 && pthread_self() != sel_alloc_thread_id) {
        ERROR("gc_sweep_select_nodes called from non-allocator thread");
        return;
    }

Step 5: Declare sel_register_alloc_thread in resreq.h

In src/include/resreq.h, after sel_optimize_set/sel_get_optimize_level:

/* Register the current thread as the sole allocator thread.
 * Must be called once before any eval_select or gc_sweep_select_nodes call.
 * A value of 0 (default) disables the check (unit-test mode). */
extern void sel_register_alloc_thread(void);

Step 6: Wire registration in scheduler main loop

In src/scheduler/src/main.c, in the allocator thread's first while(1) iteration (line ~119, before sched_begin_session), add a one-time call:

    /* Register this thread as the sole eval/GC thread (write-once) */
    static int sel_thread_registered = 0;
    if (!sel_thread_registered) {
        sel_register_alloc_thread();
        sel_thread_registered = 1;
    }

Add #include "resreq.h" to main.c includes.

Step 7: Build and run

nix develop --command bash -c "cd /workspace/src && ninja -C build && build/lib/testsuite/xce_test -t test_sel_register_alloc_thread"

Expected: PASS. Run full lib suite:

nix develop --command bash -c "cd /workspace/src && build/lib/testsuite/xce_test"

Expected: all tests PASS.

Step 8: Commit

git add src/lib/xce_select.c src/include/resreq.h src/scheduler/src/main.c
git commit -m "feat: add sel_alloc_thread_id ownership check in eval_select and gc_sweep (6-10)"

Task 9: Scheduler testsuite — link resource.c + integration tests (6-11)

Files:

Modify: src/scheduler/testsuite/meson.build
Modify: src/scheduler/testsuite/test.c
Step 1: Update meson.build to link scheduler source files

Replace src/scheduler/testsuite/meson.build with:

scheduler_test_src = [
  'test.c',
  '../src/resource.c',
]

scheduler_test_deps = [libxce_deps, librtti_dep, unity_dep]

incdir = include_directories('../include', '../../include', '.')

scheduler_test = executable('sched_test',
                            scheduler_test_src,
                            include_directories:[incdir],
                            dependencies: [scheduler_test_deps],
                            install: false)

Note: Only resource.c is needed — it implements the functions under test. scheduler.c (which calls sched_init and starts threads) is intentionally excluded to keep tests free of thread/network side effects.

Step 2: Write integration test with a known-good resreq/host pair

In src/scheduler/testsuite/test.c, add:

#include "xstring.h"

static void
test_meet_host_selection_numeric_match(void)
{
    /* Job requires: ncpus >= 4.
     * Worker has ncpus=8 at res_idx=0 → must match (rc=1).
     * Worker has ncpus=2 at res_idx=0 → must not match (rc=0). */

    sel_optimize_set(1);
    xstring_table_init(64 * 1024);
    sel_node_table_init();

    /* Build resreq AST: ncpus >= 4
     * SEL_GE(SEL_RESNAME(res_idx=0, RES_TYPE_NUMERIC), SEL_NUM(4.0)) */
    xce_select_node_t* resname = makeSelectNode(SEL_RESNAME, 0,
                                                RES_TYPE_NUMERIC);
    xce_select_node_t* num4    = makeSelectNode(SEL_NUM,
                                                (uintptr_t)(double)4.0, 0);
    xce_select_node_t* ge      = makeSelectNode(SEL_GE,
                                                (uintptr_t)resname,
                                                (uintptr_t)num4);
    TEST_ASSERT_NOT_NULL(ge);

    xce_select_t sel = { .root = ge };

    /* --- host with 8 cpus --- */
    xce_sched_worker_t w8;
    memset(&w8, 0, sizeof(w8));
    zarray_init(&w8.resvals, 2, NULL);

    xce_val_t v8;
    memset(&v8, 0, sizeof(v8));
    v8.which    = Value_u_fvalue;
    v8.u.fvalue = 8.0f;
    xce_val_t* v8p = &v8;
    zarray_set(&w8.resvals, 0, &v8p);

    xce_scheduler_t sched;
    memset(&sched, 0, sizeof(sched));

    TEST_ASSERT_EQUAL_INT(1, sched_meet_host_selection(&sched, &w8, &sel));

    /* --- host with 2 cpus --- */
    xce_sched_worker_t w2;
    memset(&w2, 0, sizeof(w2));
    zarray_init(&w2.resvals, 2, NULL);

    xce_val_t v2;
    memset(&v2, 0, sizeof(v2));
    v2.which    = Value_u_fvalue;
    v2.u.fvalue = 2.0f;
    xce_val_t* v2p = &v2;
    zarray_set(&w2.resvals, 0, &v2p);

    TEST_ASSERT_EQUAL_INT(0, sched_meet_host_selection(&sched, &w2, &sel));

    zarray_fini(&w8.resvals);
    zarray_fini(&w2.resvals);
    gc_clear_marks();
    gc_sweep_select_nodes(NULL, 0);
    sel_node_table_fini();
    sel_optimize_test_reset();
}

static void
test_meet_host_selection_string_match(void)
{
    /* Job requires: hosttype == "gpu".
     * Worker has hosttype="gpu" at res_idx=1 → must match.
     * Worker has hosttype="cpu" at res_idx=1 → must not match. */

    sel_optimize_set(1);
    xstring_table_init(64 * 1024);
    sel_node_table_init();

    /* Build resreq AST: hosttype == "gpu" */
    xce_select_node_t* resname = makeSelectNode(SEL_RESNAME, 1,
                                                RES_TYPE_STRING);
    const char* gpu_str = xstring_get("gpu", 3);
    xce_select_node_t* str_gpu = makeSelectNode(SEL_STR,
                                                (uintptr_t)gpu_str, 0);
    xce_select_node_t* eq = makeSelectNode(SEL_EQ,
                                           (uintptr_t)resname,
                                           (uintptr_t)str_gpu);
    TEST_ASSERT_NOT_NULL(eq);

    xce_select_t sel = { .root = eq };

    /* helper to build worker with string at res_idx=1 */
    xce_scheduler_t sched;
    memset(&sched, 0, sizeof(sched));

    xce_sched_worker_t wgpu;
    memset(&wgpu, 0, sizeof(wgpu));
    zarray_init(&wgpu.resvals, 2, NULL);

    xce_val_t vgpu;
    memset(&vgpu, 0, sizeof(vgpu));
    vgpu.which    = Value_u_svalue;
    vgpu.u.svalue = "gpu";
    xce_val_t* vgpup = &vgpu;
    zarray_set(&wgpu.resvals, 1, &vgpup);

    TEST_ASSERT_EQUAL_INT(1, sched_meet_host_selection(&sched, &wgpu, &sel));

    xce_sched_worker_t wcpu;
    memset(&wcpu, 0, sizeof(wcpu));
    zarray_init(&wcpu.resvals, 2, NULL);

    xce_val_t vcpu;
    memset(&vcpu, 0, sizeof(vcpu));
    vcpu.which    = Value_u_svalue;
    vcpu.u.svalue = "cpu";
    xce_val_t* vcpup = &vcpu;
    zarray_set(&wcpu.resvals, 1, &vcpup);

    TEST_ASSERT_EQUAL_INT(0, sched_meet_host_selection(&sched, &wcpu, &sel));

    zarray_fini(&wgpu.resvals);
    zarray_fini(&wcpu.resvals);
    gc_clear_marks();
    gc_sweep_select_nodes(NULL, 0);
    sel_node_table_fini();
    sel_optimize_test_reset();
}

In main, register:

    TEST(test_meet_host_selection_null_select_passes);
    TEST(test_meet_host_selection_always_true_passes);
    TEST(test_meet_host_selection_always_false_fails);
    TEST(test_build_host_eval_ctx_float);
    TEST(test_meet_host_selection_numeric_match);
    TEST(test_meet_host_selection_string_match);
    TEST(test_scheduler_t_has_atomic_gc_fields);
    TEST(test_gc_predicates_idle_invariant);
    TEST(test_sel_optimize_level_set_at_init);

Step 3: Check xce_val_t field names

grep -n 'svalue\|fvalue\|bvalue\|xce_val_t' src/include/xce/types.h | head -20

Adjust v.u.fvalue, v.u.svalue, v.u.bvalue in the tests if the generated capnp struct uses different field names.

Step 4: Build and run all scheduler tests

nix develop --command bash -c "cd /workspace/src && ninja -C build && build/scheduler/testsuite/sched_test"

Expected: all tests PASS.

Step 5: Run full build + lib testsuite to verify no regressions

nix develop --command bash -c "cd /workspace/src && ninja -C build && build/lib/testsuite/xce_test"

Expected: all tests PASS.

Step 6: Commit

git add src/scheduler/testsuite/meson.build src/scheduler/testsuite/test.c
git commit -m "test: scheduler integration tests for sched_meet_host_selection (6-11)"

Self-Review Checklist

Spec item	Task
6-1 pre-audit	Task 1
6-2 blocking gate grep	Task 1
6-3 _Atomic fields	Task 3
6-4 GC predicate wrappers (real scheduler)	Task 4
6-5 implement sched_meet_host_selection	Task 6
6-6 (deferred — void return + session restructure)	—
6-7 build_host_eval_ctx	Task 5
6-8 live_jobs flat array + mark phase	Task 7
6-9 sel_optimize_set(1)	Task 4
6-10 sel_alloc_thread_id	Task 8
6-11 integration tests	Task 9
6-12 atomic_thread_fence	Task 7

Post-review fixes (final code review, 2026-04-30)

#	Severity	Finding	Fix	Commit
R1	CRITICAL	`staging_merge_complete` bootstrap deadlock — `sched_begin_session` reset it to 0 but never set it back to 1; `sched_end_session`'s guard `(!staging_merge_complete)` always fired → GC never ran, select-node memory leaked indefinitely	Added `atomic_store_explicit(&sched->staging_merge_complete, 1, memory_order_release)` at end of `sched_begin_session` after `sched_account_jobs`	`d438742`
R2	HIGH	`recompact_node_ids()` missing from GC protocol in `sched_end_session` — without recompaction `sel_node_next_id` grows monotonically; `select_eval_ctx_t` arrays (sized by max node_id) grow without bound across GC cycles	Added `recompact_node_ids()` call after `gc_sweep_select_nodes(NULL, 0)`	`d438742`

Known deferred items

6-6 (outer-host/inner-job session restructuring + void return): The current implementation uses eval_select_simple which re-evaluates the expression for every (host, job) call independently. The design doc describes a future optimisation where a single select_eval_ctx_t is reused per host across all jobs in a session. Deferred to a follow-on plan.
eval_in_progress guard: The sched_eval_in_progress check in sched_end_session is always false (field never set to 1). Correct under the current single-threaded model; wiring is deferred to when concurrent eval is introduced.

41 KiB Raw Blame History

Phase 6 — Scheduler Integration Implementation Plan

File Map

Task 1: Pre-audit — confirm single-threaded invariant and stub locations (6-1, 6-2)

Task 2: Export GC mark primitives — split gc_sweep_select_nodes (6-8 prerequisite)

Task 3: Add _Atomic GC-predicate fields to xce_scheduler_t (6-3)

Task 4: Add GC-predicate statics in scheduler.c + wire sel_optimize_set (6-4, 6-9)

Task 5: Implement build_host_eval_ctx (6-7)

Task 6: Implement sched_meet_host_selection (6-5)

Task 7: Drive GC from sched_end_session with live_jobs + atomic fence (6-8, 6-4, 6-12)

Task 8: Add sel_alloc_thread_id thread-ownership checks (6-10)

Task 9: Scheduler testsuite — link resource.c + integration tests (6-11)

Self-Review Checklist

Post-review fixes (final code review, 2026-04-30)

Known deferred items

41 KiB

Raw Blame History