myxce/.kiro/specs/xce-harness-component/research.md

# Research & Design Decisions

## Summary
- **Feature**: `xce-harness-component`
- **Discovery Scope**: Extension (existing system) — Light Discovery
- **Key Findings**:
  - 14 of xce_test's 15 source files have zero dependency on controller-private headers or sources; only `ctrlstatus_fmt.h` is genuinely shared with controller, and it is header-only (`static inline`), so it can be promoted to a neutral shared location instead of creating a cross-component dependency in either direction.
  - **Corrected during `/kiro-validate-design` review**: only 13 of the 17 xce_test-related test files actually live inside the single `controller_unit_test` binary (`test_waldump.c` through `test_test_cli.c`). The other 4 — `test_resval_typeresolve.c`, `test_joblist_display.c`, `test_join_worker_args.c`, `test_listrow_json.c` — are **already separate standalone executables** in `src/controller/meson.build` (`resval_typeresolve_unit_test`, `joblist_display_unit_test`, `join_worker_args_unit_test`, `listrow_json_unit_test`), each split out specifically because they have "zero dependency on controller.c/cluster.c internals" — the same property verified for all 15 xce_test production files. User decision: preserve all 5 as 5 separate binaries after the move (no consolidation) — see Design Decisions.
  - Two of the 13 files in the main group (`test_test_cli.c`, `test_xcetest_process_entry.c`) are process-spawning integration tests: they locate and `exec` the actual built `xce_test` binary over a pty (`clitest_spawn`/`resolve_xce_test_path`), not white-box linkage. The new `harness_unit_test` binary (the one genuinely new binary — the other 4 relocate verbatim) therefore has a genuine build-order dependency on the `xce_test` executable target — a pattern `controller_unit_test` never needed (it never had to build another executable as its own prerequisite).
  - `controller_test.c` (`src/controller/testsuite/controller_test.c`) is the master Unity runner for the 13-file group — confirmed by reading it directly: it declares one `extern`/`RUN_TEST` pair per test function (lines 68-80 for the 13 moving functions; `test_ctrlstatus` at line 64 stays, since it tests controller's own `ctrlstatus.c`). This file must be edited, not just the meson source lists — an item the original design pass missed entirely.
  - The exact site to decouple the build is concrete and single-line: `src/tools/xce-build.sh:295`'s `controller)` case compiles `xce_controller controller_unit_test controller/xce_test` — `controller/xce_test` is the one target to drop, and a new `harness)` case follows the identical pattern already used by `scheduler)`/`worker)`.
  - Four downstream skills assume xce_test exists once controller is built: `xce-test-ha` (`run-ha-soak.sh`), `xce-deploy-ha` (`SKILL.md`, soft-skip pattern), `xce-test-multihost` (`run-multihost-verify.sh`), `xce-test-resources` (`run-resource-verify.sh`) — each needs an explicit harness-build step or prerequisite check.

## Research Log

### Controller-dependency audit of xce_test's 15 source files
- **Context**: Requirement 1.4 requires the harness to have no dependency on controller-private sources/headers; this must be verified, not assumed.
- **Sources Consulted**: `grep -h '^#include "'` across all 15 `xce_test_src` files (`test.c`, `waldump.c`, `walcompare.c`, `snapverify.c`, `snapdiff.c`, `configcompare.c`, `resval_typeresolve.c`, `joblist_display.c`, `join_worker_args.c`, `rcfile.c`, `flagspec.c`, `interruptcall.c`, `confirm.c`, `listfmt_json.c`, `listrow_json.c`).
- **Findings**: Every include resolves to either a libxce public header (`src/include/*.h`: `client.h`, `config.h`, `delay.h`, `idgen.h`, `persist.h`, `preason.h`, `utils.h`, `wal_types.h`, `xce/cluster.h`, `xce/types.h`), the generated `controller.capnp.h` schema header, xce_test's own component-local headers (14 of the 15, one-to-one with the `.c` files, plus `xcetest_interrupt.h`), or the one shared exception below (`ctrlstatus_fmt.h`).
- **Implications**: The move is a pure relocation for 14/15 files — no header untangling needed. Only `ctrlstatus_fmt.h` requires an explicit decision (see Decision below).

### `ctrlstatus_fmt.h` shared-header investigation
- **Context**: `#include "ctrlstatus_fmt.h"` appeared in both an xce_test file (`joblist_display.c`) and a controller test file — needed to confirm whether this is a real controller-private dependency (which would violate Requirement 1.4) or something safely shareable.
- **Sources Consulted**: `grep -rl "ctrlstatus_fmt.h"` across `src/controller/*.c` and `src/controller/testsuite/*.c`; read the function definition.
- **Findings**: `ctrlstatus_fmt.h` declares exactly one function, `ctrlstatus_format_time_ms()`, as `static inline` (`ctrlstatus_fmt.h:33`) — header-only, no `.c` implementation, no link-time dependency. Four consumers: `test_ctrlstatus.c` (controller's own white-box test — stays with controller), and `joblist_display.c` / `test.c` / `test_joblist_display.c` (all three move to harness). No production controller `.c` file includes it.
- **Implications**: Because it is header-only, it can move to a neutral shared location without creating a link dependency in either direction. See Decision below.

### `test_test_cli.c` / `test_xcetest_process_entry.c` linkage mechanism
- **Context**: These two files are part of `controller_unit_test_src` but have no matching production `.c` counterpart (unlike the other 14 pairs) — needed to understand how they test xce_test if `test.c` itself isn't recompiled into `controller_unit_test`.
- **Sources Consulted**: Read `test_test_cli.c` (`clitest_spawn`/`clitest_spawn_ex`/`resolve_xce_test_path`) and grep for `resolve_xce_test_path` across the testsuite.
- **Findings**: `test_test_cli.c` defines `resolve_xce_test_path()` and `clitest_spawn()`/`clitest_spawn_ex()` itself — these locate the already-built `xce_test` binary on disk and `fork`/`exec` it under a pty, then drive it as a real subprocess (matching bug-1792's fix in `.wolf/buglog.json`, which touched this exact spawn path). `test_xcetest_process_entry.c` reuses `resolve_xce_test_path` via an `extern` declaration into the same `controller_unit_test` binary (both are linked into one executable today, so `extern` resolves within that one binary).
- **Implications**: These are integration tests wearing a unit-test binary's clothes. The new harness unit-test binary must build (or otherwise guarantee the existence of) the `xce_test` executable *before* running these two tests — a real build-order dependency to declare explicitly in `src/harness/meson.build`, not an incidental one.

### Downstream consumer inventory
- **Context**: Requirement 5 requires every workflow that assumes "building controller also builds xce_test" to keep working once that coupling is removed.
- **Sources Consulted**: `grep -rl "xce_test\|xce-build controller"` across the four named skills' directories.
- **Findings**: `xce-test-ha/run-ha-soak.sh` + `SKILL.md`, `xce-deploy-ha/SKILL.md` (no script of its own — `/xce-deploy-ha` deploys already-built binaries and soft-skips missing ones per `build-deployment.md`), `xce-test-multihost/run-multihost-verify.sh` + `SKILL.md`, `xce-test-resources/run-resource-verify.sh` + `SKILL.md`.
- **Implications**: Three shell scripts plus four `SKILL.md` docs are the concrete edit surface for Requirement 5; `xce-deploy-ha` needs only its soft-skip documentation/message updated (mirroring its existing worker/scheduler/xceadmin soft-skip pattern), not a new build step of its own.

### `xce-build.sh` decoupling site
- **Context**: Needed the exact, current mechanism coupling `controller` and `xce_test` builds to plan the decoupling precisely rather than guessing at meson internals.
- **Sources Consulted**: Read `src/tools/xce-build.sh:282-340` (the `case "$COMPONENT" in ... esac` dispatch).
- **Findings**: `controller)` (lines 292-299) runs `meson compile ... xce_controller controller_unit_test controller/xce_test` then `meson test --suite controller`. Sibling cases (`scheduler)`, `worker)`) follow an identical three-part pattern: `setup_build`, an explicit `meson compile` target list, then a conditional `meson test --suite <name>`. `all)` (not shown here, documented in `build-deployment.md`) compiles with no explicit target list, so it will automatically pick up a new `harness` target once `src/meson.build` gains `subdir('harness')` — no `all)` case change needed.
- **Implications**: The fix is mechanical and low-risk: drop `controller/xce_test` from line 295, and add a new `harness)` case built from the same three-part template already proven by `scheduler)`/`worker)`.

## Architecture Pattern Evaluation

| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|---------------------|-------|
| Nested `src/` per structure.md's general pattern (`src/harness/src/*.c`, `src/harness/include/*.h`) | Follow the worker/scheduler layout exactly | Consistent with the *documented* general pattern | xce_test's files were already flat siblings in `testsuite/`; adding a nesting layer is pure churn with no benefit for a single-purpose CLI tool | Rejected — no multi-domain internal structure to justify the extra nesting |
| Flat root, controller-exception style (`src/harness/*.c`, `src/harness/include/*.h`, `src/harness/testsuite/*.c`) | Mirror controller's own documented layout exception | Matches the closest sibling in both size and shape (single-purpose C program, not a multi-file daemon); minimal file churn (headers/`.c` files keep flat sibling relationships they already have) | Diverges from the "general" worker/scheduler pattern, but that pattern is documented as having an existing exception already (controller) | **Selected** |

## Design Decisions

### Decision: Directory layout mirrors controller's flat-root exception, not the nested worker/scheduler pattern
- **Context**: `structure.md`'s C Daemon Layout Pattern nests implementation under `src/<component>/src/`, but documents `controller/` as an explicit exception that keeps `.c` files flat in the component root while still having its own `testsuite/`.
- **Alternatives Considered**: (1) nested `src/`/`include/` per the general pattern; (2) flat root matching controller's exception.
- **Selected Approach**: `src/harness/*.c` + `src/harness/include/*.h` for production sources (the 15 files), `src/harness/testsuite/*.c` for the 17 white-box/integration test files plus a new Unity runner.
- **Rationale**: xce_test's files are already flat siblings today; nesting them would be churn without a corresponding benefit, since there is no multi-domain internal structure (unlike worker/scheduler, which separate plugin/provider subsystems).
- **Trade-offs**: A future reader must know controller and harness are the two flat-root exceptions to the general pattern — mitigated by recording both in `structure.md`'s Exceptions list (Requirement 6.3).
- **Follow-up**: None.

### Decision: Promote `ctrlstatus_fmt.h` to `src/include/xce/`
- **Context**: It is the one header genuinely shared between controller (`test_ctrlstatus.c`) and harness (`joblist_display.c`, `test.c`, `test_joblist_display.c`); leaving it in either component's tree would make the other component depend on it.
- **Alternatives Considered**: (1) leave it in `src/controller/testsuite/`, have harness include it via a relative path back into controller (violates Requirement 1.4's "no dependency on any controller-private header" literally, and creates a harness→controller direction the standalone-component goal is meant to eliminate); (2) move it to `src/controller/include/` (controller's real component-header dir) and have harness depend on that explicitly as an allowed exception; (3) promote it to bare `src/include/` (the libxce public-header boundary); (4) promote it to `src/include/xce/` (the existing small subset of cross-daemon shared headers — `cluster.h`, `types.h`, `errnox.h`, `client_failover.h`, `cluster_def.h` — distinct from libxce's own API headers).
- **Selected Approach**: (4) — move to `src/include/xce/ctrlstatus_fmt.h`, update its three moving consumers plus controller's `test_ctrlstatus.c` to include it as `xce/ctrlstatus_fmt.h`.
- **Rationale**: It is header-only (`static inline`, zero link surface) and used for display formatting, not controller business logic, so it fits a shared boundary cleanly and satisfies Requirement 1.4 literally rather than by exception. Raised as a `/kiro-validate-design` critical issue (bare `src/include/` reads as libxce's real runtime API surface, which this isn't) — resolved to the existing `xce/` prefix instead, which is explicitly precedented for exactly this kind of small, cross-daemon, non-core-API shared header.
- **Trade-offs**: Nominally grows `src/include/xce/`'s surface by one small formatting helper; judged acceptable given the existing precedent and that it removes a real cross-component coupling.
- **Follow-up**: None.

### Decision: Keep the 5 harness test binaries separate rather than consolidating
- **Context**: `/kiro-validate-design` review found the original design's "one new `harness_unit_test` binary" model was factually incomplete — 4 of the 17 test files are already independent binaries in `src/controller/meson.build`, split out because they have no dependency on controller internals. Once relocated, that original isolation reason no longer distinguishes them from the rest of harness (everything in harness has that same property), which made consolidation into one binary a plausible simplification.
- **Alternatives Considered**: (1) consolidate all 17 files into one `harness_unit_test` binary; (2) preserve today's 5-binary structure verbatim, just relocated and re-suited.
- **Selected Approach**: (2) — user decision.
- **Rationale**: Lower risk and higher file-for-file fidelity to the current, working structure; avoids introducing a new merge step (and its own risk of a missed extern/RUN_TEST wiring mistake) into a spec whose entire purpose is a low-risk relocation.
- **Trade-offs**: Forgoes a small simplification opportunity (one binary instead of five); judged acceptable given this spec's goal is decoupling, not build consolidation.
- **Follow-up**: None — the 4 standalone binaries relocate with no internal changes beyond their `suite:` tag and include/dependency paths.

### Decision: `harness_unit_test` declares an explicit build-order dependency on the `xce_test` executable
- **Context**: `test_test_cli.c` and `test_xcetest_process_entry.c` spawn the built `xce_test` binary as a subprocess; they will silently exec a stale or missing binary if the harness's own build doesn't guarantee `xce_test` is built first.
- **Alternatives Considered**: (1) leave build ordering implicit (risk: flaky/stale-binary test runs); (2) declare the dependency explicitly in `src/harness/meson.build` so meson's own DAG enforces it.
- **Selected Approach**: (2).
- **Rationale**: This is exactly the kind of implicit-but-load-bearing coupling this spec exists to eliminate elsewhere (controller↔harness); introducing a new implicit one inside harness itself would be inconsistent.
- **Trade-offs**: None material — this is the same pattern meson already supports for any test that needs another target built first.
- **Follow-up**: Verify during implementation that the two spawning tests fail loudly (not silently pass against a stale binary) if the dependency were ever removed by accident — worth one regression test.

## Risks & Mitigations
- **Risk**: A downstream skill script is missed, silently breaking a workflow the first time someone runs it without having built harness. — **Mitigation**: Requirement 5.4 requires a clear, actionable failure message wherever the harness binary is expected but absent; the four consumers were enumerated by grep across all four skill directories, not guessed.
- **Risk**: Splitting `controller_unit_test_src`/`controller_test.c` by hand introduces a copy/paste error (a test moves but its production pairing or its `extern`/`RUN_TEST` entry doesn't, or one of the 4 standalone `executable()`/`test()` blocks is only partially relocated) and the build fails or silently drops coverage. — **Mitigation**: Requirement 3 requires the same pass/fail semantics and a distinct `harness` suite name, so a dropped test surfaces as a suite-count regression rather than a silent gap; `design.md`'s File Structure Plan now names every moving file and every moving meson block explicitly (13 files into `harness_unit_test` + 4 named standalone binaries + the `controller_test.c` extern list), rather than referring to "the test files" generically.
- **Risk**: `meson setup`'s full-wipe-by-default behavior (`build-deployment.md`, bug-1565) means a `harness` build after a `controller` build in the same dir (or vice versa) silently deletes the other's binaries unless `--incremental` or `/xce-build all` is used — pre-existing project behavior, not introduced by this spec, but now applies to one more pair of components. — **Mitigation**: no design change needed; document as an existing, already-understood constraint (Boundary Commitments/Constraints in `design.md`).

## References
- `.kiro/specs/xce-harness-component/brief.md` — original discovery brief with locked-in scope decisions (directory name `src/harness/`, binary name unchanged, fully decoupled build).
- `.kiro/steering/structure.md` — C Daemon Layout Pattern and its documented controller exception.
- `.kiro/steering/build-deployment.md` — `/xce-build` command semantics, meson structure, full-wipe-vs-incremental behavior.
- `.wolf/buglog.json` bug-1792 — prior fix to `clitest_spawn()`'s pty echo handling, confirms the subprocess-spawning nature of `test_test_cli.c`.