# Integration Test Plan — P0 + P1 Features

## Purpose

Verify end-to-end correctness of every feature delivered by tasks P0-1 through P1-3,
exercising the full stack: handler → store → handler merge → store readback.
These tests **do not** require a live C controller or CGO — they use the pure-Go stores
and `DefaultHandler` directly, which is what unit tests already do. The integration layer
adds: **multi-step sequences**, **cross-feature interactions**, and **edge-case coverage**
that unit tests leave out.

---

## Test Environment

- All tests run with `CGO_ENABLED=0` (pure Go)
- Each test calls `store.InitStores()` to reset all global state
- `utils.Log = zap.NewNop()` to suppress log output
- **§1–§5, §7, §8**: call handler/store methods directly — no HTTP server required
- **§6 (RBAC)**: requires a test HTTP server via `httptest.NewRecorder` with a full Gin router; JWT tokens must be real HS256-signed tokens using the configured signing secret (typically `"xce-api-key"` in test config); `ctx.Set("user", "admin")` does **not** satisfy `GetUsernameFromToken()` which reads and parses the `Authorization` header
  - **JWT IP binding:** `isValidToken` checks `claims.IP == ctx.ClientIP()`; in `httptest`, `ClientIP()` reads from `req.RemoteAddr` — set `req.RemoteAddr = "127.0.0.1:0"` on every test request and embed `"ip": "127.0.0.1"` in the JWT claims; a mismatch returns 401 for the wrong reason, silently invalidating the test
  - **Auth config:** `isValidToken` dereferences `utils.AuthenticationYamlConfig`, a package-level pointer that is `nil` until `LoadConfig()` is called — before any JWT test set `utils.AuthenticationYamlConfig = &utils.Authentication{Token: utils.TokenConfig{SecretKey: "xce-api-key"}}` directly; do not call `LoadConfig()` (reads a file that won't exist in CI)

---

## 1. P0-1: Cluster Config Push

### 1.1 Config push populates all stores in a single call

**Steps:**
1. Call `dispatchConfigPush(handler, queues, resources, hostConfigs, cluster)` with non-nil data for all four.
2. Verify `store.Queues`, `store.Resources`, `store.Hosts`, `store.Clusters` all contain the pushed data.

**Asserts:** A single config push updates all four store types in sequence (queues → resources → hosts → cluster); the four updates are NOT atomic — a concurrent reader may see a partial state.

### 1.2 Second config push merges (does not replace)

**Steps:**
1. Seed `store.Hosts` with worker-sourced data (HostStatus, CPUFactor, NumCPUs, etc.)
2. Call `h.OnHostConfigUpdated()` with config-sourced fields (HostModel, HostType, MaxSlot)
3. Call `h.OnHostConfigUpdated()` a second time with `HostModel: "modelB"` (distinct from step 2's "modelA")
4. Verify worker-sourced fields from step 1 are still preserved
5. Verify `HostModel == "modelB"` (config-sourced field reflects the second push, not the first)

**Asserts:** Partial merge is idempotent and correctly preserves the other source's fields across multiple pushes.

### 1.3 Cluster update with zero HeartbeatInterval preserves existing value

**Steps:**
1. `store.Clusters.Put("c1", ClusterInfo{HeartbeatInterval: 45})`
2. Call `h.OnClusterUpdated(&ClusterInfo{ClusterName: "c1", HeartbeatInterval: 0, Vcpus: 8})`
3. Verify `HeartbeatInterval` is still 45 (zero is ignored per `if incoming.HeartbeatInterval > 0` guard)

**Asserts:** Cluster push with `HeartbeatInterval=0` does not zero out the stored interval.

### 1.4 Config push with nil cluster argument is a no-op

**Steps:**
1. Call `dispatchConfigPush(handler, nil, nil, nil, nil)`
2. Verify `store.Clusters.Count() == 0`

**Covers:** `dispatchConfigPush` skips the cluster update when the cluster argument is nil. (The empty-name guard lives in the CGO layer `cClusterConfToClusterInfo` and is not reachable in `CGO_ENABLED=0` tests; a `ClusterInfo{ClusterName:""}` passed directly would be stored under an empty key.)

### 1.5 Config push with nil queues/resources/hosts is safe

**Steps:**
1. Call `dispatchConfigPush(handler, nil, nil, nil, nil)`
2. Verify no panics, no store entries.

**Covers:** Nil-safety of the dispatch loop.

---

## 2. P0-2: Host Control via Backend

### 2.1 Valid control action (open/close) routes through to backend

**Steps:**
1. `store.Hosts.Put("node01", HostInfo{HostName: "node01"})`
2. Call `preCheckHostControlRequest(&HostControlRequest{ControlAction: "open", Hostname: []string{"node01"}})`
3. Assert `allFlag == false`, `err == nil`

**Asserts:** Valid actions pass pre-check.

### 2.2 Invalid action is rejected

**Steps:**
1. Call `preCheckHostControlRequest(&HostControlRequest{ControlAction: "reboot", Hostname: []string{"node01"}})`
2. Assert error is returned.

**Asserts:** Only "open" and "close" are accepted.

### 2.3 Comment length limit (512 chars)

**Steps:**
1. Call `preCheckHostControlRequest` with `Comment` of 513 chars
2. Assert error
3. Call with 512 chars — assert no error

**Already covered in unit tests.** Integration adds: verify the error message is human-readable.

### 2.4 Hostname validation checks store existence

**Steps:**
1. `store.Hosts.Put("node01", HostInfo{HostName: "node01"})`
2. Call `validateHost(ctx, action, []string{"node01", "nonexistent"})`
3. Assert `node01` in success list, `nonexistent` in failed map

**Note:** `validateHost` calls `GetUsernameFromToken()` internally, requiring the same JWT setup as §6 — set `utils.AuthenticationYamlConfig`, set `req.RemoteAddr = "127.0.0.1:0"`, embed `"ip": "127.0.0.1"` in the JWT claims, and pass the token via `Authorization` header. `ctx.Set("user", "admin")` is **not** sufficient. See §6 Test Environment notes for the full setup checklist. Unlike §2.1–2.3, this test is grouped with the §6 JWT-required tests in `controllers/integration_test.go`.

### 2.5 RBAC: host:control requires cluster_admin role

**Steps:**
1. Load RBAC config with `cluster_admin` having `host:control`
2. Use `httptest.NewRecorder` + Gin router with a spy handler that sets a response body
3. Call with a user that is NOT cluster_admin — assert 403 response
4. Call with cluster_admin user — assert 2xx and spy handler body present

**Note:** `RequirePermission` short-circuits for `cluster_admin` via early `return` (without calling `ctx.Next()`). Asserting `ctx.IsAborted() == false` gives a **false pass** — the middleware doesn't abort, but the handler is also never called. Use an HTTP recorder to verify the handler response actually appeared.

**Covers:** Route middleware integration.

### 2.6 RBAC: host:view accessible to basic_user

**Steps:**
1. Load RBAC config with `basic_user` having `host:view`
2. Call `RequirePermission("host:view")` with basic_user
3. Assert request proceeds

### 2.7 Concurrent control requests do not race

**Already covered:** `TestPreCheckHostControlRequest_NoRace` (50 goroutines).
Integration adds: test that concurrent `ControlHost` calls to the same host don't corrupt store state.

### 2.8 Batch response construction

**Steps:**
1. Build success and failed slices directly (do NOT call `ControlHost` — it calls the CGO stub which returns `errNoCGO` under `CGO_ENABLED=0`):
   - `succeededItems := []map[string]interface{}{{"node": "node01"}, {"node": "node02"}}`
   - `failedItems := []map[string]interface{}{{"node": "bad", "reason": "not found"}}`
2. Call `BuildBatchOperationResponse("close", "host", succeededItems, failedItems)`
3. Verify returned status is `"partially_accepted"`, success list has 2 entries, failed map has 1

**Covers:** Batch response construction logic; the CGO backend call is out of scope for pure-Go integration tests.

---

## 3. P1-1: Resource Value Push

### 3.1 All 7 value types convert correctly (pure-Go path)

**Steps:**
1. For each value type (i32, i64, u32, u64, fval, sval, bval):
   - Create `rawResvalData` with that type
   - Call `rawResvalToDistribution(r)`
   - Assert `ValueType`, `NumericValue`/`StringValue`/`BooleanValue` match expected

**Already covered in unit tests.** Integration adds: round-trip through `OnResourceLocationUpdated`:
2. Call `h.OnResourceLocationUpdated(rd)` for each type
3. Read back from `store.ResourceLocations` using composite key `name:host`
4. Assert stored values match

### 3.2 Nil val produces nil return (no store update)

**Steps:**
1. Create `rawResvalData{valNil: true, ...}`
2. Assert `rawResvalToDistribution` returns nil
3. Assert `store.ResourceLocations.Count() == 0`

### 3.3 Unknown discriminant produces nil return with warn log

**Steps:**
1. Create `rawResvalData{valWhich: 99, ...}`
2. Assert `rawResvalToDistribution` returns nil
3. Assert `store.ResourceLocations.Count() == 0`

### 3.4 Multiple resource values in a single push

**Steps:**
1. Call `h.OnResourceLocationUpdated(rd1)` then `h.OnResourceLocationUpdated(rd2)` with different `name:host` keys
2. Assert `store.ResourceLocations.Count() == 2`
3. Assert each key returns the correct value

### 3.5 Resource value update overwrites previous value

**Steps:**
1. `h.OnResourceLocationUpdated(rd)` with `NumericValue: 100.0`
2. `h.OnResourceLocationUpdated(rd)` with same `name:host` but `NumericValue: 200.0`
3. Assert stored value is 200.0 (last-write-wins)

**Covers:** Store `Put` is a full replacement, not merge.

### 3.6 Composite key format is `res_name:location`

**Steps:**
1. Push a resource with `ResourceName: "mem"` and `Location: "node01"`
2. Assert `store.ResourceLocations.Has("mem:node01") == true`
3. Assert `store.ResourceLocations.Has("mem") == false`

---

## 4. P1-3: Heartbeat Keep-Alive

> **Isolation requirement:** `heartbeatIntervalSecs` is a package-level `var int32` in the `backend` package shared across all tests in that package. Tests in this section MUST NOT call `t.Parallel()`. Run the full heartbeat test group sequentially to avoid cross-test interference via `SetHeartbeatInterval`.

### 4.1 Heartbeat interval updates on cluster config push

**Steps:**
1. `atomic.StoreInt32(&heartbeatIntervalSecs, 30)` (default)
2. Call `h.OnClusterUpdated(&ClusterInfo{ClusterName: "c1", HeartbeatInterval: 45})`
3. Assert `atomic.LoadInt32(&heartbeatIntervalSecs) == 45`

**Covers:** The P1-3 integration point — cluster config push triggers heartbeat interval change.

### 4.2 Cluster push with HeartbeatInterval=0 preserves current interval

**Steps:**
1. `atomic.StoreInt32(&heartbeatIntervalSecs, 45)`
2. Call `h.OnClusterUpdated(&ClusterInfo{ClusterName: "c1", HeartbeatInterval: 0})`
3. Assert `atomic.LoadInt32(&heartbeatIntervalSecs) == 45` (unchanged)

**Covers:** The `> 0` guard in `OnClusterUpdated` and `SetHeartbeatInterval`.

---

## 5. Cross-Feature Integration

### 5.1 Config push updates heartbeat interval AND stores cluster data

**Steps:**
1. Default `heartbeatIntervalSecs = 30`
2. Call `dispatchConfigPush(h, nil, nil, nil, &ClusterInfo{ClusterName: "c1", HeartbeatInterval: 60, Vcpus: 32})`
3. Assert `store.Clusters.Get("c1")` has `HeartbeatInterval=60, Vcpus=32`
4. Assert `atomic.LoadInt32(&heartbeatIntervalSecs) == 60`

**Covers:** P0-1 + P1-3 interaction in a single dispatch.

### 5.2 Config push + host control on same host

**Steps:**
1. `dispatchConfigPush(h, nil, nil, []HostInfo{{HostName: "node01", HostModel: "x86_64"}}, nil)`
2. Assert `store.Hosts.Get("node01")` has `HostModel="x86_64"`
3. Assert `store.Hosts.Has("node01") == true` — verifies config-push data is visible in the host store
4. Call `preCheckHostControlRequest(&HostControlRequest{ControlAction: "close", Hostname: []string{"node01"}})` — assert `allFlag == false`, `err == nil` (validates action format; store lookup is handled by `validateHost` which requires JWT — see §2.4 note)

**Covers:** Data created by config push is written to the host store and is visible to subsequent operations.

### 5.3 Resource push after host config push — same host referenced

**Steps:**
1. `h.OnHostConfigUpdated(&HostInfo{HostName: "node01", HostModel: "x86_64"})`
2. `h.OnResourceLocationUpdated(&ResourceDistributionInfo{ResourceName: "mem", Location: "node01", NumericValue: 65536.0, ValueType: NUMERIC, IsDynamic: true})`
3. Assert `store.Hosts.Get("node01")` has `HostModel="x86_64"` (preserved)
4. Assert `store.ResourceLocations.Get("mem:node01")` has `NumericValue=65536.0`
5. Assert `store.Hosts.Get("node01").HostModel` still "x86_64" (host config was not overwritten)

**Covers:** Two independent push paths don't interfere with each other's store data.

### 5.4 Worker push after config push preserves config fields

**Steps:**
1. `h.OnHostConfigUpdated(&HostInfo{HostName: "node01", HostModel: "x86_64", MaxSlot: 8})`
2. `h.OnHostUpdated(&HostInfo{HostName: "node01", HostStatus: 3, CPUFactor: 2.5, NumCPUs: 16})`
3. Assert `got.HostModel == "x86_64" && got.MaxSlot == 8` (config preserved)
4. Assert `got.HostStatus == 3 && got.CPUFactor == 2.5` (worker updated)

**Covers:** Two-push merge correctness — already tested in unit tests but verified here at integration scope.

### 5.5 Multiple sequential config pushes (idempotent)

**Steps:**
1. Push config with `ClusterInfo{Vcpus: 16, HeartbeatInterval: 30}`
2. Push config with `ClusterInfo{Vcpus: 32, HeartbeatInterval: 60}`
3. Assert store has `Vcpus=32, HeartbeatInterval=60` (last-write-wins for config fields)
4. Assert `heartbeatIntervalSecs == 60`

**Covers:** Rapid config push updates don't leave stale data.

---

## 6. RBAC Integration (End-to-End)

> **RBAC isolation:** Call `resetRBACState()` (not `store.InitStores()`) at the start of each RBAC test. `resetRBACState()` zeros `rbacConfig`, preventing `AddPermissionsForRole` from appending duplicate permissions across test reloads. `store.InitStores()` does not reset RBAC state.

### 6.1 Full middleware chain: JWT → RBAC → handler

**Steps:**
1. Set up Gin router with real middleware (`RequirePermission("host:control")`)
2. Create a valid JWT token for a cluster_admin user
3. POST `/xce/v1/hosts/control` with `action=close, host_name=["node01"]`
4. Assert 202 Accepted (or mocked backend response)

5. Create a valid JWT token for a basic_user
6. POST `/xce/v1/hosts/control` — assert 403 Forbidden

7. POST `/xce/v1/hosts/select` with basic_user — assert passes (host:view)

**Note:** This requires spinning up a test HTTP server via `httptest.NewRecorder`. Do not mock at the middleware level — the purpose of §6 is to exercise the real middleware chain end-to-end.

### 6.2 RBAC: cluster_admin bypasses permission check

**Steps:**
1. Set up Gin router with `RequirePermission("host:control")`
2. Sign a JWT for "admin1" with role "cluster_admin" and issue `POST /xce/v1/hosts/control`
3. Assert response status is **not 403** — verifies the middleware did not block the request

**Note:** `RequirePermission` returns early for `cluster_admin` without calling `ctx.Next()`, so downstream handlers are **never reached**. Do not assert a spy handler body or `ctx.IsAborted() == false` — both give false results. To verify full handler execution for a privileged user, write a separate test with a non-cluster_admin user that has explicit `host:control` permission (see §6.3).

### 6.3 RBAC: basic_user cannot control hosts but can view

**Steps:**
1. Assign user "user1" role "basic_user"
2. Call `RequirePermission("host:control")` — assert 403
3. Call `RequirePermission("host:view")` — assert passes

### 6.4 JWT algorithm confusion is rejected

**Steps:**
1. Configure test server with HS256 signing secret `"xce-api-key"`
2. Craft a token header claiming `"alg": "none"` or a mismatched algorithm (RS256)
3. POST to `/xce/v1/hosts/control` with that token (use valid IP binding so IP mismatch is not the cause of rejection)
4. Assert 401 Unauthorized
5. Assert response body contains a token/signature error string (e.g., `"token"`, `"invalid"`, or `"signature"`) — not `"ip"` or `"expired"` — to confirm the rejection is due to algorithm/signature failure, not an unrelated check

**Covers:** Prevents algorithm confusion attacks where an attacker downgrades signing to `none`.

### 6.5 Expired JWT is rejected

**Steps:**
1. Sign a JWT with `exp = time.Now().Unix() - 3600` (expired 1 hour ago)
2. POST to any protected endpoint
3. Assert 401 Unauthorized

**Covers:** Token replay after expiry — ensures expired tokens cannot be reused.

### 6.6 Known gap: hostgroup domain restriction bypass

> **Documents a known limitation, not a working feature.** `validateHost` at `hosts.go:505-507` contains a `// TODO` that silently skips hostgroup domain enforcement. This test makes the gap explicit so it is visible in code review and prevents accidental regression if the TODO is removed.

**Steps:**
1. Provision a user with `host:control` scoped to hostgroup "gpu-nodes"
2. Issue `POST /xce/v1/hosts/control` targeting a host NOT in "gpu-nodes"
3. **Assert request is accepted** (not 403) — documents that hostgroup scoping is not yet enforced

**Note:** When P1-2b implements hostgroup enforcement, flip this assertion to assert 403 and remove the `// TODO` marker.

### 6.7 Post-logout token reuse

> **Design gap:** JWTs are stateless and the current server does not maintain a JTI blacklist. A token valid at logout time remains valid until expiry. This test documents the current behavior as an explicit design decision.

**Steps:**
1. Sign a valid JWT with a 1-hour expiry
2. Call `POST /xce/v1/users/logout` (or equivalent) — record any logout response
3. Immediately reuse the same token on `GET /xce/v1/hosts` — assert **200 OK** (not 401)

**Note:** If a JTI blacklist is added in future, flip this assertion to expect 401 and add the blacklist provisioning step.

---

## 7. Store Thread Safety

> **CI requirement:** All §7 tests must run with `-race`: `go test -race ./store/...`. A test that passes without `-race` provides no guarantee of absence of data races — the Go race detector instruments memory accesses at runtime and is the authoritative check.

### 7.1 Concurrent config pushes don't corrupt data

**Steps:**
1. Start 10 goroutines, each calling `h.OnHostConfigUpdated(&HostInfo{HostName: "node01", ...})` with slightly different HostModel values
2. Wait for all goroutines to complete
3. Assert `store.Hosts.Get("node01")` returns a valid HostInfo (no zero-value corruption)
4. Assert `HostName` field is "node01"

**Race detection note:** A simple value-validity assertion cannot prove absence of a data race — concurrent writes all produce plausible values. Run this test with `go test -race ./store/...` to detect RMW races via the Go race detector. The assertion above verifies structural integrity; `-race` detects the underlying race condition.

### 7.2 Concurrent host control + config push

**Steps:**
1. Goroutine A: call `OnHostConfigUpdated` for "node01" in a loop
2. Goroutine B: call `validateHost` checking "node01" existence in a loop
3. Assert no panics and no store corruption after 100 iterations

### 7.3 Concurrent resource value pushes for same key

**Steps:**
1. Start 10 goroutines, each calling `OnResourceLocationUpdated` for `"mem:node01"` with different NumericValue
2. Assert final stored value is one of the 10 values (no corruption)
3. Assert key exists exactly once

---

## 8. Input Safety

### 8.1 Hostname length limits

> **Skip until P1-2b is implemented.** No hostname length validation exists in `preCheckHostControlRequest` yet — this test will pass vacuously. Add `//go:build p1_2b` build tag or comment out pending that task.

**Steps:**
1. Call `preCheckHostControlRequest` with hostname exceeding the P1-2b limit (>1024 chars)
2. Assert `err != nil` with message indicating length exceeded

### 8.2 Empty hostname list

**Steps:**
1. Call `preCheckHostControlRequest` with `Hostname: []`
2. Assert `err != nil` — inspect the actual error message from `preCheckHostControlRequest` and assert it contains the expected string (e.g., "no hostnames provided" or similar)

### 8.3 Select expression length (current limit: 4094)

**Steps:**
1. Call handler with `Select` string > 4094 chars
2. Assert error
3. Call with 4094 chars — assert accepted

### 8.4 Special characters in hostnames

**Steps:**
1. Call with hostname containing `<script>alert(1)</script>`
2. Assert the hostname is stored as a string without interpretation (XSS safety at API layer)

---

## Test File Organization

| File | Tests |
|------|-------|
| `backend/integration_test.go` | §1 (config push), §3 (resource push), §4 (heartbeat), §5 (cross-feature) |
| `controllers/integration_test.go` | §2.1–2.3, §2.5–2.8 (host control — no JWT required), §2.4 (host validation — **requires full JWT setup; see §6 prerequisites**), §6 (RBAC middleware), §8 (input safety) |
| `store/concurrency_test.go` | §7 (thread safety) |

All test files use `//go:build !cgo` or `CGO_ENABLED=0` compatible patterns.

**Package declarations:**
- `backend/integration_test.go` must declare `package backend` (not `package backend_test`) to access unexported symbols: `heartbeatIntervalSecs`, `rawResvalToDistribution`, `dispatchConfigPush`
- `controllers/integration_test.go` may use `package controllers_test` for black-box tests, but any test accessing unexported helpers must use `package controllers`

**JWT signing secret in tests:** Integration tests in `controllers/integration_test.go` that create real JWTs must use a test-only signing secret (e.g., `"xce-api-key"` from the test config file). Never use the production signing secret. Document the expected test secret in a comment at the top of the test file so it is clearly identifiable during security audits.

---

## Priority Classification

| Priority | Tests | Rationale |
|----------|-------|-----------|
| **P0 — Must have** | 1.1, 1.2, 1.3, 2.1, 2.2, 2.5, 3.1, 3.4, 4.1, 5.1 | Core functionality and merge correctness |
| **P1 — Should have** | 1.4, 1.5, 2.4, 2.6, 2.8, 3.2, 3.3, 3.5, 3.6, 4.2, 5.2, 5.3, 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 6.7 | Edge cases, RBAC, JWT security, and known gap documentation |
| **P2 — Nice to have** | 2.3, 2.7, 5.4, 5.5, 7.1, 7.2, 7.3, 8.1–8.4 | Robustness and stress |

---

## Existing Test Coverage

The following unit tests already cover much of the ground above and should NOT be duplicated:

| Existing Test File | What It Covers | Integration Gap |
|--------------------|----------------|-----------------|
| `backend/handler_test.go` | Single-push merge for hosts, clusters, dispatch | Multi-push sequences, cross-feature |
| `backend/convert_test.go` | All 7 value types, nil val, unknown discriminant | Store round-trip after conversion |
| `backend/heartbeat_test.go` | SetHeartbeatInterval, runHeartbeat call/cancel | Interval change triggered by OnClusterUpdated |
| `controllers/host_control_test.go` | preCheck validation, concurrent no-race | Full handler flow with store interaction |
| `controllers/rbac_check_test.go` | host:control/host:view permissions | Full middleware chain with JWT |