# Integration Test Report: xcecli ↔ apiserver

> Date: 2026-04-08  
> Server: `apiserver` built from `apiserver/` with builtin test data (3 hosts, 3 queues, 4 jobs, 8 resources)  
> Client: `xcecli` built from `xcecli/` (CGO_ENABLED=0 static binary)  
> Auth: `config_file` provider — users: `admin` (cluster_admin), `user1`, `user2` (basic_user)

> **Updated: 2026-04-08** — All critical bugs fixed. See resolution status table below.

---

## Summary (at time of initial test)

| Category | Count |
|---|---|
| ✅ Commands working correctly | 13 |
| ⚠️ Commands partially working (wrong/empty data) | 11 |
| ❌ Commands totally broken | 3 |
| Server endpoints not covered by CLI | 2 |

## Summary (after fixes)

| Category | Count |
|---|---|
| ✅ Commands working correctly | ~27 |
| ⚠️ Minor issues remaining | 2 (clusters empty — no backend; jobs list no filter = current user only, by design) |
| ❌ Broken | 0 |

---

## Test Results by Command

### Auth

| Command | Result | Notes |
|---|---|---|
| `xce login -u admin -p admin --url http://...` | ✅ Works | Token saved to `~/.xce/config.json` |
| `xce logout --url http://...` | ✅ Works | Token cleared |
| `xce list` | ✅ Works | Shows server URL and default marker |
| `xce users whoami` | ✅ Works | Returns current username from token |

**Bug found:** `xce logout` requires `--url` flag — if omitted it errors with `required flag(s) "url" not set`. Should default to the active server.

---

### Permissions

| Command | Result | Notes |
|---|---|---|
| `xce permissions list` | ✅ Works | Returns all 21 permissions correctly |

---

### Roles

| Command | Result | Notes |
|---|---|---|
| `xce roles list` | ❌ Broken | Server rejects with `invalid role info query parameters of : unique` |
| `xce roles view -r <rolename>` | ⚠️ Partial | Request succeeds, returns correct row count, but all fields display as `-` |
| `xce roles add -r <name> -p "job:view=all"` | ✅ Works | Role created successfully |
| `xce roles delete -r <name>` | ✅ Works | Role deleted successfully |

**Root cause — `roles list`:** Client sends `params["unique"] = "true"` as a query parameter. Server's `SupportedQueryParams` does not include `unique` → rejected with 400.

**Root cause — `roles view` empty data:** Server returns `role_name`, `action`, `domain_type`, `domain_values` fields. Client `RolesResponse` model maps to `models.RoleInfo` which has different field names — JSON unmarshal silently produces zero values.

---

### Users

| Command | Result | Notes |
|---|---|---|
| `xce users list` | ❌ Broken | Server rejects with `invalid user_role info query parameters of : unique` |
| `xce users view -u <username>` | ⚠️ Partial | Returns correct row count, all fields display as `-` |
| `xce users whoami` | ✅ Works | Returns `admin` |
| `xce users assignrole -u user1 -r basic_user` | ✅ Works | Role assigned |
| `xce users unassignrole -u user1 -r basic_user` | ✅ Works | Role removed |
| `xce users permissions4user -u admin` | ⚠️ Partial | Returns correct row count, all fields display as `-` |

**Root cause — `users list`:** Same `unique` query param issue as `roles list`.

**Root cause — `users view` / `permissions4user` empty data:** Same field name mismatch as roles view.

---

### Hosts

| Command | Result | Notes |
|---|---|---|
| `xce hosts list` | ⚠️ Partial | Returns 3 rows (correct count), but `HOST_NAME`, `STATUS`, `HOST_TYPE` all show `-`; `NCPUS` shows `0` |
| `xce hosts open <hostname>` | ✅ Works | `host <host1> is being <open>` |
| `xce hosts close <hostname>` | ✅ Works | `host <host1> is being <close>` |

**Root cause — `hosts list` empty fields:**
1. `HostStatus` is `int` on server, `string` in client `XCEHost` → JSON unmarshal fails silently, `HostStatus = ""`
2. `XCEHost.NumCPUs` has json tag `"num_cores"` but server sends `"num_cpus"` → always 0
3. `HostName` and `HostType` are present in server response but `HostsResponse.Data.Rows` is `[]XCEHost` — the field names match (`host_name`, `host_type`) so these should work. Actual cause: server serializes `HostInfo` via `StructToOrderMap` using reflect + json tags, then re-serializes to JSON. The `HostStatus int` becomes a JSON number; client `XCEHost.HostStatus string` cannot unmarshal a number → entire row decode fails, leaving all string fields empty.

---

### Queues

| Command | Result | Notes |
|---|---|---|
| `xce queues list` | ⚠️ Partial | Returns 3 rows (correct count), all fields show `-` or `0` |
| `xce queues open <name>` | ✅ Works | `queue <normal> is being <open>` |
| `xce queues close <name>` | ✅ Works | `queue <normal> is being <close>` |
| `xce queues active <name>` | ✅ Works | `queue <normal> is being <active>` |
| `xce queues deactive <name>` | ✅ Works | `queue <normal> is being <deactive>` |

**Root cause — `queues list` empty fields:** `QueueInfo.QueueStatus` is `int` on server, `string` in client `DBQueueInfo` → same unmarshal failure as hosts.

---

### Jobs

| Command | Result | Notes |
|---|---|---|
| `xce jobs list` (no filter) | ⚠️ Partial | Returns "no job found" — server filters by current user AND accessible queues; admin has no jobs |
| `xce jobs list -u all` | ⚠️ Partial | Returns 3 rows (correct count for builtin data), but `JOBID=0`, all string fields `-` |
| `xce jobs list -u user1` | ⚠️ Partial | Returns 2 rows, same display issue |
| `xce jobs submit -J name -q normal <cmd>` | ✅ Works | Job ID returned, submitted to queue |
| `xce jobs submit -q normal <cmd>` (no `-J`) | ✅ Works | Server accepts without job name (client `binding:"required"` not enforced server-side) |
| `xce jobs submit -f <file> ...` | ⚠️ Partial | Job submitted but `parse file transfer config failed: [<path>]` printed — file upload path parsing error |
| `xce jobs kill <jobid>` | ⚠️ Partial | Request reaches server, server finds job by `job_id` but submitted job IDs (Snowflake) don't match builtin IDs (1001-1004) → `job not found` |
| `xce jobs stop <jobid>` | ⚠️ Partial | Same as kill |
| `xce jobs resume <jobid>` | ⚠️ Partial | Same as kill |
| `xce jobs checkpoint <jobid>` | ⚠️ Partial | Same as kill |
| `xce jobs restart <jobid>` | ⚠️ Partial | Reaches server, but fails: `job not submitted by <user1>, cannot be restarted by <admin>` — correct RBAC behavior |

**Root cause — `jobs list` display issue:** `QueryJob.JobStatus` is `string` in client but server sends `int` (bitmask). `QueryJob.JobID` is `int64` but server sends `unique_job_id` as the primary key — client maps `job_id` which is the array-level ID (same as `unique_job_id` for non-array jobs in builtin data, but Snowflake-generated IDs differ). The `JobStatus` type mismatch causes the entire row JSON decode to fail.

**Root cause — `jobs list` no results for admin:** Server `GetJobsInfo` adds `submit_user = <logon_user>` filter unless user has `job:view` permission with `all` domain. Admin's RBAC domain is not configured for `all` in the default `rbac.yaml`.

**Root cause — `jobs submit -f` parse error:** `xcecli/cmd/jobs/submit.go` tries to parse the `-f` flag value as a data spec file format (`src > dst`) before treating it as a plain file path. Plain paths without `>` fail the parse but the job still submits.

---

### Clusters

| Command | Result | Notes |
|---|---|---|
| `xce id` | ⚠️ Partial | Returns "no cluster found" — builtin data has no cluster entries in store |

---

### Resources

| Command | Result | Notes |
|---|---|---|
| `xce resources distribution list` | ⚠️ Partial | Returns 16 rows (correct count), all values show `-` |
| `xce resources definition list` | ⚠️ Partial | Returns 8 rows (correct count), all values show `-` |

**Root cause:** Server `ResourceDistributionInfo` has fields `numeric_value`, `boolean_value`, `string_value`. Client `ResourceDist` reads from a single `"value"` key via `ResourceValue.UnmarshalJSON`. Server never sends a `"value"` key → always empty. Same for definition: server sends `value_type` as `int`, client `ResourceDef.ResourceType` is `string`.

---

## Server Endpoints Not Covered by CLI

| Endpoint | Method | Notes |
|---|---|---|
| `/xce/v1/hosts/select` | POST | Filter hosts by resource requirement string (`-R select(...)`) — no CLI command |
| `/xce/v1/files` | GET | List uploaded files — no `xce files list` command (file ops are embedded in `jobs submit`) |
| `/xce/v1/files/:uuid/:filename` | GET | Download file — no `xce files download` command |
| `/xce/v1/jobs/:jobids` | GET | Get specific jobs by ID — CLI uses the general list with filter instead |

---

## Root Cause Summary

```mermaid
graph TD
    A[Display shows all dashes / zeros] --> B{Type mismatch\nserver int → client string}
    B --> C[QueueStatus: int vs string]
    B --> D[HostStatus: int vs string]
    B --> E[JobStatus: int vs string]
    B --> F[ResourceType: int vs string]
    B --> G[ResourceValue: typed fields vs single value key]

    H[roles list / users list broken] --> I[Client sends unique=true\nServer rejects unknown param]

    J[roles view / users view / permissions4user\nshow dashes] --> K[Field name mismatch\nserver action/domain_type vs\nclient RoleInfo struct]

    L[jobs list no results for admin] --> M[RBAC: admin role missing\njob:view with all domain]

    N[jobs control not found] --> O[Snowflake job IDs don't\nmatch builtin test IDs 1001-1004]

    P[jobs submit -f parse error] --> Q[File path parsed as\ndata spec format src>dst]
```

---

## New Bugs Found — Resolution Status

| ID | Severity | Component | Description | Status |
|---|---|---|---|---|
| T1 | 🔴 | xcecli | `roles list`/`users list`: remove `unique=true` param; deduplicate client-side | ✅ fixed |
| T2 | 🔴 | xcecli | `QueueStatus`/`HostStatus`/`JobStatus` type mismatch | ✅ fixed (server sends strings; T3b root cause) |
| T3 | 🔴 | xcecli | Resource distribution value fields | ✅ fixed (part of T3b) |
| T3b | 🔴 | xcecli | All rows wrapped in `{Fields,Data}` OrderMap — `UnmarshalOrderMapRow` added | ✅ fixed |
| T4 | 🔴 | xcecli | `ResourceDef` display alignment | ✅ verified OK |
| T5 | 🔴 | xcecli | Column truncation in roles/users view | ✅ fixed |
| T6 | 🟠 | apiserver | Admin `job:view` RBAC | ✅ already correct in rbac.yaml |
| T7 | 🟠 | xcecli | `xce logout --url` optional | ✅ fixed |
| T8 | 🟠 | xcecli | `jobs submit -f` plain path parse error | ✅ fixed |
| T9 | 🟡 | xcecli | Add `xce hosts select` command | ⏳ pending |
| T10 | 🟡 | xcecli | Add `xce files list`/`download` commands | ⏳ pending |
| M2 | 🔴 | xcecli | `QueryJob.ExitStatus` type mismatch | ✅ fixed |
| M3 | 🔴 | xcecli | `XCEHost.NumCPUs` json tag wrong | ✅ fixed (part of T3b) |
| A1 | 🔴 | xcecli | Token header leading space | ✅ fixed |
| A3 | 🟠 | xcecli | TLS `InsecureSkipVerify` always set | ✅ fixed |
| S1 | 🔴 | apiserver | JWT secret hardcoded in config | ✅ fixed (env var override) |
| S3 | 🟠 | apiserver | Hardcoded mock RBAC maps | ✅ fixed |
| S4 | 🟠 | apiserver | Whitelist pattern truncation bug | ✅ fixed |

| ID | Severity | Component | Description |
|---|---|---|---|
| **T1** | 🔴 | xcecli | `roles list` / `users list`: remove `params["unique"] = "true"` — server rejects this param. Implement client-side deduplication instead |
| **T2** | 🔴 | xcecli | `QueueStatus`, `HostStatus`, `JobStatus`: change client model fields from `string` → `int`, add display helper functions to convert bitmask to string |
| **T3** | 🔴 | xcecli | `ResourceDist`: read `numeric_value`/`boolean_value`/`string_value` from server response instead of `"value"` |
| **T4** | 🔴 | xcecli | `ResourceDef.ResourceType`: change from `string` → `int`, add display helper |
| **T5** | 🔴 | xcecli | `roles view` / `users view` / `permissions4user`: fix field name mapping — server returns `action`, `domain_type`, `domain_values`; client struct must match |
| **T6** | 🟠 | apiserver | `rbac.yaml` default config: `cluster_admin` role needs `job:view` with `all` domain so admin can see all jobs |
| **T7** | 🟠 | xcecli | `xce logout`: make `--url` optional, default to active server URL from config |
| **T8** | 🟠 | xcecli | `jobs submit -f`: fix file path parsing — plain paths (no `>`) should not trigger data spec parse error |
| **T9** | 🟡 | xcecli | Add `xce hosts select` command to expose `POST /xce/v1/hosts/select` |
| **T10** | 🟡 | xcecli | Add `xce files list` and `xce files download` commands |

---

## Updated Full Todo List

### 🔴 Critical — breaks functionality

| ID | Component | Task | Source |
|---|---|---|---|
| T1 | xcecli | Remove `unique=true` query param from roles/users list; deduplicate client-side | New |
| T2 | xcecli | Fix `QueueStatus`/`HostStatus`/`JobStatus` type: `string` → `int` + display helpers | New (confirms M1, M2) |
| T3 | xcecli | Fix resource distribution value fields: read `numeric_value`/`boolean_value`/`string_value` | New (confirms M4) |
| T4 | xcecli | Fix `ResourceDef.ResourceType`: `string` → `int` + display helper | New |
| T5 | xcecli | Fix roles/users view field name mapping to match server response | New |
| M3 | xcecli | Fix `XCEHost.NumCPUs` json tag: `"num_cores"` → `"num_cpus"` | Existing |
| A1 | xcecli | Fix token header: standardize to `"Bearer "+token` in all request methods | Existing |
| S4 | apiserver | Fix whitelist pattern truncation bug | Existing |

### 🟠 High — security / usability

| ID | Component | Task | Source |
|---|---|---|---|
| T6 | apiserver | Fix `rbac.yaml`: add `job:view` with `all` domain to `cluster_admin` role | New |
| T7 | xcecli | Make `xce logout --url` optional, default to active server | New |
| T8 | xcecli | Fix `jobs submit -f` file path parsing (plain path triggers data spec error) | New |
| S1 | apiserver | Move JWT secret out of `config/app.yaml` into environment variable | Existing |
| S3 | apiserver | Remove hardcoded mock RBAC maps from `rbac_check.go` | Existing |
| A3 | xcecli | Fix TLS: wire `loadRootCAs`, only use `InsecureSkipVerify` when opted in | Existing |

### 🟡 Missing features

| ID | Component | Task | Source |
|---|---|---|---|
| T9 | xcecli | Add `xce hosts select` command (`POST /xce/v1/hosts/select`) | New |
| T10 | xcecli | Add `xce files list` and `xce files download` commands | New |
| F1 | server+cli | Implement file delete endpoint + `xce files delete` command | Existing |
| F2 | server+cli | Decide on API key management: implement or remove dead code | Existing |
| F3 | xcecli | Add array job stats display (`?type=array_stats`) | Existing |
| F4 | apiserver | Enable CORS middleware | Existing |

### 🔵 Cleanup

| ID | Component | Task | Source |
|---|---|---|---|
| Q1 | apiserver | Regenerate Swagger docs (`swag init`) | Existing |
| Q2 | apiserver | Fix `validateRequestBody` inverted condition | Existing |
| Q3 | apiserver | Replace `ioutil.ReadFile` with `os.ReadFile` | Existing |
| Q4 | apiserver | Remove unused `AlecAivazis/survey` dependency | Existing |
| Q5 | xcecli | Upgrade `resty.dev/v3` beta to stable `go-resty/resty/v2` | Existing |
| Q6 | apiserver | Fix rate limiter map unbounded growth | Existing |
| Q7 | xcecli | Add retry logic for transient network errors | Existing |