Integration Test Report: xcecli ↔ apiserver
Date: 2026-04-08
Server: apiserver built from apiserver/ with builtin test data (3 hosts, 3 queues, 4 jobs, 8 resources)
Client: xcecli built from xcecli/ (CGO_ENABLED=0 static binary)
Auth: config_file provider — users: admin (cluster_admin), user1, user2 (basic_user)
Updated: 2026-04-08 — All critical bugs fixed. See resolution status table below.
Summary (at time of initial test)
| Category |
Count |
| ✅ Commands working correctly |
13 |
| ⚠️ Commands partially working (wrong/empty data) |
11 |
| ❌ Commands totally broken |
3 |
| Server endpoints not covered by CLI |
2 |
Summary (after fixes)
| Category |
Count |
| ✅ Commands working correctly |
~27 |
| ⚠️ Minor issues remaining |
2 (clusters empty — no backend; jobs list no filter = current user only, by design) |
| ❌ Broken |
0 |
Test Results by Command
Auth
| Command |
Result |
Notes |
xce login -u admin -p admin --url http://... |
✅ Works |
Token saved to ~/.xce/config.json |
xce logout --url http://... |
✅ Works |
Token cleared |
xce list |
✅ Works |
Shows server URL and default marker |
xce users whoami |
✅ Works |
Returns current username from token |
Bug found: xce logout requires --url flag — if omitted it errors with required flag(s) "url" not set. Should default to the active server.
Permissions
| Command |
Result |
Notes |
xce permissions list |
✅ Works |
Returns all 21 permissions correctly |
Roles
| Command |
Result |
Notes |
xce roles list |
❌ Broken |
Server rejects with invalid role info query parameters of : unique |
xce roles view -r <rolename> |
⚠️ Partial |
Request succeeds, returns correct row count, but all fields display as - |
xce roles add -r <name> -p "job:view=all" |
✅ Works |
Role created successfully |
xce roles delete -r <name> |
✅ Works |
Role deleted successfully |
Root cause — roles list: Client sends params["unique"] = "true" as a query parameter. Server's SupportedQueryParams does not include unique → rejected with 400.
Root cause — roles view empty data: Server returns role_name, action, domain_type, domain_values fields. Client RolesResponse model maps to models.RoleInfo which has different field names — JSON unmarshal silently produces zero values.
Users
| Command |
Result |
Notes |
xce users list |
❌ Broken |
Server rejects with invalid user_role info query parameters of : unique |
xce users view -u <username> |
⚠️ Partial |
Returns correct row count, all fields display as - |
xce users whoami |
✅ Works |
Returns admin |
xce users assignrole -u user1 -r basic_user |
✅ Works |
Role assigned |
xce users unassignrole -u user1 -r basic_user |
✅ Works |
Role removed |
xce users permissions4user -u admin |
⚠️ Partial |
Returns correct row count, all fields display as - |
Root cause — users list: Same unique query param issue as roles list.
Root cause — users view / permissions4user empty data: Same field name mismatch as roles view.
Hosts
| Command |
Result |
Notes |
xce hosts list |
⚠️ Partial |
Returns 3 rows (correct count), but HOST_NAME, STATUS, HOST_TYPE all show -; NCPUS shows 0 |
xce hosts open <hostname> |
✅ Works |
host <host1> is being <open> |
xce hosts close <hostname> |
✅ Works |
host <host1> is being <close> |
Root cause — hosts list empty fields:
HostStatus is int on server, string in client XCEHost → JSON unmarshal fails silently, HostStatus = ""XCEHost.NumCPUs has json tag "num_cores" but server sends "num_cpus" → always 0HostName and HostType are present in server response but HostsResponse.Data.Rows is []XCEHost — the field names match (host_name, host_type) so these should work. Actual cause: server serializes HostInfo via StructToOrderMap using reflect + json tags, then re-serializes to JSON. The HostStatus int becomes a JSON number; client XCEHost.HostStatus string cannot unmarshal a number → entire row decode fails, leaving all string fields empty.
Queues
| Command |
Result |
Notes |
xce queues list |
⚠️ Partial |
Returns 3 rows (correct count), all fields show - or 0 |
xce queues open <name> |
✅ Works |
queue <normal> is being <open> |
xce queues close <name> |
✅ Works |
queue <normal> is being <close> |
xce queues active <name> |
✅ Works |
queue <normal> is being <active> |
xce queues deactive <name> |
✅ Works |
queue <normal> is being <deactive> |
Root cause — queues list empty fields: QueueInfo.QueueStatus is int on server, string in client DBQueueInfo → same unmarshal failure as hosts.
Jobs
| Command |
Result |
Notes |
xce jobs list (no filter) |
⚠️ Partial |
Returns "no job found" — server filters by current user AND accessible queues; admin has no jobs |
xce jobs list -u all |
⚠️ Partial |
Returns 3 rows (correct count for builtin data), but JOBID=0, all string fields - |
xce jobs list -u user1 |
⚠️ Partial |
Returns 2 rows, same display issue |
xce jobs submit -J name -q normal <cmd> |
✅ Works |
Job ID returned, submitted to queue |
xce jobs submit -q normal <cmd> (no -J) |
✅ Works |
Server accepts without job name (client binding:"required" not enforced server-side) |
xce jobs submit -f <file> ... |
⚠️ Partial |
Job submitted but parse file transfer config failed: [<path>] printed — file upload path parsing error |
xce jobs kill <jobid> |
⚠️ Partial |
Request reaches server, server finds job by job_id but submitted job IDs (Snowflake) don't match builtin IDs (1001-1004) → job not found |
xce jobs stop <jobid> |
⚠️ Partial |
Same as kill |
xce jobs resume <jobid> |
⚠️ Partial |
Same as kill |
xce jobs checkpoint <jobid> |
⚠️ Partial |
Same as kill |
xce jobs restart <jobid> |
⚠️ Partial |
Reaches server, but fails: job not submitted by <user1>, cannot be restarted by <admin> — correct RBAC behavior |
Root cause — jobs list display issue: QueryJob.JobStatus is string in client but server sends int (bitmask). QueryJob.JobID is int64 but server sends unique_job_id as the primary key — client maps job_id which is the array-level ID (same as unique_job_id for non-array jobs in builtin data, but Snowflake-generated IDs differ). The JobStatus type mismatch causes the entire row JSON decode to fail.
Root cause — jobs list no results for admin: Server GetJobsInfo adds submit_user = <logon_user> filter unless user has job:view permission with all domain. Admin's RBAC domain is not configured for all in the default rbac.yaml.
Root cause — jobs submit -f parse error: xcecli/cmd/jobs/submit.go tries to parse the -f flag value as a data spec file format (src > dst) before treating it as a plain file path. Plain paths without > fail the parse but the job still submits.
Clusters
| Command |
Result |
Notes |
xce id |
⚠️ Partial |
Returns "no cluster found" — builtin data has no cluster entries in store |
Resources
| Command |
Result |
Notes |
xce resources distribution list |
⚠️ Partial |
Returns 16 rows (correct count), all values show - |
xce resources definition list |
⚠️ Partial |
Returns 8 rows (correct count), all values show - |
Root cause: Server ResourceDistributionInfo has fields numeric_value, boolean_value, string_value. Client ResourceDist reads from a single "value" key via ResourceValue.UnmarshalJSON. Server never sends a "value" key → always empty. Same for definition: server sends value_type as int, client ResourceDef.ResourceType is string.
Server Endpoints Not Covered by CLI
| Endpoint |
Method |
Notes |
/xce/v1/hosts/select |
POST |
Filter hosts by resource requirement string (-R select(...)) — no CLI command |
/xce/v1/files |
GET |
List uploaded files — no xce files list command (file ops are embedded in jobs submit) |
/xce/v1/files/:uuid/:filename |
GET |
Download file — no xce files download command |
/xce/v1/jobs/:jobids |
GET |
Get specific jobs by ID — CLI uses the general list with filter instead |
Root Cause Summary
graph TD
A[Display shows all dashes / zeros] --> B{Type mismatch\nserver int → client string}
B --> C[QueueStatus: int vs string]
B --> D[HostStatus: int vs string]
B --> E[JobStatus: int vs string]
B --> F[ResourceType: int vs string]
B --> G[ResourceValue: typed fields vs single value key]
H[roles list / users list broken] --> I[Client sends unique=true\nServer rejects unknown param]
J[roles view / users view / permissions4user\nshow dashes] --> K[Field name mismatch\nserver action/domain_type vs\nclient RoleInfo struct]
L[jobs list no results for admin] --> M[RBAC: admin role missing\njob:view with all domain]
N[jobs control not found] --> O[Snowflake job IDs don't\nmatch builtin test IDs 1001-1004]
P[jobs submit -f parse error] --> Q[File path parsed as\ndata spec format src>dst]
New Bugs Found — Resolution Status
| ID |
Severity |
Component |
Description |
Status |
| T1 |
🔴 |
xcecli |
roles list/users list: remove unique=true param; deduplicate client-side |
✅ fixed |
| T2 |
🔴 |
xcecli |
QueueStatus/HostStatus/JobStatus type mismatch |
✅ fixed (server sends strings; T3b root cause) |
| T3 |
🔴 |
xcecli |
Resource distribution value fields |
✅ fixed (part of T3b) |
| T3b |
🔴 |
xcecli |
All rows wrapped in {Fields,Data} OrderMap — UnmarshalOrderMapRow added |
✅ fixed |
| T4 |
🔴 |
xcecli |
ResourceDef display alignment |
✅ verified OK |
| T5 |
🔴 |
xcecli |
Column truncation in roles/users view |
✅ fixed |
| T6 |
🟠 |
apiserver |
Admin job:view RBAC |
✅ already correct in rbac.yaml |
| T7 |
🟠 |
xcecli |
xce logout --url optional |
✅ fixed |
| T8 |
🟠 |
xcecli |
jobs submit -f plain path parse error |
✅ fixed |
| T9 |
🟡 |
xcecli |
Add xce hosts select command |
⏳ pending |
| T10 |
🟡 |
xcecli |
Add xce files list/download commands |
⏳ pending |
| M2 |
🔴 |
xcecli |
QueryJob.ExitStatus type mismatch |
✅ fixed |
| M3 |
🔴 |
xcecli |
XCEHost.NumCPUs json tag wrong |
✅ fixed (part of T3b) |
| A1 |
🔴 |
xcecli |
Token header leading space |
✅ fixed |
| A3 |
🟠 |
xcecli |
TLS InsecureSkipVerify always set |
✅ fixed |
| S1 |
🔴 |
apiserver |
JWT secret hardcoded in config |
✅ fixed (env var override) |
| S3 |
🟠 |
apiserver |
Hardcoded mock RBAC maps |
✅ fixed |
| S4 |
🟠 |
apiserver |
Whitelist pattern truncation bug |
✅ fixed |
| ID |
Severity |
Component |
Description |
| T1 |
🔴 |
xcecli |
roles list / users list: remove params["unique"] = "true" — server rejects this param. Implement client-side deduplication instead |
| T2 |
🔴 |
xcecli |
QueueStatus, HostStatus, JobStatus: change client model fields from string → int, add display helper functions to convert bitmask to string |
| T3 |
🔴 |
xcecli |
ResourceDist: read numeric_value/boolean_value/string_value from server response instead of "value" |
| T4 |
🔴 |
xcecli |
ResourceDef.ResourceType: change from string → int, add display helper |
| T5 |
🔴 |
xcecli |
roles view / users view / permissions4user: fix field name mapping — server returns action, domain_type, domain_values; client struct must match |
| T6 |
🟠 |
apiserver |
rbac.yaml default config: cluster_admin role needs job:view with all domain so admin can see all jobs |
| T7 |
🟠 |
xcecli |
xce logout: make --url optional, default to active server URL from config |
| T8 |
🟠 |
xcecli |
jobs submit -f: fix file path parsing — plain paths (no >) should not trigger data spec parse error |
| T9 |
🟡 |
xcecli |
Add xce hosts select command to expose POST /xce/v1/hosts/select |
| T10 |
🟡 |
xcecli |
Add xce files list and xce files download commands |
Updated Full Todo List
🔴 Critical — breaks functionality
| ID |
Component |
Task |
Source |
| T1 |
xcecli |
Remove unique=true query param from roles/users list; deduplicate client-side |
New |
| T2 |
xcecli |
Fix QueueStatus/HostStatus/JobStatus type: string → int + display helpers |
New (confirms M1, M2) |
| T3 |
xcecli |
Fix resource distribution value fields: read numeric_value/boolean_value/string_value |
New (confirms M4) |
| T4 |
xcecli |
Fix ResourceDef.ResourceType: string → int + display helper |
New |
| T5 |
xcecli |
Fix roles/users view field name mapping to match server response |
New |
| M3 |
xcecli |
Fix XCEHost.NumCPUs json tag: "num_cores" → "num_cpus" |
Existing |
| A1 |
xcecli |
Fix token header: standardize to "Bearer "+token in all request methods |
Existing |
| S4 |
apiserver |
Fix whitelist pattern truncation bug |
Existing |
🟠 High — security / usability
| ID |
Component |
Task |
Source |
| T6 |
apiserver |
Fix rbac.yaml: add job:view with all domain to cluster_admin role |
New |
| T7 |
xcecli |
Make xce logout --url optional, default to active server |
New |
| T8 |
xcecli |
Fix jobs submit -f file path parsing (plain path triggers data spec error) |
New |
| S1 |
apiserver |
Move JWT secret out of config/app.yaml into environment variable |
Existing |
| S3 |
apiserver |
Remove hardcoded mock RBAC maps from rbac_check.go |
Existing |
| A3 |
xcecli |
Fix TLS: wire loadRootCAs, only use InsecureSkipVerify when opted in |
Existing |
🟡 Missing features
| ID |
Component |
Task |
Source |
| T9 |
xcecli |
Add xce hosts select command (POST /xce/v1/hosts/select) |
New |
| T10 |
xcecli |
Add xce files list and xce files download commands |
New |
| F1 |
server+cli |
Implement file delete endpoint + xce files delete command |
Existing |
| F2 |
server+cli |
Decide on API key management: implement or remove dead code |
Existing |
| F3 |
xcecli |
Add array job stats display (?type=array_stats) |
Existing |
| F4 |
apiserver |
Enable CORS middleware |
Existing |
🔵 Cleanup
| ID |
Component |
Task |
Source |
| Q1 |
apiserver |
Regenerate Swagger docs (swag init) |
Existing |
| Q2 |
apiserver |
Fix validateRequestBody inverted condition |
Existing |
| Q3 |
apiserver |
Replace ioutil.ReadFile with os.ReadFile |
Existing |
| Q4 |
apiserver |
Remove unused AlecAivazis/survey dependency |
Existing |
| Q5 |
xcecli |
Upgrade resty.dev/v3 beta to stable go-resty/resty/v2 |
Existing |
| Q6 |
apiserver |
Fix rate limiter map unbounded growth |
Existing |
| Q7 |
xcecli |
Add retry logic for transient network errors |
Existing |
