Merge Go frontend (apiserver + xcecli) into meson build
This commit is contained in:
parent
43dc12a36c
commit
c9dc8c4676
205 changed files with 30793 additions and 328 deletions
244
src/frontend/docs/analysis.md
Normal file
244
src/frontend/docs/analysis.md
Normal file
|
|
@ -0,0 +1,244 @@
|
|||
# API Server & Client Analysis
|
||||
|
||||
> Generated: 2026-04-07
|
||||
> Scope: `apiserver/` ↔ `xcecli/` communication layer only.
|
||||
> Backend integration is excluded.
|
||||
|
||||
---
|
||||
|
||||
## 1. Component Overview
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
subgraph xcecli
|
||||
CMD["cobra commands"]
|
||||
APICLIENT["APIClient\nresty.dev/v3"]
|
||||
CFG["~/.xce/config.json\ntoken · url · expiry"]
|
||||
CMD --> APICLIENT
|
||||
APICLIENT --> CFG
|
||||
end
|
||||
|
||||
subgraph apiserver
|
||||
MW["middleware stack\nIP whitelist → URL whitelist\n→ rate limit → JWT validate"]
|
||||
ROUTER["Gin router\n/xce/v1/..."]
|
||||
CTRL["controllers"]
|
||||
STORE["in-memory DataStore\n(sync.RWMutex)"]
|
||||
MW --> ROUTER --> CTRL --> STORE
|
||||
end
|
||||
|
||||
APICLIENT -- "HTTP/HTTPS Bearer JWT" --> MW
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. API Completeness
|
||||
|
||||
### 2.1 Endpoint Coverage
|
||||
|
||||
All routes registered in `router/routers.go` have a corresponding CLI command. The table below marks the few gaps.
|
||||
|
||||
| Endpoint | Server | CLI | Gap |
|
||||
|---|:---:|:---:|---|
|
||||
| POST `/xce/v1/auth/logon` | ✅ | ✅ | — |
|
||||
| POST `/xce/v1/auth/logout` | ✅ | ✅ | — |
|
||||
| GET `/xce/v1/permissions` | ✅ | ✅ | — |
|
||||
| GET/POST/DELETE `/xce/v1/roles/...` | ✅ | ✅ | — |
|
||||
| GET/POST/DELETE `/xce/v1/users/...` | ✅ | ✅ | — |
|
||||
| GET `/xce/v1/clusters` | ✅ | ✅ | — |
|
||||
| GET/POST `/xce/v1/hosts/...` | ✅ | ✅ | — |
|
||||
| GET/POST `/xce/v1/queues/...` | ✅ | ✅ | — |
|
||||
| GET/POST `/xce/v1/jobs/...` | ✅ | ✅ | — |
|
||||
| GET `/xce/v1/resources/...` | ✅ | ✅ | — |
|
||||
| POST `/xce/v1/files/upload` | ✅ | ✅ | — |
|
||||
| GET `/xce/v1/files` | ✅ | ✅ | — |
|
||||
| GET `/xce/v1/files/:uuid/:filename` | ✅ | ✅ | — |
|
||||
| DELETE `/xce/v1/files` | ❌ commented out | ❌ no command | Route removed from server but still in URL whitelist |
|
||||
| API key CRUD | ❌ model exists, no route | ❌ no command | `models/apikey.go` + `controllers/apikey.go` are dead code |
|
||||
| `?type=array_stats` on jobs | ✅ | ⚠️ partial | Server supports it; CLI has no dedicated display path for array stats |
|
||||
|
||||
### 2.2 Missing Functional Areas
|
||||
|
||||
- **File delete** — commented out on server, no CLI command. Either implement or remove the dead code and whitelist entry.
|
||||
- **API key management** — controller and model exist but are unreachable. Wire a route or delete the files.
|
||||
- **CORS** — middleware is commented out in `router/routers.go`. Any web-based client will be blocked.
|
||||
|
||||
---
|
||||
|
||||
## 3. Client–Server Communication Gaps
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant CLI as xcecli
|
||||
participant SRV as apiserver
|
||||
|
||||
CLI->>SRV: POST /xce/v1/auth/logon
|
||||
SRV-->>CLI: {code, msg, data:{token, token_expire_time, ...}}
|
||||
Note over CLI: stores token in ~/.xce/config.json
|
||||
|
||||
CLI->>SRV: GET /xce/v1/jobs Authorization: Bearer <token>
|
||||
Note over SRV: JWT validated, IP checked
|
||||
SRV-->>CLI: {code, msg, data:{count, rows:[], page_token}}
|
||||
|
||||
CLI->>SRV: POST /xce/v1/jobs/create {command, job_name, ...}
|
||||
SRV-->>CLI: {code, msg, data:{jobid, queue_name}}
|
||||
```
|
||||
|
||||
### 3.1 Type Mismatches (will cause silent data loss or parse errors)
|
||||
|
||||
| # | Field | Server type | Client type | Effect |
|
||||
|---|---|---|---|---|
|
||||
| **M1** | `QueueInfo.queue_status` | `int` (bitmask) | `string` in `DBQueueInfo` | JSON unmarshal error — queue status always blank |
|
||||
| **M2** | `JobInfo.exit_status` | `Jsonb` (raw JSON string) | `int` in `QueryJob` | Parse error — exit status always 0 |
|
||||
| **M3** | `HostInfo.num_cpus` | field `num_cpus` | mapped to `num_cores` in `XCEHost` | CPU count always 0 in CLI output |
|
||||
| **M4** | `ResourceDistributionInfo` | separate fields: `numeric_value`, `boolean_value`, `string_value` | `ResourceDist.ResourceValue` reads from `"value"` key | Resource values always empty |
|
||||
| **M5** | `JobSubmitRequest.job_name` | `omitempty`, not required | `binding:"required"` | Client enforces job name; server does not — inconsistent validation |
|
||||
| **M6** | `JobControlJobIDRequest` | has `method` field (checkpoint method) | no `method` field | Checkpoint method cannot be set from CLI |
|
||||
|
||||
### 3.2 Token / Auth Issues
|
||||
|
||||
| # | Issue | Location | Effect |
|
||||
|---|---|---|---|
|
||||
| **A1** | Leading space in Authorization header | `request.go` `SubmitJob`: `" " + token` vs `GetRequest`: `"" + token` | Inconsistent — some endpoints may reject the token |
|
||||
| **A2** | Token expiry checked client-side only | `config.go` `IsAuthenticated()` | Server-side revocation (logout) not reflected until client's local timer expires |
|
||||
| **A3** | `SetRootCAs` always sets `InsecureSkipVerify: true` | `request.go:SetRootCAs` | TLS cert is never actually verified regardless of `certPath` |
|
||||
| **A4** | JWT IP binding | `validatehandler.go` `isValidToken` | Token tied to login IP — fails behind NAT/proxy or on IP change |
|
||||
|
||||
### 3.3 Response Envelope Inconsistency
|
||||
|
||||
The server wraps all responses in `{code, msg, data}`. Most client models match this, but:
|
||||
|
||||
- `LogonResponse` on the server returns `data` as a flat object (`token`, `path`, `ip`, `token_expire_time`).
|
||||
The client's `LogonResponse` wraps it in a `LogonData` sub-struct — this matches, but `token_expire_time` is `int` on server and `int64` on client. Works in practice but is fragile.
|
||||
- `ReturnError` in `response.go` calls `ctx.JSON(code, ...)` using the error code as the HTTP status. The client checks `httpResp.StatusCode() != http.StatusOK` — this is correct, but means any non-200 internal code (e.g. `500`, `400`) is also the HTTP status, which is actually good practice. Just needs to be documented.
|
||||
|
||||
---
|
||||
|
||||
## 4. Server-Side Issues (apiserver only)
|
||||
|
||||
### 4.1 Security
|
||||
|
||||
| # | Issue | File | Severity |
|
||||
|---|---|---|---|
|
||||
| **S1** | JWT secret hardcoded in config file | `config/app.yaml` `secretKey: "xce-api-key"` | 🔴 High — anyone with the file can forge tokens |
|
||||
| **S2** | Token stored as plain file on disk | `controllers/auth.go` `storeToken()` | 🟠 Medium — long-lived (86400s), not cleared on restart |
|
||||
| **S3** | Hardcoded mock RBAC group maps | `controllers/rbac_check.go` lines 19–31 | 🟠 Medium — test fixtures in production code affect access control |
|
||||
| **S4** | Whitelist pattern truncation bug | `middleware/whitelist.go` `pattern[:len(pattern)-1]` | 🟠 Medium — strips last char from all patterns, not just `*`-suffixed ones; can block or allow wrong paths |
|
||||
|
||||
### 4.2 Correctness
|
||||
|
||||
| # | Issue | File | Detail |
|
||||
|---|---|---|---|
|
||||
| **C1** | `validateRequestBody` condition is inverted | `middleware/validatehandler.go` line ~30 | `strings.Contains("multipart/form-data", contentType)` checks if the literal string contains the header value, not the other way around — always `false`. Works by accident. |
|
||||
| **C2** | Rate limiter `limiters` map grows unboundedly | `middleware/rate_limit.go` | New entries added per unique route path, never evicted |
|
||||
| **C3** | `ioutil.ReadFile` (deprecated Go 1.16) | `controllers/rbac_check.go` | Replace with `os.ReadFile` |
|
||||
| **C4** | `AlecAivazis/survey` in server `go.mod` | `go.mod` | Interactive prompt library has no use in a REST server — dead dependency |
|
||||
|
||||
### 4.3 Swagger Docs Are Stale
|
||||
|
||||
`docs/swagger.yaml` references:
|
||||
- Old host: `localhost:8443`
|
||||
- Old path: `/api/v1/hosts`
|
||||
- Old models: `models.BasicHost`, `models.FullHost`
|
||||
|
||||
None of these exist in the current codebase. The Swagger UI will show completely wrong information.
|
||||
|
||||
---
|
||||
|
||||
## 5. Client-Side Issues (xcecli only)
|
||||
|
||||
| # | Issue | File | Detail |
|
||||
|---|---|---|---|
|
||||
| **CL1** | `resty.dev/v3` is a beta release | `go.mod` | Not production-stable; consider `github.com/go-resty/resty/v2` |
|
||||
| **CL2** | TLS verification disabled | `request.go` `SetRootCAs` | `InsecureSkipVerify: true` always set; `loadRootCAs` is defined but never called |
|
||||
| **CL3** | Token header space inconsistency | `request.go` | `SubmitJob` uses `" "+token`; all other methods use `""+token` |
|
||||
| **CL4** | No retry on transient errors | `request.go` | Single attempt; network blips cause immediate failure |
|
||||
| **CL5** | `xce jobs` array stats has no display | `cmd/jobs/list.go` | `?type=array_stats` is sent but output formatter doesn't handle `ArrayJobStatsData` |
|
||||
|
||||
---
|
||||
|
||||
## 6. Prioritized Fix List
|
||||
|
||||
```mermaid
|
||||
quadrantChart
|
||||
title Fix Priority (Impact vs Effort)
|
||||
x-axis Low Effort --> High Effort
|
||||
y-axis Low Impact --> High Impact
|
||||
quadrant-1 Do First
|
||||
quadrant-2 Plan
|
||||
quadrant-3 Later
|
||||
quadrant-4 Quick Wins
|
||||
M1 QueueStatus type: [0.15, 0.90]
|
||||
M2 ExitStatus type: [0.15, 0.85]
|
||||
M3 num_cpus mapping: [0.10, 0.70]
|
||||
M4 Resource value fields: [0.20, 0.75]
|
||||
A1 Token header space: [0.05, 0.60]
|
||||
S1 JWT secret in config: [0.15, 0.95]
|
||||
S3 Mock RBAC data: [0.20, 0.80]
|
||||
S4 Whitelist pattern bug: [0.15, 0.75]
|
||||
A3 TLS InsecureSkipVerify: [0.25, 0.65]
|
||||
C1 validateRequestBody: [0.10, 0.40]
|
||||
C3 ioutil deprecated: [0.05, 0.20]
|
||||
CL1 resty beta: [0.30, 0.50]
|
||||
Swagger regen: [0.35, 0.55]
|
||||
CORS enable: [0.10, 0.45]
|
||||
```
|
||||
|
||||
### Ordered Action Items
|
||||
|
||||
**Do immediately (bugs that break functionality):**
|
||||
|
||||
1. **M1** — Change `DBQueueInfo.QueueStatus` from `string` to `int` in xcecli, add a `QueueStateStr()` helper for display
|
||||
2. **M2** — Change `QueryJob.ExitStatus` from `int` to `string` (or parse the Jsonb on server before sending)
|
||||
3. **M3** — Fix `XCEHost.NumCPUs` json tag from `"num_cores"` to `"num_cpus"`
|
||||
4. **M4** — Align resource distribution response: server should return a single `value` field, or client should read the typed fields
|
||||
5. **A1** — Standardize token header to `"Bearer "+token` in all `request.go` methods
|
||||
6. **S4** — Fix whitelist pattern matching: only strip trailing `*`, not all patterns
|
||||
|
||||
**Do before any production deployment:**
|
||||
|
||||
7. **S1** — Load JWT secret from environment variable, not config file
|
||||
8. **S3** — Remove hardcoded `hostGroupMap`, `userGroupMap`, `queueMap` from `rbac_check.go`
|
||||
9. **A3** — Wire `loadRootCAs` properly; only use `InsecureSkipVerify` when explicitly opted in
|
||||
10. **Swagger** — Run `swag init` after fixing controller annotations; add to CI
|
||||
|
||||
**Cleanup / low risk:**
|
||||
|
||||
11. **C1** — Fix `validateRequestBody` condition (currently works by accident)
|
||||
12. **C3** — Replace `ioutil.ReadFile` with `os.ReadFile`
|
||||
13. **C4** — Remove `AlecAivazis/survey` from server `go.mod`
|
||||
14. **CL1** — Upgrade resty to stable `v2`
|
||||
15. **File delete** — Implement or remove the dead route + whitelist entry
|
||||
|
||||
---
|
||||
|
||||
## 7. Dependencies Summary
|
||||
|
||||
### apiserver
|
||||
|
||||
| Package | Purpose | Note |
|
||||
|---|---|---|
|
||||
| `gin-gonic/gin` v1.10 | HTTP framework | stable |
|
||||
| `golang-jwt/jwt/v4` v4.5.1 | JWT auth | stable |
|
||||
| `google/uuid` v1.6 | File upload UUIDs | stable |
|
||||
| `lib/pq` v1.10.9 | PostgreSQL driver | in go.mod, unused (store is in-memory) |
|
||||
| `msteinert/pam` v1.2 | PAM auth (Linux) | build-tag guarded |
|
||||
| `spf13/viper` v1.19 | Config loading | stable |
|
||||
| `swaggo/gin-swagger` v1.6 | Swagger UI | docs are stale |
|
||||
| `xeipuuv/gojsonschema` v1.2 | Config validation | stable |
|
||||
| `uber-go/zap` v1.27 | Logging | stable |
|
||||
| `golang.org/x/time` v0.9 | Rate limiter | stable |
|
||||
| `lumberjack.v2` v2.2.1 | Log rotation | stable |
|
||||
| `AlecAivazis/survey/v2` v2.3.7 | Interactive prompts | **unused in server — remove** |
|
||||
|
||||
### xcecli
|
||||
|
||||
| Package | Purpose | Note |
|
||||
|---|---|---|
|
||||
| `spf13/cobra` v1.9.1 | CLI framework | stable |
|
||||
| `spf13/viper` v1.20.1 | Config loading | stable |
|
||||
| `xeipuuv/gojsonschema` v1.2 | Config validation | stable |
|
||||
| `uber-go/zap` v1.27 | Logging | stable |
|
||||
| `golang.org/x/term` v0.27 | Password input | stable |
|
||||
| `lumberjack.v2` v2.2.1 | Log rotation | stable |
|
||||
| `resty.dev/v3` v3.0.0-beta.2 | HTTP client | **⚠️ beta — replace with go-resty/resty/v2** |
|
||||
| `gopkg.in/yaml.v2` v2.4 | YAML parsing | stable |
|
||||
BIN
src/frontend/docs/analysis.pdf
Normal file
BIN
src/frontend/docs/analysis.pdf
Normal file
Binary file not shown.
163
src/frontend/docs/improvement_suggestions.md
Normal file
163
src/frontend/docs/improvement_suggestions.md
Normal file
|
|
@ -0,0 +1,163 @@
|
|||
# Code Improvement Suggestions
|
||||
|
||||
> Date: 2026-04-09
|
||||
> Based on current codebase state after all bug fixes.
|
||||
|
||||
---
|
||||
|
||||
## 1. Critical Bugs Still Present
|
||||
|
||||
### 1.1 `queue_name` filter added N times in `GetJobsInfo` (apiserver)
|
||||
|
||||
**File:** `controllers/jobs.go` ~line 226
|
||||
**Problem:** The `queue_name:in:(...)` RBAC filter is appended inside the per-filter loop — once for every non-queue filter in the request. With 2 filters (e.g. `job_status` + `submit_user`), it gets added twice. Seen in server logs: `queue_name:in:(normal,priority,short)` appears twice in every query.
|
||||
|
||||
**Fix:** Move the `accessQueues` append **outside** the loop, after it completes.
|
||||
|
||||
```go
|
||||
// BEFORE (inside loop — adds queue_name once per non-queue filter)
|
||||
for i, filter := range filters {
|
||||
...
|
||||
} else {
|
||||
if len(accessQueues) > 0 {
|
||||
filters = append(filters, ...) // BUG: runs N times
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// AFTER (outside loop — adds once)
|
||||
for i, filter := range filters { ... }
|
||||
if len(accessQueues) > 0 && !queueFilterPresent {
|
||||
filters = append(filters, utils.FilterCondition{
|
||||
Field: "queue_name", Operator: "in",
|
||||
Value: "(" + strings.Join(accessQueues, ",") + ")",
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Design Issues
|
||||
|
||||
### 2.1 `controllers/jobs.go` is 3590 lines — needs splitting
|
||||
|
||||
The file handles job query, submit, control, restart, array jobs, and verbose output. Each is a distinct concern.
|
||||
|
||||
**Suggested split:**
|
||||
```
|
||||
controllers/
|
||||
jobs_query.go — GetJobsInfo, GetJobByIDs (~600 lines)
|
||||
jobs_submit.go — CreateJob, RestartJobByID (~500 lines)
|
||||
jobs_control.go — ControlJob, ControlJobByID (~400 lines)
|
||||
jobs_array.go — array job handling (~300 lines)
|
||||
jobs_helpers.go — shared: parseArrayName, processJobData, etc.
|
||||
```
|
||||
|
||||
### 2.2 RBAC permission check duplicated across controllers
|
||||
|
||||
`GetRequireResourcePermission` + `accessQueues` logic is copy-pasted in `GetJobsInfo`, `ControlJob`, `ControlJobByID`, `RestartJobByID`. Extract to a helper:
|
||||
|
||||
```go
|
||||
// controllers/rbac_helpers.go
|
||||
func getJobAccessQueues(username string) ([]string, error) { ... }
|
||||
func checkJobViewPermission(username, queueName string) error { ... }
|
||||
```
|
||||
|
||||
### 2.3 `xcecli/cmd/jobs/submit.go` is 734 lines
|
||||
|
||||
`JobSubmitter` mixes CLI flag parsing, file upload, interactive job handling, and HTTP submission. Split:
|
||||
|
||||
```
|
||||
cmd/jobs/
|
||||
submit.go — cobra command + flag binding only
|
||||
submitter.go — JobSubmitter struct + BuildJobRequest
|
||||
submit_files.go — file upload logic (already in files.go ✓)
|
||||
submit_interactive.go — interactive/TTY/xjob handling
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Code Quality
|
||||
|
||||
### 3.1 `ioutil.ReadFile` deprecated (Go 1.16)
|
||||
|
||||
**File:** `controllers/rbac_check.go:54`
|
||||
**Fix:** `os.ReadFile(path)` — one-line change, tracked as Q3.
|
||||
|
||||
### 3.2 Inconsistent error return convention
|
||||
|
||||
Some functions return `(error, T)`, others return `(T, error)`. Go convention is always `(T, error)`.
|
||||
|
||||
```go
|
||||
// Non-standard (apiserver/controllers/rbac_check.go)
|
||||
func GetRequireResourcePermission(...) (error, models.RolePermission)
|
||||
|
||||
// Should be
|
||||
func GetRequireResourcePermission(...) (models.RolePermission, error)
|
||||
```
|
||||
|
||||
This affects ~8 call sites in `jobs.go` that do `err, perm := GetRequire...`.
|
||||
|
||||
### 3.3 Magic strings for domain types
|
||||
|
||||
`"all"`, `"self"`, `"self_ug"`, `"queue"` appear as raw strings in multiple files. They're defined as constants in `models/roles.go` but not always used.
|
||||
|
||||
**Fix:** Replace all raw string comparisons with the constants (`models.DomainTypeAll`, etc.).
|
||||
|
||||
### 3.4 `xcecli` output printers have no shared time formatting
|
||||
|
||||
`page_print_jobs.go` now has `parseJobTime`/`formatJobTime`. The same pattern will be needed for `start_time`, `end_time` in other printers. Move to `common/output/time.go`.
|
||||
|
||||
---
|
||||
|
||||
## 4. Architecture
|
||||
|
||||
### 4.1 In-memory store has no persistence
|
||||
|
||||
After restart, all submitted jobs (non-builtin) are lost. The `lib/pq` PostgreSQL driver is already in `go.mod`. When backend integration happens, the notification handlers (`OnJobCreated`, etc.) should write through to PostgreSQL, not just the in-memory store.
|
||||
|
||||
### 4.2 `UserRoleSeeder` only implemented for `config_file`
|
||||
|
||||
After the startup fix, PAM/LDAP/OAuth2 providers don't implement `AllUsers()` — so after restart with those providers, users still need to re-login to get their roles. Consider storing user-role assignments in a persistent store (DB or file) so they survive restarts regardless of provider.
|
||||
|
||||
### 4.3 Rate limiter `limiters` map grows unboundedly
|
||||
|
||||
**File:** `middleware/rate_limit.go`
|
||||
The per-endpoint `limiters` map is never evicted. Under normal operation with fixed routes this is fine, but worth noting for future dynamic routing.
|
||||
|
||||
---
|
||||
|
||||
## 5. Priority Order
|
||||
|
||||
```mermaid
|
||||
quadrantChart
|
||||
title Impact vs Effort
|
||||
x-axis Low Effort --> High Effort
|
||||
y-axis Low Impact --> High Impact
|
||||
quadrant-1 Do First
|
||||
quadrant-2 Plan
|
||||
quadrant-3 Later
|
||||
quadrant-4 Quick Wins
|
||||
queue_name double-add: [0.10, 0.90]
|
||||
ioutil fix: [0.05, 0.20]
|
||||
error convention: [0.25, 0.60]
|
||||
split jobs.go: [0.60, 0.70]
|
||||
split submit.go: [0.50, 0.50]
|
||||
RBAC helper: [0.35, 0.65]
|
||||
time.go shared: [0.15, 0.35]
|
||||
persistence: [0.80, 0.85]
|
||||
UserRoleSeeder all providers: [0.30, 0.55]
|
||||
```
|
||||
|
||||
**Do immediately:**
|
||||
1. Fix `queue_name` double-add bug in `GetJobsInfo` (10 min, high impact)
|
||||
2. `ioutil` → `os.ReadFile` (1 min)
|
||||
|
||||
**Next sprint:**
|
||||
3. Fix `(error, T)` → `(T, error)` return convention
|
||||
4. Extract RBAC helpers to reduce duplication in `jobs.go`
|
||||
5. Split `jobs.go` into focused files
|
||||
|
||||
**Backend integration phase:**
|
||||
6. PostgreSQL persistence via notification handlers
|
||||
7. `UserRoleSeeder` for PAM/LDAP providers
|
||||
16
src/frontend/docs/improvement_tasks.md
Normal file
16
src/frontend/docs/improvement_tasks.md
Normal file
|
|
@ -0,0 +1,16 @@
|
|||
# Code Improvement Tasks
|
||||
|
||||
> Last updated: 2026-04-09
|
||||
|
||||
| ID | Status | Category | Component | Task | Effort |
|
||||
|---|---|---|---|---|---|
|
||||
| I1 | ✅ done | 🔴 Bug | apiserver | `queue_name:in:(...)` filter added N times in `GetJobsInfo` loop | 10 min |
|
||||
| I2 | ✅ done | 🟠 Quality | apiserver | Fix `(error, T)` → `(T, error)` return convention in `GetRequireResourcePermission` and call sites | 1h |
|
||||
| I3 | ⏳ open | 🟠 Quality | apiserver | Replace `ioutil.ReadFile` with `os.ReadFile` in `rbac_check.go` | 5 min |
|
||||
| I4 | ⏳ open | 🟠 Quality | apiserver | Replace raw domain strings (`"all"`, `"self"`) with `models.DomainTypeAll` constants | 30 min |
|
||||
| I5 | ⏳ open | 🟠 Quality | xcecli | Move `parseJobTime`/`formatJobTime` to `common/output/time.go` for reuse | 15 min |
|
||||
| I6 | ⏳ open | 🟡 Refactor | apiserver | Extract RBAC `accessQueues` logic — duplicated in `GetJobsInfo`, `ControlJob`, `ControlJobByID`, `RestartJobByID` | 2h |
|
||||
| I7 | ⏳ open | 🟡 Refactor | apiserver | Split `controllers/jobs.go` (3590 lines) into query / submit / control / array / helpers | 4h |
|
||||
| I8 | ⏳ open | 🟡 Refactor | xcecli | Split `cmd/jobs/submit.go` (734 lines) — cobra command / BuildJobRequest / interactive | 2h |
|
||||
| I10 | ⏳ open | 🔵 Arch | apiserver | `UserRoleSeeder` for PAM/LDAP — users need re-login after restart with non-config_file providers | 2h |
|
||||
| I11 | ⏳ open | 🔵 Arch | apiserver | Rate limiter `limiters` map grows unboundedly — add eviction or use fixed route keys | 1h |
|
||||
278
src/frontend/docs/integration_test.md
Normal file
278
src/frontend/docs/integration_test.md
Normal file
|
|
@ -0,0 +1,278 @@
|
|||
# Integration Test Report: xcecli ↔ apiserver
|
||||
|
||||
> Date: 2026-04-08
|
||||
> Server: `apiserver` built from `apiserver/` with builtin test data (3 hosts, 3 queues, 4 jobs, 8 resources)
|
||||
> Client: `xcecli` built from `xcecli/` (CGO_ENABLED=0 static binary)
|
||||
> Auth: `config_file` provider — users: `admin` (cluster_admin), `user1`, `user2` (basic_user)
|
||||
|
||||
> **Updated: 2026-04-08** — All critical bugs fixed. See resolution status table below.
|
||||
|
||||
---
|
||||
|
||||
## Summary (at time of initial test)
|
||||
|
||||
| Category | Count |
|
||||
|---|---|
|
||||
| ✅ Commands working correctly | 13 |
|
||||
| ⚠️ Commands partially working (wrong/empty data) | 11 |
|
||||
| ❌ Commands totally broken | 3 |
|
||||
| Server endpoints not covered by CLI | 2 |
|
||||
|
||||
## Summary (after fixes)
|
||||
|
||||
| Category | Count |
|
||||
|---|---|
|
||||
| ✅ Commands working correctly | ~27 |
|
||||
| ⚠️ Minor issues remaining | 2 (clusters empty — no backend; jobs list no filter = current user only, by design) |
|
||||
| ❌ Broken | 0 |
|
||||
|
||||
---
|
||||
|
||||
## Test Results by Command
|
||||
|
||||
### Auth
|
||||
|
||||
| Command | Result | Notes |
|
||||
|---|---|---|
|
||||
| `xce login -u admin -p admin --url http://...` | ✅ Works | Token saved to `~/.xce/config.json` |
|
||||
| `xce logout --url http://...` | ✅ Works | Token cleared |
|
||||
| `xce list` | ✅ Works | Shows server URL and default marker |
|
||||
| `xce users whoami` | ✅ Works | Returns current username from token |
|
||||
|
||||
**Bug found:** `xce logout` requires `--url` flag — if omitted it errors with `required flag(s) "url" not set`. Should default to the active server.
|
||||
|
||||
---
|
||||
|
||||
### Permissions
|
||||
|
||||
| Command | Result | Notes |
|
||||
|---|---|---|
|
||||
| `xce permissions list` | ✅ Works | Returns all 21 permissions correctly |
|
||||
|
||||
---
|
||||
|
||||
### Roles
|
||||
|
||||
| Command | Result | Notes |
|
||||
|---|---|---|
|
||||
| `xce roles list` | ❌ Broken | Server rejects with `invalid role info query parameters of : unique` |
|
||||
| `xce roles view -r <rolename>` | ⚠️ Partial | Request succeeds, returns correct row count, but all fields display as `-` |
|
||||
| `xce roles add -r <name> -p "job:view=all"` | ✅ Works | Role created successfully |
|
||||
| `xce roles delete -r <name>` | ✅ Works | Role deleted successfully |
|
||||
|
||||
**Root cause — `roles list`:** Client sends `params["unique"] = "true"` as a query parameter. Server's `SupportedQueryParams` does not include `unique` → rejected with 400.
|
||||
|
||||
**Root cause — `roles view` empty data:** Server returns `role_name`, `action`, `domain_type`, `domain_values` fields. Client `RolesResponse` model maps to `models.RoleInfo` which has different field names — JSON unmarshal silently produces zero values.
|
||||
|
||||
---
|
||||
|
||||
### Users
|
||||
|
||||
| Command | Result | Notes |
|
||||
|---|---|---|
|
||||
| `xce users list` | ❌ Broken | Server rejects with `invalid user_role info query parameters of : unique` |
|
||||
| `xce users view -u <username>` | ⚠️ Partial | Returns correct row count, all fields display as `-` |
|
||||
| `xce users whoami` | ✅ Works | Returns `admin` |
|
||||
| `xce users assignrole -u user1 -r basic_user` | ✅ Works | Role assigned |
|
||||
| `xce users unassignrole -u user1 -r basic_user` | ✅ Works | Role removed |
|
||||
| `xce users permissions4user -u admin` | ⚠️ Partial | Returns correct row count, all fields display as `-` |
|
||||
|
||||
**Root cause — `users list`:** Same `unique` query param issue as `roles list`.
|
||||
|
||||
**Root cause — `users view` / `permissions4user` empty data:** Same field name mismatch as roles view.
|
||||
|
||||
---
|
||||
|
||||
### Hosts
|
||||
|
||||
| Command | Result | Notes |
|
||||
|---|---|---|
|
||||
| `xce hosts list` | ⚠️ Partial | Returns 3 rows (correct count), but `HOST_NAME`, `STATUS`, `HOST_TYPE` all show `-`; `NCPUS` shows `0` |
|
||||
| `xce hosts open <hostname>` | ✅ Works | `host <host1> is being <open>` |
|
||||
| `xce hosts close <hostname>` | ✅ Works | `host <host1> is being <close>` |
|
||||
|
||||
**Root cause — `hosts list` empty fields:**
|
||||
1. `HostStatus` is `int` on server, `string` in client `XCEHost` → JSON unmarshal fails silently, `HostStatus = ""`
|
||||
2. `XCEHost.NumCPUs` has json tag `"num_cores"` but server sends `"num_cpus"` → always 0
|
||||
3. `HostName` and `HostType` are present in server response but `HostsResponse.Data.Rows` is `[]XCEHost` — the field names match (`host_name`, `host_type`) so these should work. Actual cause: server serializes `HostInfo` via `StructToOrderMap` using reflect + json tags, then re-serializes to JSON. The `HostStatus int` becomes a JSON number; client `XCEHost.HostStatus string` cannot unmarshal a number → entire row decode fails, leaving all string fields empty.
|
||||
|
||||
---
|
||||
|
||||
### Queues
|
||||
|
||||
| Command | Result | Notes |
|
||||
|---|---|---|
|
||||
| `xce queues list` | ⚠️ Partial | Returns 3 rows (correct count), all fields show `-` or `0` |
|
||||
| `xce queues open <name>` | ✅ Works | `queue <normal> is being <open>` |
|
||||
| `xce queues close <name>` | ✅ Works | `queue <normal> is being <close>` |
|
||||
| `xce queues active <name>` | ✅ Works | `queue <normal> is being <active>` |
|
||||
| `xce queues deactive <name>` | ✅ Works | `queue <normal> is being <deactive>` |
|
||||
|
||||
**Root cause — `queues list` empty fields:** `QueueInfo.QueueStatus` is `int` on server, `string` in client `DBQueueInfo` → same unmarshal failure as hosts.
|
||||
|
||||
---
|
||||
|
||||
### Jobs
|
||||
|
||||
| Command | Result | Notes |
|
||||
|---|---|---|
|
||||
| `xce jobs list` (no filter) | ⚠️ Partial | Returns "no job found" — server filters by current user AND accessible queues; admin has no jobs |
|
||||
| `xce jobs list -u all` | ⚠️ Partial | Returns 3 rows (correct count for builtin data), but `JOBID=0`, all string fields `-` |
|
||||
| `xce jobs list -u user1` | ⚠️ Partial | Returns 2 rows, same display issue |
|
||||
| `xce jobs submit -J name -q normal <cmd>` | ✅ Works | Job ID returned, submitted to queue |
|
||||
| `xce jobs submit -q normal <cmd>` (no `-J`) | ✅ Works | Server accepts without job name (client `binding:"required"` not enforced server-side) |
|
||||
| `xce jobs submit -f <file> ...` | ⚠️ Partial | Job submitted but `parse file transfer config failed: [<path>]` printed — file upload path parsing error |
|
||||
| `xce jobs kill <jobid>` | ⚠️ Partial | Request reaches server, server finds job by `job_id` but submitted job IDs (Snowflake) don't match builtin IDs (1001-1004) → `job not found` |
|
||||
| `xce jobs stop <jobid>` | ⚠️ Partial | Same as kill |
|
||||
| `xce jobs resume <jobid>` | ⚠️ Partial | Same as kill |
|
||||
| `xce jobs checkpoint <jobid>` | ⚠️ Partial | Same as kill |
|
||||
| `xce jobs restart <jobid>` | ⚠️ Partial | Reaches server, but fails: `job not submitted by <user1>, cannot be restarted by <admin>` — correct RBAC behavior |
|
||||
|
||||
**Root cause — `jobs list` display issue:** `QueryJob.JobStatus` is `string` in client but server sends `int` (bitmask). `QueryJob.JobID` is `int64` but server sends `unique_job_id` as the primary key — client maps `job_id` which is the array-level ID (same as `unique_job_id` for non-array jobs in builtin data, but Snowflake-generated IDs differ). The `JobStatus` type mismatch causes the entire row JSON decode to fail.
|
||||
|
||||
**Root cause — `jobs list` no results for admin:** Server `GetJobsInfo` adds `submit_user = <logon_user>` filter unless user has `job:view` permission with `all` domain. Admin's RBAC domain is not configured for `all` in the default `rbac.yaml`.
|
||||
|
||||
**Root cause — `jobs submit -f` parse error:** `xcecli/cmd/jobs/submit.go` tries to parse the `-f` flag value as a data spec file format (`src > dst`) before treating it as a plain file path. Plain paths without `>` fail the parse but the job still submits.
|
||||
|
||||
---
|
||||
|
||||
### Clusters
|
||||
|
||||
| Command | Result | Notes |
|
||||
|---|---|---|
|
||||
| `xce id` | ⚠️ Partial | Returns "no cluster found" — builtin data has no cluster entries in store |
|
||||
|
||||
---
|
||||
|
||||
### Resources
|
||||
|
||||
| Command | Result | Notes |
|
||||
|---|---|---|
|
||||
| `xce resources distribution list` | ⚠️ Partial | Returns 16 rows (correct count), all values show `-` |
|
||||
| `xce resources definition list` | ⚠️ Partial | Returns 8 rows (correct count), all values show `-` |
|
||||
|
||||
**Root cause:** Server `ResourceDistributionInfo` has fields `numeric_value`, `boolean_value`, `string_value`. Client `ResourceDist` reads from a single `"value"` key via `ResourceValue.UnmarshalJSON`. Server never sends a `"value"` key → always empty. Same for definition: server sends `value_type` as `int`, client `ResourceDef.ResourceType` is `string`.
|
||||
|
||||
---
|
||||
|
||||
## Server Endpoints Not Covered by CLI
|
||||
|
||||
| Endpoint | Method | Notes |
|
||||
|---|---|---|
|
||||
| `/xce/v1/hosts/select` | POST | Filter hosts by resource requirement string (`-R select(...)`) — no CLI command |
|
||||
| `/xce/v1/files` | GET | List uploaded files — no `xce files list` command (file ops are embedded in `jobs submit`) |
|
||||
| `/xce/v1/files/:uuid/:filename` | GET | Download file — no `xce files download` command |
|
||||
| `/xce/v1/jobs/:jobids` | GET | Get specific jobs by ID — CLI uses the general list with filter instead |
|
||||
|
||||
---
|
||||
|
||||
## Root Cause Summary
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[Display shows all dashes / zeros] --> B{Type mismatch\nserver int → client string}
|
||||
B --> C[QueueStatus: int vs string]
|
||||
B --> D[HostStatus: int vs string]
|
||||
B --> E[JobStatus: int vs string]
|
||||
B --> F[ResourceType: int vs string]
|
||||
B --> G[ResourceValue: typed fields vs single value key]
|
||||
|
||||
H[roles list / users list broken] --> I[Client sends unique=true\nServer rejects unknown param]
|
||||
|
||||
J[roles view / users view / permissions4user\nshow dashes] --> K[Field name mismatch\nserver action/domain_type vs\nclient RoleInfo struct]
|
||||
|
||||
L[jobs list no results for admin] --> M[RBAC: admin role missing\njob:view with all domain]
|
||||
|
||||
N[jobs control not found] --> O[Snowflake job IDs don't\nmatch builtin test IDs 1001-1004]
|
||||
|
||||
P[jobs submit -f parse error] --> Q[File path parsed as\ndata spec format src>dst]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## New Bugs Found — Resolution Status
|
||||
|
||||
| ID | Severity | Component | Description | Status |
|
||||
|---|---|---|---|---|
|
||||
| T1 | 🔴 | xcecli | `roles list`/`users list`: remove `unique=true` param; deduplicate client-side | ✅ fixed |
|
||||
| T2 | 🔴 | xcecli | `QueueStatus`/`HostStatus`/`JobStatus` type mismatch | ✅ fixed (server sends strings; T3b root cause) |
|
||||
| T3 | 🔴 | xcecli | Resource distribution value fields | ✅ fixed (part of T3b) |
|
||||
| T3b | 🔴 | xcecli | All rows wrapped in `{Fields,Data}` OrderMap — `UnmarshalOrderMapRow` added | ✅ fixed |
|
||||
| T4 | 🔴 | xcecli | `ResourceDef` display alignment | ✅ verified OK |
|
||||
| T5 | 🔴 | xcecli | Column truncation in roles/users view | ✅ fixed |
|
||||
| T6 | 🟠 | apiserver | Admin `job:view` RBAC | ✅ already correct in rbac.yaml |
|
||||
| T7 | 🟠 | xcecli | `xce logout --url` optional | ✅ fixed |
|
||||
| T8 | 🟠 | xcecli | `jobs submit -f` plain path parse error | ✅ fixed |
|
||||
| T9 | 🟡 | xcecli | Add `xce hosts select` command | ⏳ pending |
|
||||
| T10 | 🟡 | xcecli | Add `xce files list`/`download` commands | ⏳ pending |
|
||||
| M2 | 🔴 | xcecli | `QueryJob.ExitStatus` type mismatch | ✅ fixed |
|
||||
| M3 | 🔴 | xcecli | `XCEHost.NumCPUs` json tag wrong | ✅ fixed (part of T3b) |
|
||||
| A1 | 🔴 | xcecli | Token header leading space | ✅ fixed |
|
||||
| A3 | 🟠 | xcecli | TLS `InsecureSkipVerify` always set | ✅ fixed |
|
||||
| S1 | 🔴 | apiserver | JWT secret hardcoded in config | ✅ fixed (env var override) |
|
||||
| S3 | 🟠 | apiserver | Hardcoded mock RBAC maps | ✅ fixed |
|
||||
| S4 | 🟠 | apiserver | Whitelist pattern truncation bug | ✅ fixed |
|
||||
|
||||
| ID | Severity | Component | Description |
|
||||
|---|---|---|---|
|
||||
| **T1** | 🔴 | xcecli | `roles list` / `users list`: remove `params["unique"] = "true"` — server rejects this param. Implement client-side deduplication instead |
|
||||
| **T2** | 🔴 | xcecli | `QueueStatus`, `HostStatus`, `JobStatus`: change client model fields from `string` → `int`, add display helper functions to convert bitmask to string |
|
||||
| **T3** | 🔴 | xcecli | `ResourceDist`: read `numeric_value`/`boolean_value`/`string_value` from server response instead of `"value"` |
|
||||
| **T4** | 🔴 | xcecli | `ResourceDef.ResourceType`: change from `string` → `int`, add display helper |
|
||||
| **T5** | 🔴 | xcecli | `roles view` / `users view` / `permissions4user`: fix field name mapping — server returns `action`, `domain_type`, `domain_values`; client struct must match |
|
||||
| **T6** | 🟠 | apiserver | `rbac.yaml` default config: `cluster_admin` role needs `job:view` with `all` domain so admin can see all jobs |
|
||||
| **T7** | 🟠 | xcecli | `xce logout`: make `--url` optional, default to active server URL from config |
|
||||
| **T8** | 🟠 | xcecli | `jobs submit -f`: fix file path parsing — plain paths (no `>`) should not trigger data spec parse error |
|
||||
| **T9** | 🟡 | xcecli | Add `xce hosts select` command to expose `POST /xce/v1/hosts/select` |
|
||||
| **T10** | 🟡 | xcecli | Add `xce files list` and `xce files download` commands |
|
||||
|
||||
---
|
||||
|
||||
## Updated Full Todo List
|
||||
|
||||
### 🔴 Critical — breaks functionality
|
||||
|
||||
| ID | Component | Task | Source |
|
||||
|---|---|---|---|
|
||||
| T1 | xcecli | Remove `unique=true` query param from roles/users list; deduplicate client-side | New |
|
||||
| T2 | xcecli | Fix `QueueStatus`/`HostStatus`/`JobStatus` type: `string` → `int` + display helpers | New (confirms M1, M2) |
|
||||
| T3 | xcecli | Fix resource distribution value fields: read `numeric_value`/`boolean_value`/`string_value` | New (confirms M4) |
|
||||
| T4 | xcecli | Fix `ResourceDef.ResourceType`: `string` → `int` + display helper | New |
|
||||
| T5 | xcecli | Fix roles/users view field name mapping to match server response | New |
|
||||
| M3 | xcecli | Fix `XCEHost.NumCPUs` json tag: `"num_cores"` → `"num_cpus"` | Existing |
|
||||
| A1 | xcecli | Fix token header: standardize to `"Bearer "+token` in all request methods | Existing |
|
||||
| S4 | apiserver | Fix whitelist pattern truncation bug | Existing |
|
||||
|
||||
### 🟠 High — security / usability
|
||||
|
||||
| ID | Component | Task | Source |
|
||||
|---|---|---|---|
|
||||
| T6 | apiserver | Fix `rbac.yaml`: add `job:view` with `all` domain to `cluster_admin` role | New |
|
||||
| T7 | xcecli | Make `xce logout --url` optional, default to active server | New |
|
||||
| T8 | xcecli | Fix `jobs submit -f` file path parsing (plain path triggers data spec error) | New |
|
||||
| S1 | apiserver | Move JWT secret out of `config/app.yaml` into environment variable | Existing |
|
||||
| S3 | apiserver | Remove hardcoded mock RBAC maps from `rbac_check.go` | Existing |
|
||||
| A3 | xcecli | Fix TLS: wire `loadRootCAs`, only use `InsecureSkipVerify` when opted in | Existing |
|
||||
|
||||
### 🟡 Missing features
|
||||
|
||||
| ID | Component | Task | Source |
|
||||
|---|---|---|---|
|
||||
| T9 | xcecli | Add `xce hosts select` command (`POST /xce/v1/hosts/select`) | New |
|
||||
| T10 | xcecli | Add `xce files list` and `xce files download` commands | New |
|
||||
| F1 | server+cli | Implement file delete endpoint + `xce files delete` command | Existing |
|
||||
| F2 | server+cli | Decide on API key management: implement or remove dead code | Existing |
|
||||
| F3 | xcecli | Add array job stats display (`?type=array_stats`) | Existing |
|
||||
| F4 | apiserver | Enable CORS middleware | Existing |
|
||||
|
||||
### 🔵 Cleanup
|
||||
|
||||
| ID | Component | Task | Source |
|
||||
|---|---|---|---|
|
||||
| Q1 | apiserver | Regenerate Swagger docs (`swag init`) | Existing |
|
||||
| Q2 | apiserver | Fix `validateRequestBody` inverted condition | Existing |
|
||||
| Q3 | apiserver | Replace `ioutil.ReadFile` with `os.ReadFile` | Existing |
|
||||
| Q4 | apiserver | Remove unused `AlecAivazis/survey` dependency | Existing |
|
||||
| Q5 | xcecli | Upgrade `resty.dev/v3` beta to stable `go-resty/resty/v2` | Existing |
|
||||
| Q6 | apiserver | Fix rate limiter map unbounded growth | Existing |
|
||||
| Q7 | xcecli | Add retry logic for transient network errors | Existing |
|
||||
151
src/frontend/docs/plan_t9_f1_f4.md
Normal file
151
src/frontend/docs/plan_t9_f1_f4.md
Normal file
|
|
@ -0,0 +1,151 @@
|
|||
# Implementation Plan: T9, F1, F4
|
||||
|
||||
> Created: 2026-04-08
|
||||
> Scope: Three remaining feature items before backend integration
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
| ID | Item | Effort | Order |
|
||||
|---|---|---|---|
|
||||
| F4 | CORS middleware (apiserver) | ~2h | 1st |
|
||||
| F1 | File delete (server route + CLI command) | ~2.5h | 2nd |
|
||||
| T9 | `xce hosts select` CLI command | ~3h | 3rd |
|
||||
|
||||
---
|
||||
|
||||
## F4 — Enable CORS Middleware
|
||||
|
||||
**Problem:** CORS middleware is commented out in `router/routers.go`. Any web-based client will be blocked by browser same-origin policy.
|
||||
|
||||
**Approach:** Inline CORS middleware (no new dependency), configurable via `app.yaml`.
|
||||
|
||||
### Files to change
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `apiserver/middleware/cors.go` | New: `CORSMiddleware(cfg CORSConfig)` |
|
||||
| `apiserver/utils/config.go` | Add `CORSConfig` struct + parse from viper |
|
||||
| `apiserver/config/app.yaml` | Add `cors:` section |
|
||||
| `apiserver/router/routers.go` | Replace commented block with `r.Use(middleware.CORSMiddleware(...))` |
|
||||
|
||||
### Implementation steps (TDD)
|
||||
|
||||
1. Write `TestCORSHeaders` — OPTIONS preflight returns `Access-Control-Allow-Origin`
|
||||
2. Implement `CORSMiddleware()` setting response headers
|
||||
3. Wire into router, add config to `app.yaml`
|
||||
|
||||
### Config shape
|
||||
|
||||
```yaml
|
||||
cors:
|
||||
enabled: true
|
||||
allowed_origins:
|
||||
- "*"
|
||||
allowed_methods:
|
||||
- "GET"
|
||||
- "POST"
|
||||
- "DELETE"
|
||||
- "OPTIONS"
|
||||
allowed_headers:
|
||||
- "Origin"
|
||||
- "Content-Type"
|
||||
- "Authorization"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## F1 — File Delete
|
||||
|
||||
**Problem:** `DeleteFiles` controller is fully implemented but the route is commented out. No CLI command exists.
|
||||
|
||||
### Server side
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `apiserver/router/routers.go` | Uncomment `filesApi.DELETE("", ...)` |
|
||||
|
||||
`DELETE /xce/v1/files` is already in `WhitelistDELETEURL` — no whitelist change needed.
|
||||
|
||||
### Client side
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `xcecli/cmd/files/files.go` | New: root `files` cobra command |
|
||||
| `xcecli/cmd/files/delete.go` | New: `xce files delete --path <url_or_path>` |
|
||||
| `xcecli/router/router.go` | Register `files.NewFilesCmd(configManager)` |
|
||||
|
||||
### CLI usage
|
||||
|
||||
```
|
||||
xce files delete --path http://server/xce/v1/files/uuid/filename
|
||||
xce files delete --path uuid/filename,uuid2/filename2
|
||||
```
|
||||
|
||||
### Implementation steps (TDD)
|
||||
|
||||
1. Write `TestRunFilesDelete_RequiresPath` — errors when `--path` missing
|
||||
2. Uncomment server route
|
||||
3. Implement `xce files delete` command
|
||||
4. Add regression test case to `tests/regression/run.sh`
|
||||
|
||||
---
|
||||
|
||||
## T9 — `xce hosts select`
|
||||
|
||||
**Problem:** `POST /xce/v1/hosts/select` exists on the server (returns empty list pre-backend) but has no CLI command.
|
||||
|
||||
**Server note:** `FilterHostByResReqSelect` is a stub — returns empty until backend integration. The CLI command should work end-to-end; it will return empty results until the backend is connected.
|
||||
|
||||
### Client side
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `xcecli/models/hosts.go` | Add `ResReqSelectRequest` struct |
|
||||
| `xcecli/cmd/hosts/select.go` | New: `xce hosts select -R "select_string"` |
|
||||
| `xcecli/cmd/hosts/hosts.go` | Register new subcommand |
|
||||
|
||||
### CLI usage
|
||||
|
||||
```
|
||||
xce hosts select -R "type==linux && ncpus>=8"
|
||||
xce hosts select -R "select(!mg)" --order "ncpus" --limit 10
|
||||
```
|
||||
|
||||
### ResReqSelectRequest model (mirrors server)
|
||||
|
||||
```go
|
||||
type ResReqSelectRequest struct {
|
||||
Select string `json:"select"`
|
||||
Order string `json:"order,omitempty"`
|
||||
Limit int `json:"limit,omitempty"`
|
||||
PageToken string `json:"page_token,omitempty"`
|
||||
}
|
||||
```
|
||||
|
||||
### Implementation steps (TDD)
|
||||
|
||||
1. Write `TestParseSelectArgs` — `-R` flag required, max 4094 chars
|
||||
2. Add `ResReqSelectRequest` to `xcecli/models/hosts.go`
|
||||
3. Implement `xce hosts select` — POST to `/xce/v1/hosts/select`, display with existing `HostsPagePrinter`
|
||||
4. Add regression test case
|
||||
|
||||
---
|
||||
|
||||
## Execution Order
|
||||
|
||||
```mermaid
|
||||
gantt
|
||||
title Implementation Schedule
|
||||
dateFormat HH:mm
|
||||
section F4 CORS
|
||||
Test + implement middleware :f4, 00:00, 2h
|
||||
section F1 File Delete
|
||||
Uncomment server route :f1a, after f4, 30m
|
||||
CLI command + test :f1b, after f1a, 2h
|
||||
section T9 Hosts Select
|
||||
Model + CLI command + test :t9, after f1b, 3h
|
||||
```
|
||||
|
||||
**Total estimated effort: ~7.5h**
|
||||
94
src/frontend/docs/status.md
Normal file
94
src/frontend/docs/status.md
Normal file
|
|
@ -0,0 +1,94 @@
|
|||
# Project Status
|
||||
|
||||
> Last updated: 2026-04-09
|
||||
|
||||
---
|
||||
|
||||
## Overall Progress
|
||||
|
||||
```mermaid
|
||||
pie title Component Completion
|
||||
"Done" : 99
|
||||
"Missing features" : 1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Integration Test Summary
|
||||
|
||||
Live test: `xcecli` against running `apiserver` with builtin test data.
|
||||
|
||||
| Status | Commands (after all fixes) |
|
||||
|---|---|
|
||||
| ✅ Working | `login`, `logout`, `list`, `whoami`, `permissions list`, `roles list/view/add/delete`, `users list/view/assignrole/unassignrole/permissions4user`, `hosts list/open/close`, `queues list/open/close/active/deactive`, `jobs submit/list/kill/stop/resume/checkpoint`, `resources distribution/definition list` |
|
||||
| ⚠️ Expected | `jobs list` (no filter = current user only, by design); `clusters list` (empty — no backend data) |
|
||||
| ❌ Broken | none |
|
||||
|
||||
---
|
||||
|
||||
## Done
|
||||
|
||||
### apiserver
|
||||
- [x] Gin HTTP server with config loading (Viper + JSON schema validation)
|
||||
- [x] JWT authentication — logon / logout, token stored server-side per user
|
||||
- [x] Middleware stack: IP whitelist → URL whitelist → rate limiter → JWT validator
|
||||
- [x] RBAC engine — roles, permissions, domain-scoped access control
|
||||
- [x] Auth provider abstraction — PAM (Linux), config-file, LDAP stub, OAuth2 stub
|
||||
- [x] Full REST API surface (30 endpoints across jobs, hosts, queues, clusters, resources, files, roles, users, permissions)
|
||||
- [x] In-memory generic `DataStore[K,V]` with filter / sort / paginate pipeline
|
||||
- [x] Job ID generator (Snowflake-style, cluster-aware)
|
||||
- [x] File upload with MIME type validation and size limits
|
||||
- [x] Structured logging with `zap` + log rotation (`lumberjack`)
|
||||
- [x] Per-endpoint rate limiting (token bucket, configurable window)
|
||||
- [x] Backend stub + notification handler interfaces ready for integration
|
||||
- [x] **S1** — JWT secret loaded from `XCE_JWT_SECRET` env var; falls back to config file
|
||||
- [x] **S3** — Removed hardcoded mock RBAC maps from `rbac_check.go`
|
||||
- [x] **S4** — Whitelist pattern matching: `pathMatchesPattern()` — exact paths no longer truncated; wildcards fixed for all sub-paths
|
||||
- [x] **F4** — CORS middleware: configurable via `app.yaml`, handles preflight (OPTIONS 204)
|
||||
|
||||
### xcecli
|
||||
- [x] Cobra CLI with command groups; auth-check middleware on all commands
|
||||
- [x] Multi-server config (`~/.xce/config.json`) with token + expiry tracking
|
||||
- [x] All command groups: auth, permissions, roles, users, jobs, hosts, queues, clusters, resources
|
||||
- [x] Paginated table output for all resource types
|
||||
- [x] **T1** — `roles list` / `users list`: removed `unique=true` param; client-side cross-page dedup
|
||||
- [x] **T3b/M3** — `UnmarshalOrderMapRow`: all row types unwrap server `{Fields,Data}` wrapper; all list commands show real data; `num_cpus` tag fixed
|
||||
- [x] **T5** — Column truncation fixed in `roles view` / `users view` / `permissions4user`
|
||||
- [x] **T7** — `xce logout --url` now optional; defaults to active server
|
||||
- [x] **T8** — `jobs submit -f /path` plain paths no longer error
|
||||
- [x] **A1** — Authorization header: removed leading space; all 5 request methods consistent
|
||||
- [x] **A3** — `SetRootCAs`: HTTPS + empty cert → `InsecureSkipVerify`; HTTPS + cert path → load pool
|
||||
- [x] **T9** — `xce hosts select -R "expr"` command; POSTs to `/xce/v1/hosts/select` (returns empty until backend)
|
||||
- [x] **Job time display** — `SUBMIT_TIME` now shows `Apr 08 17:00` instead of `-`; `parseJobTime()` supports RFC3339 and space formats
|
||||
- [x] **Build** — xcecli builds static (no libssl.so.3) via `CGO_ENABLED=0`; pam/openssl removed from nix shell until backend integration
|
||||
|
||||
### Infrastructure
|
||||
- [x] Nix flake dev shell (`nix develop`) — Go 1.25, PAM, OpenSSL, meson, gnumake, swag, gopls, delve
|
||||
- [x] Root `Makefile`: `make test`, `make test-integration`, `make test-all`
|
||||
- [x] Regression suite: `tests/regression/run.sh` — 29 tests, all pass
|
||||
- [x] Unit tests across: `apiserver/store`, `apiserver/utils`, `apiserver/controllers`, `xcecli/cmd/auth`, `xcecli/cmd/jobs`, `xcecli/cmd/roles`, `xcecli/cmd/users`, `xcecli/common/request`, `xcecli/models`
|
||||
|
||||
---
|
||||
|
||||
## Todo
|
||||
|
||||
### 🟡 Missing features
|
||||
|
||||
| ID | Component | Task |
|
||||
|---|---|---|
|
||||
| F1 | server+cli | File delete: uncomment server route + `xce files delete` command |
|
||||
|
||||
### 🔵 Cleanup
|
||||
|
||||
| ID | Component | Task |
|
||||
|---|---|---|
|
||||
| Q1 | apiserver | Regenerate Swagger docs (`swag init`) |
|
||||
|
||||
---
|
||||
|
||||
## Upcoming: Backend Integration Prep
|
||||
|
||||
- [ ] Register `DefaultHandler` with backend notification system so store is populated from real data
|
||||
- [ ] Replace `backend/stub.go` no-ops with real C library / gRPC calls
|
||||
- [ ] Decide persistence strategy — `lib/pq` (PostgreSQL) in `go.mod` but unused; store is pure in-memory
|
||||
- [ ] Make `clusterID` configurable (hardcoded `const clusterID = 1` in `controllers/jobs.go`)
|
||||
Loading…
Add table
Add a link
Reference in a new issue