forked from lei.chen/myxce
Add HostConfig + Cluster fields to models and update docs
This commit is contained in:
parent
960ef62342
commit
00c4472d6c
9 changed files with 1805 additions and 9 deletions
225
src/docs/apiserver-backend-gaps.md
Normal file
225
src/docs/apiserver-backend-gaps.md
Normal file
|
|
@ -0,0 +1,225 @@
|
|||
# API Server Backend Gap Analysis
|
||||
|
||||
Source: `src/controller/testsuite/test.c` (C test tool) vs `src/frontend/apiserver/` (Go API server)
|
||||
|
||||
## Overview
|
||||
|
||||
The C test tool (`test.c`) directly calls `xcecli_*` C library functions to interact
|
||||
with the controller. The Go API server currently operates in a **push-based, read-only-from-store**
|
||||
model: the controller pushes data (jobs, workers, queues, resources) via Cap'n Proto, and
|
||||
the API server reads from its in-memory stores. Write/control operations are forwarded
|
||||
via the CGO backend layer, but only a subset of the C API is wired up.
|
||||
|
||||
---
|
||||
|
||||
## Currently Implemented (Backend Forwards to C Controller)
|
||||
|
||||
| C Test Command | C API Call | Go Backend Function | REST Endpoint |
|
||||
|----------------|------------------------|-------------------------|------------------------------------|
|
||||
| `jobsub` | `xcecli_jobsubmit()` | `backend.SubmitJob()` | `POST /xce/v1/jobs/create` |
|
||||
| `jobkill/jobckpt` | `xcecli_jobctrl()` | `backend.ControlJob()` | `POST /xce/v1/jobs/:jobids/control` |
|
||||
| `qctrl` | `xcecli_queuectrl()` | `backend.ControlQueue()`| `POST /xce/v1/queues/control` |
|
||||
| *(partial)* | `xcecli_hostctrl()` (N/A) | `backend.ControlHost()` **stub only** | `POST /xce/v1/hosts/control` |
|
||||
|
||||
Push notifications (read path, already working via `handlePush`):
|
||||
|
||||
- Job updates via `OnJobUpdated` -> `store.Jobs`
|
||||
- Worker/host updates via `OnHostUpdated` -> `store.Hosts`
|
||||
- Queue updates via `OnQueueUpdated` -> `store.Queues`
|
||||
- Resource definitions via `OnResourceUpdated` -> `store.Resources`
|
||||
- Resource locations via `OnResourceLocationUpdated` -> `store.ResourceLocations`
|
||||
|
||||
---
|
||||
|
||||
## NOT Implemented -- Backend Request Gaps
|
||||
|
||||
### 1. Resource Value Query (`cmd_resval` -> `xcecli_resval()`)
|
||||
|
||||
- **What it does**: Queries current numeric/string values of a named resource on a
|
||||
specific host (or cluster-wide). Returns typed values (float, string, bool).
|
||||
- **REST impact**: No `GET /xce/v1/resources/values` endpoint exists. The resource
|
||||
controller only serves definitions and distributions, not live values.
|
||||
- **Priority**: **High** -- Essential for monitoring dashboards to show real-time
|
||||
`mem`, `r1m`, `slots` etc.
|
||||
|
||||
[feedback] api server will not pull resources from controller, it just waits controller to push
|
||||
updated resource information, so api server just handle resource information update will be OK.
|
||||
|
||||
### 2. Resource Value Update (`cmd_resupdate` -> `xcecli_update_resvals()`)
|
||||
|
||||
- **What it does**: Worker pushes updated resource values to the controller.
|
||||
- **REST impact**: No equivalent. Currently workers call the C API directly. If the
|
||||
API server should support an admin endpoint to update resource values (e.g., for
|
||||
testing or override), this is missing.
|
||||
- **Priority**: **Low-Medium** -- Workers typically call this directly; rarely needed
|
||||
via REST. Could be useful for admin overrides.
|
||||
|
||||
[feedback] This item will not be applied to api server, resource update function need to be
|
||||
used to update resources information of worker node(such as cpu/memory/..., and api
|
||||
server will not talke workload, so it do not need this)
|
||||
|
||||
### 3. Job Dispatch (`cmd_jobrun` -> `xcecli_jobdispatch()`)
|
||||
|
||||
- **What it does**: Dispatches a job to specific hosts with allocation info
|
||||
(host:ntasks pairs). This is how the scheduler tells the controller to start
|
||||
running a job on specific workers.
|
||||
- **REST impact**: Not exposed. The `POST /xce/v1/jobs/create` handles submission,
|
||||
but **dispatch** (assignment to hosts) is a separate operation currently done by
|
||||
the scheduler directly.
|
||||
- **Priority**: **Low** -- Dispatch is an internal scheduler->controller operation.
|
||||
Not typically user-facing.
|
||||
|
||||
[feedback] treat this as low priority, IBM LSF has a brun command which issued by user to
|
||||
force a job to run in given host, we may need this. also cmd_jobrun() has
|
||||
been used to test execution flow/protocol between controller and work daemon
|
||||
|
||||
### 4. Job Info / Single Job Detail (`cmd_jobinfo` -> `xcecli_jobinfo()`)
|
||||
|
||||
- **What it does**: Queries detailed info for a single job by ID, including pend
|
||||
reasons.
|
||||
- **REST impact**: The `GET /xce/v1/jobs/:jobids` endpoint reads from the in-memory
|
||||
store, which is populated by push notifications. This works for already-known jobs
|
||||
but **cannot query a job that hasn't been pushed yet**. The C `xcecli_jobinfo()` is
|
||||
a synchronous request-response to the controller -- fundamentally different from
|
||||
the store-read pattern.
|
||||
- **Priority**: **Medium** -- Store-based reads are sufficient for most cases. The
|
||||
sync query path would only matter if pushes are delayed or the store is stale.
|
||||
|
||||
[feedback] no need this, as job information will be pushed to api server
|
||||
|
||||
### 5. Cluster Configuration (`cmd_getconf` -> `xcecli_clusterconf()`)
|
||||
|
||||
- **What it does**: Synchronously fetches the full cluster config (resources, queues,
|
||||
workers, host groups, user groups, global configs).
|
||||
- **REST impact**: Currently the API server relies entirely on push notifications to
|
||||
populate stores. There's no way to do a one-shot **full sync** with the controller.
|
||||
On startup, it joins via `xcecli_join(HostRole_apiserver)` but doesn't explicitly
|
||||
request the current state.
|
||||
- **Priority**: **High** -- Without an initial sync, if the API server restarts
|
||||
mid-session, it loses all state until the next push. The store is stale until a
|
||||
new push arrives.
|
||||
|
||||
[feedback] api server need handle pushed cluster configuration
|
||||
|
||||
### 6. Host Control (`backend.ControlHost()` -> `xcecli_hostctrl()`)
|
||||
|
||||
- **What it does**: The C library doesn't expose `xcecli_hostctrl()` yet (the Go
|
||||
stub returns an error).
|
||||
- **REST impact**: `POST /xce/v1/hosts/control` exists but will always fail:
|
||||
`"host control not yet supported via C library"`.
|
||||
- **Priority**: **High** -- Users cannot open/close hosts via the API.
|
||||
|
||||
[feedback] we need this
|
||||
|
||||
### 7. Set Pending Reason (`cmd_setpend` -> `xcecli_set_pend_reason()`)
|
||||
|
||||
- **What it does**: Scheduler sets pend reasons on a job (host-based or
|
||||
resource-based).
|
||||
- **REST impact**: No equivalent endpoint. Job pend reasons are currently populated
|
||||
only via push data.
|
||||
- **Priority**: **Low** -- This is a scheduler->controller internal operation.
|
||||
|
||||
[feedback] set pending reason is for scheduler to set why job will be pend, this is
|
||||
used for test scheduler to controller job pending api
|
||||
|
||||
### 8. Run Tasks (`cmd_runtasks` -> `xcecli_runtask()`)
|
||||
|
||||
- **What it does**: Launches individual tasks on specific hosts (not a full job, but
|
||||
sub-task execution).
|
||||
- **REST impact**: No endpoint exists.
|
||||
- **Priority**: **Low-Medium** -- Used for advanced task launch scenarios. Could be
|
||||
needed for MPI job management.
|
||||
|
||||
[feedback] this function is used by worker to run non rank 0 tasks(this is for MPI jobs) and api server
|
||||
need not handle this
|
||||
|
||||
### 9. Shutdown (`cmd_shutdown` -> `xcecli_shutdown()`)
|
||||
|
||||
- **What it does**: Sends shutdown command to the cluster controller.
|
||||
- **REST impact**: No endpoint. Currently cluster management operations (shutdown,
|
||||
restart) are not available via REST.
|
||||
- **Priority**: **Medium** -- Operations teams need remote cluster control capability.
|
||||
|
||||
[feedback] we may need to admin tool, which talk to controller directly for this, as normal
|
||||
user do not need this at all
|
||||
|
||||
### 10. Resource Requirement Select (`FilterHostByResReqSelect`)
|
||||
|
||||
- **What it does**: The hosts `POST /select` endpoint exists but returns **empty
|
||||
results** -- it's a stub: `processHostData([]utils.OrderMap{}, "")`.
|
||||
- **REST impact**: `POST /xce/v1/hosts/select` is defined but non-functional.
|
||||
- **Priority**: **Medium** -- This is the XCE equivalent of `lsload -R` / host
|
||||
matching by resource requirements. Important for job submission workflows to
|
||||
find eligible hosts.
|
||||
|
||||
[feedback] we may need this
|
||||
|
||||
---
|
||||
|
||||
## Enhancement Proposal (Prioritized)
|
||||
|
||||
| Priority | Enhancement | Effort | Description |
|
||||
|----------|---------------------------------------------|------------|-------------|
|
||||
| **P0** | Initial state sync via `xcecli_clusterconf()` | Medium | On startup (after `RegisterToBackend`), call `xcecli_clusterconf()` to do a full state sync into stores. Prevents stale stores after API server restart. |
|
||||
| **P0** | Implement `ControlHost()` in C library + Go wrapper | Medium | Expose `xcecli_hostctrl()` in `client.h`, then replace the Go stub in `clib.go` with a real CGO call. |
|
||||
| **P1** | Resource value query endpoint | Medium | Add `GET /xce/v1/resources/values?host=X&name=Y` -> `backend.QueryResourceValues()` -> `xcecli_resval()`. This is a synchronous request-response pattern (different from push). |
|
||||
| **P1** | Implement `FilterHostByResReqSelect` | Medium-High| Wire `POST /xce/v1/hosts/select` to evaluate resource requirement expressions against current host state from the store. |
|
||||
| **P2** | Cluster shutdown endpoint | Low | Add `POST /xce/v1/clusters/control` -> `backend.ShutdownCluster()` -> `xcecli_shutdown()`. Should be admin-only with RBAC. |
|
||||
| **P2** | Resource value update endpoint | Low | Add `POST /xce/v1/resources/values/update` -> `backend.UpdateResourceValues()` -> `xcecli_update_resvals()`. Admin/testing use. |
|
||||
| **P3** | Task run endpoint | Low-Medium | Add `POST /xce/v1/tasks/run` -> `backend.RunTask()` -> `xcecli_runtask()`. For advanced MPI task management. |
|
||||
| **P3** | Set pending reason endpoint | Low | Add `POST /xce/v1/jobs/:jobid/pending-reason` -> `backend.SetPendReason()` -> `xcecli_set_pend_reason()`. Internal/scheduler-facing only. |
|
||||
|
||||
---
|
||||
|
||||
## Architecture Note: Synchronous vs Push-Based
|
||||
|
||||
The biggest design decision lurking here: several of these C APIs (`xcecli_resval()`,
|
||||
`xcecli_clusterconf()`, `xcecli_jobinfo()`) are **synchronous request-response** calls
|
||||
-- the client sends a request and blocks until the controller replies. This is
|
||||
fundamentally different from the current **push-based** pattern where the controller
|
||||
asynchronously pushes updates.
|
||||
|
||||
To implement these, we'd need to either:
|
||||
|
||||
1. **Direct CGO call** (simpler but blocks the Go goroutine -- acceptable if calls
|
||||
are fast and we set timeouts)
|
||||
2. **Request-response via push channel** (more complex -- send request, register a
|
||||
wait, receive push reply)
|
||||
|
||||
Recommendation: option 1 for initial implementation (it's what `SubmitJob` and
|
||||
`ControlJob` already do), with timeouts and goroutine isolation to prevent a slow
|
||||
controller from blocking the API server.
|
||||
|
||||
---
|
||||
|
||||
## Additional C API Calls Not Used Anywhere (Reference)
|
||||
|
||||
These exist in `client.h` but are not called by `test.c` either:
|
||||
|
||||
| C API Call | Purpose |
|
||||
|--------------------------|---------|
|
||||
| `xcecli_hello()` | Health check / greeting |
|
||||
| `xcecli_nop()` | No-op heartbeat |
|
||||
| `xcecli_heartbeat()` | Periodic heartbeat |
|
||||
| `xcecli_notify_jobstart()`| Worker notifies job started (with PID info) |
|
||||
| `xcecli_notify_jobfail()` | Worker notifies job failed |
|
||||
| `xcecli_notify_jobstop()` | Worker notifies job stopped |
|
||||
| `xcecli_notify_jobresume()`| Worker notifies job resumed |
|
||||
| `xcecli_notify_jobpostdone()`| Worker notifies job post-exec done |
|
||||
| `xcecli_update_jusage()` | Worker updates job resource usage |
|
||||
| `xcecli_notify()` | Generic notification |
|
||||
| `xcecli_notify_taskstart()`| Worker notifies task started |
|
||||
| `xcecli_notify_taskfail()`| Worker notifies task failed |
|
||||
| `xcecli_notify_taskfinish()`| Worker notifies task finished |
|
||||
| `xcecli_clear_reason()` | Clear pend reasons for jobs |
|
||||
| `xcecli_block_tasks()` | Block task processing |
|
||||
| `xcecli_unblock_tasks()` | Unblock task processing |
|
||||
| `xcecli_get_my_endpoint()`| Get local endpoint info |
|
||||
| `xcecli_get_myinfo()` | Get local peer info |
|
||||
|
||||
[feedback] I think we just need heartbeat to tell controller that I'm alive, and other
|
||||
functions for worker and scheduler
|
||||
|
||||
These are primarily worker->controller notifications. They would be needed if
|
||||
the API server were to act as a proxy for worker daemons, which is not the
|
||||
current design.
|
||||
Loading…
Add table
Add a link
Reference in a new issue