# API Server Backend Gap Analysis

Source: `src/controller/testsuite/test.c` (C test tool) vs `src/frontend/apiserver/` (Go API server)

## Overview

The C test tool (`test.c`) directly calls `xcecli_*` C library functions to interact
with the controller. The Go API server currently operates in a **push-based, read-only-from-store**
model: the controller pushes data (jobs, workers, queues, resources) via Cap'n Proto, and
the API server reads from its in-memory stores. Write/control operations are forwarded
via the CGO backend layer, but only a subset of the C API is wired up.

---

## Currently Implemented (Backend Forwards to C Controller)

| C Test Command | C API Call             | Go Backend Function     | REST Endpoint                      |
|----------------|------------------------|-------------------------|------------------------------------|
| `jobsub`       | `xcecli_jobsubmit()`   | `backend.SubmitJob()`   | `POST /xce/v1/jobs/create`        |
| `jobkill/jobckpt` | `xcecli_jobctrl()`  | `backend.ControlJob()` | `POST /xce/v1/jobs/:jobids/control` |
| `qctrl`        | `xcecli_queuectrl()`   | `backend.ControlQueue()`| `POST /xce/v1/queues/control`     |
| *(partial)*    | `xcecli_hostctrl()` (N/A) | `backend.ControlHost()` **stub only** | `POST /xce/v1/hosts/control` |

Push notifications (read path, already working via `handlePush`):

- Job updates via `OnJobUpdated` -> `store.Jobs`
- Worker/host updates via `OnHostUpdated` -> `store.Hosts`
- Queue updates via `OnQueueUpdated` -> `store.Queues`
- Resource definitions via `OnResourceUpdated` -> `store.Resources`
- Resource locations via `OnResourceLocationUpdated` -> `store.ResourceLocations`

---

## NOT Implemented -- Backend Request Gaps

### 1. Resource Value Query (`cmd_resval` -> `xcecli_resval()`)

- **What it does**: Queries current numeric/string values of a named resource on a
  specific host (or cluster-wide). Returns typed values (float, string, bool).
- **REST impact**: No `GET /xce/v1/resources/values` endpoint exists. The resource
  controller only serves definitions and distributions, not live values.
- **Priority**: **High** -- Essential for monitoring dashboards to show real-time
  `mem`, `r1m`, `slots` etc.

[feedback] api server will not pull resources from controller, it just waits controller to push
           updated resource information, so api server just handle resource information update will be OK. 

### 2. Resource Value Update (`cmd_resupdate` -> `xcecli_update_resvals()`)

- **What it does**: Worker pushes updated resource values to the controller.
- **REST impact**: No equivalent. Currently workers call the C API directly. If the
  API server should support an admin endpoint to update resource values (e.g., for
  testing or override), this is missing.
- **Priority**: **Low-Medium** -- Workers typically call this directly; rarely needed
  via REST. Could be useful for admin overrides.

[feedback] This item will not be applied to api server, resource update function need to be
           used to update resources information of worker node(such as cpu/memory/..., and api
           server will not talke workload, so it do not need this)

### 3. Job Dispatch (`cmd_jobrun` -> `xcecli_jobdispatch()`)

- **What it does**: Dispatches a job to specific hosts with allocation info
  (host:ntasks pairs). This is how the scheduler tells the controller to start
  running a job on specific workers.
- **REST impact**: Not exposed. The `POST /xce/v1/jobs/create` handles submission,
  but **dispatch** (assignment to hosts) is a separate operation currently done by
  the scheduler directly.
- **Priority**: **Low** -- Dispatch is an internal scheduler->controller operation.
  Not typically user-facing.

[feedback] treat this as low priority, IBM LSF has a brun command which issued by user to 
           force a job to run in given host, we may need this. also cmd_jobrun() has
           been used to test execution flow/protocol between controller and work daemon

### 4. Job Info / Single Job Detail (`cmd_jobinfo` -> `xcecli_jobinfo()`)

- **What it does**: Queries detailed info for a single job by ID, including pend
  reasons.
- **REST impact**: The `GET /xce/v1/jobs/:jobids` endpoint reads from the in-memory
  store, which is populated by push notifications. This works for already-known jobs
  but **cannot query a job that hasn't been pushed yet**. The C `xcecli_jobinfo()` is
  a synchronous request-response to the controller -- fundamentally different from
  the store-read pattern.
- **Priority**: **Medium** -- Store-based reads are sufficient for most cases. The
  sync query path would only matter if pushes are delayed or the store is stale.

[feedback] no need this, as job information will be pushed to api server

### 5. Cluster Configuration (`cmd_getconf` -> `xcecli_clusterconf()`)

- **What it does**: Synchronously fetches the full cluster config (resources, queues,
  workers, host groups, user groups, global configs).
- **REST impact**: Currently the API server relies entirely on push notifications to
  populate stores. There's no way to do a one-shot **full sync** with the controller.
  On startup, it joins via `xcecli_join(HostRole_apiserver)` but doesn't explicitly
  request the current state.
- **Priority**: **High** -- Without an initial sync, if the API server restarts
  mid-session, it loses all state until the next push. The store is stale until a
  new push arrives.

[feedback] api server need handle pushed cluster configuration

### 6. Host Control (`backend.ControlHost()` -> `xcecli_hostctrl()`)

- **What it does**: The C library doesn't expose `xcecli_hostctrl()` yet (the Go
  stub returns an error).
- **REST impact**: `POST /xce/v1/hosts/control` exists but will always fail:
  `"host control not yet supported via C library"`.
- **Priority**: **High** -- Users cannot open/close hosts via the API.

[feedback] we need this

### 7. Set Pending Reason (`cmd_setpend` -> `xcecli_set_pend_reason()`)

- **What it does**: Scheduler sets pend reasons on a job (host-based or
  resource-based).
- **REST impact**: No equivalent endpoint. Job pend reasons are currently populated
  only via push data.
- **Priority**: **Low** -- This is a scheduler->controller internal operation.

[feedback] set pending reason is for scheduler to set why job will be pend, this is 
           used for test scheduler to controller job pending api

### 8. Run Tasks (`cmd_runtasks` -> `xcecli_runtask()`)

- **What it does**: Launches individual tasks on specific hosts (not a full job, but
  sub-task execution).
- **REST impact**: No endpoint exists.
- **Priority**: **Low-Medium** -- Used for advanced task launch scenarios. Could be
  needed for MPI job management.

[feedback] this function is used by worker to run non rank 0 tasks(this is for MPI jobs) and api server
           need not handle this

### 9. Shutdown (`cmd_shutdown` -> `xcecli_shutdown()`)

- **What it does**: Sends shutdown command to the cluster controller.
- **REST impact**: No endpoint. Currently cluster management operations (shutdown,
  restart) are not available via REST.
- **Priority**: **Medium** -- Operations teams need remote cluster control capability.

[feedback] we may need to admin tool, which talk to controller directly  for this, as normal
           user do not need this at all

### 10. Resource Requirement Select (`FilterHostByResReqSelect`)

- **What it does**: The hosts `POST /select` endpoint exists but returns **empty
  results** -- it's a stub: `processHostData([]utils.OrderMap{}, "")`.
- **REST impact**: `POST /xce/v1/hosts/select` is defined but non-functional.
- **Priority**: **Medium** -- This is the XCE equivalent of `lsload -R` / host
  matching by resource requirements. Important for job submission workflows to
  find eligible hosts.

[feedback] we may need this

---

## Enhancement Proposal (Prioritized)

| Priority | Enhancement                                 | Effort     | Description |
|----------|---------------------------------------------|------------|-------------|
| **P0**   | Initial state sync via `xcecli_clusterconf()` | Medium  | On startup (after `RegisterToBackend`), call `xcecli_clusterconf()` to do a full state sync into stores. Prevents stale stores after API server restart. |
| **P0**   | Implement `ControlHost()` in C library + Go wrapper | Medium | Expose `xcecli_hostctrl()` in `client.h`, then replace the Go stub in `clib.go` with a real CGO call. |
| **P1**   | Resource value query endpoint               | Medium     | Add `GET /xce/v1/resources/values?host=X&name=Y` -> `backend.QueryResourceValues()` -> `xcecli_resval()`. This is a synchronous request-response pattern (different from push). |
| **P1**   | Implement `FilterHostByResReqSelect`        | Medium-High| Wire `POST /xce/v1/hosts/select` to evaluate resource requirement expressions against current host state from the store. |
| **P2**   | Cluster shutdown endpoint                   | Low        | Add `POST /xce/v1/clusters/control` -> `backend.ShutdownCluster()` -> `xcecli_shutdown()`. Should be admin-only with RBAC. |
| **P2**   | Resource value update endpoint              | Low        | Add `POST /xce/v1/resources/values/update` -> `backend.UpdateResourceValues()` -> `xcecli_update_resvals()`. Admin/testing use. |
| **P3**   | Task run endpoint                           | Low-Medium | Add `POST /xce/v1/tasks/run` -> `backend.RunTask()` -> `xcecli_runtask()`. For advanced MPI task management. |
| **P3**   | Set pending reason endpoint                 | Low        | Add `POST /xce/v1/jobs/:jobid/pending-reason` -> `backend.SetPendReason()` -> `xcecli_set_pend_reason()`. Internal/scheduler-facing only. |

---

## Architecture Note: Synchronous vs Push-Based

The biggest design decision lurking here: several of these C APIs (`xcecli_resval()`,
`xcecli_clusterconf()`, `xcecli_jobinfo()`) are **synchronous request-response** calls
-- the client sends a request and blocks until the controller replies. This is
fundamentally different from the current **push-based** pattern where the controller
asynchronously pushes updates.

To implement these, we'd need to either:

1. **Direct CGO call** (simpler but blocks the Go goroutine -- acceptable if calls
   are fast and we set timeouts)
2. **Request-response via push channel** (more complex -- send request, register a
   wait, receive push reply)

Recommendation: option 1 for initial implementation (it's what `SubmitJob` and
`ControlJob` already do), with timeouts and goroutine isolation to prevent a slow
controller from blocking the API server.

---

## Additional C API Calls Not Used Anywhere (Reference)

These exist in `client.h` but are not called by `test.c` either:

| C API Call                | Purpose |
|--------------------------|---------|
| `xcecli_hello()`         | Health check / greeting |
| `xcecli_nop()`           | No-op heartbeat |
| `xcecli_heartbeat()`     | Periodic heartbeat |
| `xcecli_notify_jobstart()`| Worker notifies job started (with PID info) |
| `xcecli_notify_jobfail()` | Worker notifies job failed |
| `xcecli_notify_jobstop()` | Worker notifies job stopped |
| `xcecli_notify_jobresume()`| Worker notifies job resumed |
| `xcecli_notify_jobpostdone()`| Worker notifies job post-exec done |
| `xcecli_update_jusage()` | Worker updates job resource usage |
| `xcecli_notify()`        | Generic notification |
| `xcecli_notify_taskstart()`| Worker notifies task started |
| `xcecli_notify_taskfail()`| Worker notifies task failed |
| `xcecli_notify_taskfinish()`| Worker notifies task finished |
| `xcecli_clear_reason()`  | Clear pend reasons for jobs |
| `xcecli_block_tasks()`   | Block task processing |
| `xcecli_unblock_tasks()` | Unblock task processing |
| `xcecli_get_my_endpoint()`| Get local endpoint info |
| `xcecli_get_myinfo()`    | Get local peer info |

[feedback] I think we just need heartbeat to tell controller that I'm alive, and other
           functions for worker and scheduler

These are primarily worker->controller notifications. They would be needed if
the API server were to act as a proxy for worker daemons, which is not the
current design.