> ## Documentation Index
> Fetch the complete documentation index at: https://gateway.forceaisecurity.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Run a multi-agent orchestration

> Submit one task; a planner splits it into a dependency graph, a router picks a handler per sub-task (an ephemeral runtime agent or one of your registered A2A agents), ForceAI runs them in order, and a synthesizer returns the answer.

The Agent Orchestrator turns a single request into a coordinated set of small agents. A planner decomposes your task into a dependency graph of sub-tasks, a router decides who handles each one, ForceAI runs them in dependency order (in parallel where the graph allows), and a synthesizer folds the sub-results into one answer. The planner, every sub-agent, and the synthesizer are all governed gateway turns, so each hop is a scoped-key call with PineSmith guardrails, a Request Logs entry, and Audit Logs rows.

<Note>
  The orchestrator is admin-only and lives on the backend control plane. It builds on the [Agent Runtime](/agents/ephemeral-agent), so an ephemeral sub-agent is a real governed instance with its own scoped key. A registered sub-agent (a Snowflake Cortex agent, an AWS AgentCore agent, an M365 Copilot agent, or a custom A2A agent) is invoked over the in-proxy A2A path under a per-sub-task scoped key. Runs are synchronous today: the POST returns once the whole run finishes
</Note>

## How a run is shaped

A run has three moving parts you control through a saved orchestrator.

**Router** decides which agent handles each sub-task:

* **Default** is an agent-aware LLM planner. It sees a capability view of your registered agents and may name one on a sub-task itself; anything it does not name runs as an ephemeral agent
* **Semantic** matches each sub-task's text to your registered agents by lexical overlap (a model-free scorer, so it needs no embedding model; an embedding scorer drops in behind the same seam). Above a threshold it routes to that agent, otherwise the sub-task stays ephemeral
* **Registered agent** delegates the routing decision itself to one of your registered agents. For each un-routed sub-task it asks that agent, over A2A, which candidate should handle it, and uses the pick. This is the pattern where a router agent you already trust owns dispatch

**Sourcing** decides what a sub-task falls back to:

* **auto** always spawns an ephemeral runtime agent
* **registered** must route to a registered agent (a sub-task with no match is an error)
* **hybrid** (the default) routes to a registered agent when one is named or matched, and spawns an ephemeral agent otherwise

**Allowed agents** is an optional whitelist. Leave it empty and the router may reach any enabled registered agent; set it and routing is restricted to just those agents, so an orchestrator can be scoped to a known, approved set.

A **supervisor** watches every run: it records a per-sub-task event trail and, if enough sub-tasks fail, marks the run `escalated`. Escalation is a signal, not a failure; the synthesizer still runs, and the escalation plus the terminal outcome are written to Audit Logs keyed to the run id, which is the seam for wiring a pager, chat, or ticket tool later.

## Create a saved orchestrator

A saved orchestrator persists its config (model, router, sourcing, whitelist, limits) so you can reuse it and reference it by id. Only the agents it spawns at run time are ephemeral.

<Steps>
  <Step title="Create it from the API">
    ```bash theme={null}
    curl -X POST https://YOUR_GATEWAY_DOMAIN/v1/orchestrator/configs \
      -H "Authorization: Bearer $ADMIN_KEY" -H "Content-Type: application/json" \
      -d '{
        "name": "research-orchestrator",
        "orchestrator_model": "smart-router",
        "router_type": "semantic",
        "sourcing_mode": "hybrid",
        "allowed_agents": ["snowflake-cortex", "test-hello"],
        "max_fanout": 4,
        "max_subtasks": 8,
        "budget_usd": 5.0
      }'
    ```

    `router_type` is `default`, `semantic`, or `registered_agent`; when it is `registered_agent` you must also set `router_agent_name` to the agent that owns routing. `sourcing_mode` is `auto`, `registered`, or `hybrid`. `allowed_agents` is optional; leave it out to allow any enabled agent. A duplicate `name` returns `409`
  </Step>

  <Step title="See which agents a router can reach">
    The capability view is the same one the routers use: every registered agent with its description and whether it is enabled

    ```bash theme={null}
    curl https://YOUR_GATEWAY_DOMAIN/v1/orchestrator/capabilities \
      -H "Authorization: Bearer $ADMIN_KEY"
    ```
  </Step>

  <Step title="List or fetch saved orchestrators">
    ```bash theme={null}
    curl https://YOUR_GATEWAY_DOMAIN/v1/orchestrator/configs -H "Authorization: Bearer $ADMIN_KEY"
    curl https://YOUR_GATEWAY_DOMAIN/v1/orchestrator/configs/$ORCHESTRATOR_ID -H "Authorization: Bearer $ADMIN_KEY"
    ```
  </Step>
</Steps>

You can also do all of this from **Agents -> Agent Orchestrator**: the Saved orchestrators table lists what exists, and **New orchestrator** opens a form for the name, model, router type, sourcing mode, a router-agent picker (shown when the router type is `registered_agent`), and an allowed-agents multi-select (shown for `hybrid` and `registered` sourcing, populated from the capability view).

## Run a task

<Steps>
  <Step title="Run against a saved orchestrator">
    Reference it by `orchestrator_id` and the run inherits its model, router, sourcing, whitelist, and limits

    ```bash theme={null}
    curl -X POST https://YOUR_GATEWAY_DOMAIN/v1/orchestrator/runs \
      -H "Authorization: Bearer $ADMIN_KEY" -H "Content-Type: application/json" \
      -d '{ "task": "Compare two approaches and recommend one", "orchestrator_id": "'$ORCHESTRATOR_ID'" }'
    ```
  </Step>

  <Step title="Or run ad-hoc">
    Omit `orchestrator_id` to use the default orchestrator. `model` sets the model for the planner, the synthesizer, and every sub-agent that does not name its own; it defaults to `smart-router`

    ```bash theme={null}
    curl -X POST https://YOUR_GATEWAY_DOMAIN/v1/orchestrator/runs \
      -H "Authorization: Bearer $ADMIN_KEY" -H "Content-Type: application/json" \
      -d '{ "task": "Compare two approaches and recommend one", "model": "smart-router" }'
    ```
  </Step>

  <Step title="Or hand in a deterministic plan">
    Skip planning entirely by passing an explicit `plan`. Each node can name the `agent_name` that should handle it, so you script the exact flow. The plan is validated the same way a planned one is: unknown or disabled agent names, cycles, dangling dependencies, and duplicate ids are rejected with `400`

    ```bash theme={null}
    curl -X POST https://YOUR_GATEWAY_DOMAIN/v1/orchestrator/runs \
      -H "Authorization: Bearer $ADMIN_KEY" -H "Content-Type: application/json" \
      -d '{
        "task": "custom flow",
        "plan": {
          "subtasks": [
            { "sub_task_id": "a", "task": "pull the data", "depends_on": [], "agent_name": "snowflake-cortex" },
            { "sub_task_id": "b", "task": "summarize it", "depends_on": ["a"] }
          ]
        }
      }'
    ```
  </Step>

  <Step title="Read the run">
    The response is the completed run. `escalated` is the supervisor's flag, `events` is its trail, and each sub-task and result tells you who handled it: `agent_name` is the registered agent (or null for an ephemeral one), `instance_id` is the ephemeral instance, and `capability` is the ephemeral agent profile

    ```json theme={null}
    {
      "run_id": "…",
      "state": "completed",
      "escalated": false,
      "answer": "…",
      "subtasks": [
        { "sub_task_id": "a", "task": "analyze approach A", "capability": "researcher", "depends_on": [], "agent_name": "snowflake-cortex" },
        { "sub_task_id": "b", "task": "analyze approach B", "capability": "chat", "depends_on": [], "agent_name": null },
        { "sub_task_id": "c", "task": "compare and recommend", "capability": "assistant", "depends_on": ["a", "b"], "agent_name": null }
      ],
      "results": [
        { "sub_task_id": "a", "ok": true, "agent_name": "snowflake-cortex", "output": "…" },
        { "sub_task_id": "b", "ok": true, "instance_id": "…", "output": "…" }
      ],
      "events": [
        { "kind": "planned", "detail": "3 subtasks" },
        { "kind": "subtask_ok", "sub_task_id": "a" },
        { "kind": "completed", "detail": "" }
      ]
    }
    ```
  </Step>

  <Step title="Fetch or list runs later">
    Runs are held in memory for the life of the process

    ```bash theme={null}
    curl https://YOUR_GATEWAY_DOMAIN/v1/orchestrator/runs -H "Authorization: Bearer $ADMIN_KEY"
    curl https://YOUR_GATEWAY_DOMAIN/v1/orchestrator/runs/$RUN_ID -H "Authorization: Bearer $ADMIN_KEY"
    ```
  </Step>
</Steps>

From the dashboard, **Agents -> Agent Orchestrator** has a **Run with** selector (run ad-hoc or against a saved orchestrator by id). When a run finishes, the drawer shows the plan (the Depends on column is the dependency graph), a Handler column for who ran each sub-task (a registered agent versus ephemeral), each sub-task's output, the final answer, an escalated badge when the supervisor tripped, and the supervisor's event list.

## Trace a run in the logs

Every governed turn of a run is tagged so you can follow it end to end.

In **Logs -> Request Logs**, an orchestrator turn carries an **Orchestrator** badge in the Type column. Open the turn and the detail drawer shows an **Agent Orchestrator run** banner with the run id, the sub-task it belonged to (`planner`, a sub-task id, or `synthesizer`), and a **View run** button that jumps to **Agents -> Agent Orchestrator** with that run's detail drawer already open. So you can go from one line in the logs to the whole run it came from, and back.

Under the hood each turn is stamped, on its scoped key, with `spend_logs_metadata` carrying `forceai_type=orchestrator`, `forceai_orchestrator_run_id`, and `forceai_sub_task_id`; that is what the badge and banner read. The supervisor's escalation and the run's terminal outcome land in **Logs -> Audit Logs** keyed to the run id.

## Which agent runs each sub-task

For an ephemeral sub-task, the planner assigns an **agent type** (the `capability` field: `chat`, `assistant`, `researcher`) from the runtime catalog, which sets the sub-agent's default profile and tool bindings, and you choose the **model** (set once on the run, applied to every sub-agent that does not name its own; `smart-router` picks a tier per sub-task). For a registered sub-task, `agent_name` names the registered A2A agent that handles it, and that agent runs on its own configured backend.

Tune a run with these backend settings: `FORCEAI_ORCHESTRATOR_PLANNER_MODEL` (the planner's model), `FORCEAI_ORCHESTRATOR_MAX_FANOUT` (how many sub-agents run at once, default 4), `FORCEAI_ORCHESTRATOR_MAX_SUBTASKS` (the cap the planner must stay under, default 8), `FORCEAI_ORCHESTRATOR_SOURCING` (the default sourcing mode, default `hybrid`), and `FORCEAI_ORCHESTRATOR_SEMANTIC_THRESHOLD` (the semantic router's match threshold, default `0.08`). A saved orchestrator's own fields override the run-shape defaults for runs that reference it.

<Note>
  The semantic router ships with a lexical scorer, so terse agent descriptions can score just under the threshold and leave a sub-task ephemeral even when a good agent exists. Give your registered agents descriptive summaries, or lower `FORCEAI_ORCHESTRATOR_SEMANTIC_THRESHOLD`, to route more sub-tasks. Real embedding scoring plugs in behind the same seam
</Note>

## A note on guardrails

Because every hop is a governed gateway turn, PineSmith runs on each one. Its default policy blocks at `risk=LOW`, which is strict: a later sub-task or the synthesizer, whose prompt embeds the outputs of earlier steps, can trip the guardrail and come back blocked. That is the guardrail doing its job, not an orchestrator error; the engine records the block, fails that node, and skips its dependents. Loosen the PineSmith risk threshold if you want more multi-step runs to complete.