> ## Documentation Index
> Fetch the complete documentation index at: https://gateway.forceaisecurity.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Model Evaluations
> Score a model against a suite of cases before anyone depends on it. Author a suite, run a bake-off, read the report, and turn on enforcement without causing an outage.
Public benchmark scores say nothing about whether a model handles your contract language, under your guardrails, at your cost. The evaluator answers a narrower and more useful question: is this model fit for this workload, in this organization
A run scores each candidate per dimension and never averages them into one number. Someone who reads only the headline should still be unable to promote a model that failed on safety
## Two kinds of run, and only one counts
This distinction is what makes the whole feature trustworthy, so it is worth getting straight before anything else
An **org score** comes only from a run against the **canonical suite**, which admins own. It is what appears beside a model when someone is choosing one, and it is the only thing the publication gate reads
An **exploratory run** is a validator's ad-hoc bake-off on their own suite. It answers a question, it shows in run history, and it never becomes the org score
Without that split the gate is theatre. Anyone who wanted a model unlocked would write a three-case suite it passes and self-certify
## Case kinds
A suite is a list of cases. Each carries a `kind` that decides how it is graded, and the choice matters because two of the three cost nothing to grade
| Kind | Graded by | Cost | Use it for |
| ----------- | ------------------------------------------------------------ | ----------------------- | --------------------------------------------------- |
| `json` | Output must validate against the case's JSON Schema | Free | Structured extraction, any response with a contract |
| `tool_call` | Expected tool chosen, expected arguments matched as a subset | Free | Tool and function selection |
| `freeform` | An LLM judge, against your rubric or a reference answer | A judge call per sample | Prose quality, explanations, summaries |
Structured cases should be the backbone of any suite. They are deterministic and free, which is what lets you run the same suite twice without the bill doubling
## Walkthrough
In the dashboard, **Model Evaluations -> Suites -> New suite**. Over the API:
```bash theme={null}
curl -s -X POST "$FORCEAI_URL/v1/evaluations/suites" \
-H "Authorization: Bearer $FORCEAI_KEY" -H 'Content-Type: application/json' -d '{
"name": "canonical-suite",
"cases": [
{"kind":"json","case_id":"person",
"prompt":"Return ONLY a JSON object with keys name (string) and age (integer). No prose, no code fences.",
"schema":{"type":"object","required":["name","age"],
"properties":{"name":{"type":"string"},"age":{"type":"integer"}}}}
]}'
```
```bash theme={null}
curl -s -X POST "$FORCEAI_URL/v1/evaluations/suites/$SUITE_ID/canonical" \
-H "Authorization: Bearer $FORCEAI_KEY"
```
Exactly one suite per organization is canonical. Designating a new one clears the previous holder, so there is never ambiguity about which numbers the badge and the gate read
Until you designate one, every model reports `not evaluated`. That is the correct answer, not an error: an organization that has not chosen a canonical suite has not measured anything
```bash theme={null}
curl -s -X POST "$FORCEAI_URL/v1/evaluations" \
-H "Authorization: Bearer $FORCEAI_KEY" -H 'Content-Type: application/json' -d '{
"suite_id": "'"$SUITE_ID"'",
"models": ["gemini-2.5-flash", "claude-haiku-4-5"],
"samples_per_case": 3,
"p95_latency_budget_ms": 15000
}'
```
```json theme={null}
{"run_id":"c9802e98-14ee-4448-8513-74d50bad6b6d","state":"queued"}
```
A run is asynchronous. Several models over a suite at three samples a case takes minutes, so the API returns a run id you poll
```bash theme={null}
curl -s "$FORCEAI_URL/v1/evaluations/$RUN_ID" -H "Authorization: Bearer $FORCEAI_KEY"
```
```
state: complete | samples/case: 3
performance claude-haiku-4-5 pass p50 1458ms, p95 1540ms against a 15000ms budget
performance gemini-2.5-flash pass p50 2368ms, p95 2393ms against a 15000ms budget
response_quality claude-haiku-4-5 pass [1.00-1.00] n=6
response_quality gemini-2.5-flash pass [1.00-1.00] n=6
notes: ['candidates could not be separated: their intervals overlap']
```
Both models passed and the run still declined to name a winner. That note is the design working: ranking two identical intervals by a difference smaller than the noise would be inventing a result
## Reading a report properly
### Every score is an interval
At any temperature above zero the same prompt gives different answers, so a single sample is an anecdote. Each case runs `samples_per_case` times (three by default) and the score carries a mean and a spread. Comparisons are made between intervals, and where they overlap the run says so instead of ranking
The interval uses the small-sample t multiplier rather than the normal 1.96. At three samples that is the difference between an honest interval and one narrow enough to declare a winner nobody earned
### The five states
| State | Meaning |
| -------------- | ------------------------------------------------------------------------------------------------ |
| `queued` | Accepted, not started |
| `running` | In progress |
| `complete` | Every enabled dimension produced a score and a verdict |
| `failed` | The run could not finish. The partial report is kept, not discarded |
| `inconclusive` | The run finished but cannot tell: intervals overlap, or a percentile fell below its sample floor |
`inconclusive` is a result, not a failure to produce one. It says this suite cannot separate these models
### What is deliberately not scored
An ungradable sample is never scored zero. A judge that timed out, a guardrail refusal, and a model that answered badly are three different facts, and averaging the first two in would blame the candidate for the run's own behaviour
Latency comes only from calls that were answered. A failed call records the time it took to fail and a blocked one the time it took to refuse, and neither says anything about how fast the model answers
Without a `p95_latency_budget_ms` the run reports latency and gives no verdict on it. A number with nothing to compare it to is not a verdict
## The judge
Freeform cases need a model to read the answer, and that is the run's main recurring expense. Set it on the control plane:
```
FORCEAI_EVALUATOR_JUDGE_MODEL=
```
The judge is called through the gateway on your own key, exactly like a candidate, so judging is routed, guardrailed, billed and logged rather than being a second LLM client nobody can see
A judge that is also a candidate **refuses the run**. A model asked to grade itself scores itself higher, and by the time anyone reads the report the money is spent, so this is refused up front rather than reported as a caveat
If the judge merely shares a provider with a candidate the run proceeds and the report carries a caveat. Family is not reliably recoverable from a deployment name, and refusing would leave an organization standardized on one provider unable to grade anything
A suite of only structured cases needs no judge at all
## Scores and the badge
Once a canonical run exists, every model carries a badge:
```
passed, weakest response_quality 0.96
failed on hallucination
stale, 94 days
not evaluated
```
The badge is never an average. It shows the verdict and the **weakest dimension**, because where a model is weak is what matters while scanning a list. A failing dimension is named even when a passing one scored lower
Staleness is reported ahead of the verdict. Providers change models under a stable name, so a pass from six months ago is not evidence about today's model, and showing `passed` with the age tucked elsewhere is how a gate stays satisfied forever on one old measurement
Scores resolve per model, not per run. A model the newest canonical run did not cover falls back to the most recent run that did, rather than reading as unevaluated
## Enforcement, and how to turn it on safely
| Level | Effect |
| ------------------- | ---------------------------------------------------------------- |
| `advisory` | Shows scores, marks unevaluated models, blocks nothing |
| `publication_gated` | Additionally refuses to publish a model that is not a fresh pass |
| `selection_gated` | Additionally keeps such a model away from non-admin keys |
It ships in `advisory`, and you should leave it there until the dry run is empty. Switching straight to `selection_gated` would make every currently-published model unevaluated overnight and pull your working models out from under your users. That is an outage caused by a governance feature, which is the fastest way to have the governance feature switched off
```bash theme={null}
curl -s "$FORCEAI_URL/v1/evaluations/policy" -H "Authorization: Bearer $FORCEAI_KEY"
```
The response carries the current level and exactly which models each stricter level would block, and why:
```
effective: advisory
publication_gated would block: (none)
selection_gated would block:
bedrock-haiku-4-5 not evaluated
forceai-auto not evaluated
forceai-auto-fallback not evaluated
```
The two levels are reported separately because they do not block the same population. Publication-gated only reaches models that are published, so its blast radius is the AI Hub; selection-gated reaches every model
Raise the level only once that list is empty or deliberately accepted:
```bash theme={null}
curl -s -X PUT "$FORCEAI_URL/v1/evaluations/policy" \
-H "Authorization: Bearer $FORCEAI_KEY" -H 'Content-Type: application/json' \
-d '{"level":"publication_gated"}'
```
The effective level for a model is the strictest that applies, exactly how budgets already resolve. A workspace may tighten an organization's rule and can never loosen it
## Evaluating without spending
Passive profiles read the spend logs and call no model, so they cost nothing and need no suite. They describe every model already carrying traffic: latency percentiles, time to first token, cost per 1k tokens, failure rate and cache hit rate
```bash theme={null}
curl -s "$FORCEAI_URL/v1/evaluations/passive?days=7" -H "Authorization: Bearer $FORCEAI_KEY"
```
A percentile below the sample floor is reported as `insufficient_samples` rather than as a number. A p95 over a handful of requests is one observation wearing a suit
Cache hits and failed requests are excluded from latency and cost. A cached response returns in single-digit milliseconds at no cost, and an errored request records a near-zero duration and no tokens, so leaving either in makes a model look faster and cheaper than it is
## Endpoints
| Method and path | What it does |
| -------------------------------------------------- | -------------------------------------------- |
| `POST /v1/evaluations/suites` | Create a suite |
| `GET /v1/evaluations/suites` | List suites |
| `POST /v1/evaluations/suites/{suite_id}/canonical` | Make this the suite that produces org scores |
| `POST /v1/evaluations` | Start a run, returns a run id |
| `GET /v1/evaluations/{run_id}` | Read a run and its report |
| `GET /v1/evaluations/scores` | Badges for every model, or for named models |
| `GET /v1/evaluations/policy` | Current enforcement level plus the dry run |
| `PUT /v1/evaluations/policy` | Set the enforcement level |
| `GET /v1/evaluations/passive` | Spend-log profiles, no tokens spent |
All require a proxy admin role. A run bills to your own key and can only reach models your key can already route to, so running an evaluation grants nobody any new access
A run stores scores and verdicts, and never the prompts sent or the completions returned. Persisting model output would make every evaluation a new prompt-retention surface, and a verdict carries no payload
## Troubleshooting
No canonical suite is designated, or no run has been made against it. Designate one under Suites, then start a run against it
The run reaches candidates through the gateway on your own key. Check the key is valid and can route to every candidate
The suite cannot tell these models apart at this sample count. Add harder cases, or raise `samples_per_case` to narrow the intervals
Pass `p95_latency_budget_ms` to get a verdict rather than a bare number
Either no judge is configured, or the judge is also one of the candidates. Set `FORCEAI_EVALUATOR_JUDGE_MODEL` to a model outside the candidate set
The table shows only the first segment of the uuid. Click it to open its report, or use the copy button beside it to get the whole value