Model Evaluations

Public benchmark scores say nothing about whether a model handles your contract language, under your guardrails, at your cost. The evaluator answers a narrower and more useful question: is this model fit for this workload, in this organization A run scores each candidate per dimension and never averages them into one number. Someone who reads only the headline should still be unable to promote a model that failed on safety

Two kinds of run, and only one counts

This distinction is what makes the whole feature trustworthy, so it is worth getting straight before anything else

An org score comes only from a run against the canonical suite, which admins own. It is what appears beside a model when someone is choosing one, and it is the only thing the publication gate reads An exploratory run is a validator’s ad-hoc bake-off on their own suite. It answers a question, it shows in run history, and it never becomes the org score Without that split the gate is theatre. Anyone who wanted a model unlocked would write a three-case suite it passes and self-certify

Case kinds

A suite is a list of cases. Each carries a kind that decides how it is graded, and the choice matters because two of the three cost nothing to grade

Kind	Graded by	Cost	Use it for
`json`	Output must validate against the case’s JSON Schema	Free	Structured extraction, any response with a contract
`tool_call`	Expected tool chosen, expected arguments matched as a subset	Free	Tool and function selection
`freeform`	An LLM judge, against your rubric or a reference answer	A judge call per sample	Prose quality, explanations, summaries

Structured cases should be the backbone of any suite. They are deterministic and free, which is what lets you run the same suite twice without the bill doubling

Walkthrough

Author a suite

In the dashboard, Model Evaluations -> Suites -> New suite. Over the API:

curl -s -X POST "$FORCEAI_URL/v1/evaluations/suites" \
  -H "Authorization: Bearer $FORCEAI_KEY" -H 'Content-Type: application/json' -d '{
  "name": "canonical-suite",
  "cases": [
    {"kind":"json","case_id":"person",
     "prompt":"Return ONLY a JSON object with keys name (string) and age (integer). No prose, no code fences.",
     "schema":{"type":"object","required":["name","age"],
               "properties":{"name":{"type":"string"},"age":{"type":"integer"}}}}
  ]}'

Make it canonical

curl -s -X POST "$FORCEAI_URL/v1/evaluations/suites/$SUITE_ID/canonical" \
  -H "Authorization: Bearer $FORCEAI_KEY"

Exactly one suite per organization is canonical. Designating a new one clears the previous holder, so there is never ambiguity about which numbers the badge and the gate read

Until you designate one, every model reports not evaluated. That is the correct answer, not an error: an organization that has not chosen a canonical suite has not measured anything

Run it

curl -s -X POST "$FORCEAI_URL/v1/evaluations" \
  -H "Authorization: Bearer $FORCEAI_KEY" -H 'Content-Type: application/json' -d '{
  "suite_id": "'"$SUITE_ID"'",
  "models": ["gemini-2.5-flash", "claude-haiku-4-5"],
  "samples_per_case": 3,
  "p95_latency_budget_ms": 15000
}'

{"run_id":"c9802e98-14ee-4448-8513-74d50bad6b6d","state":"queued"}

A run is asynchronous. Several models over a suite at three samples a case takes minutes, so the API returns a run id you poll

Read the report

curl -s "$FORCEAI_URL/v1/evaluations/$RUN_ID" -H "Authorization: Bearer $FORCEAI_KEY"

state: complete | samples/case: 3

  performance       claude-haiku-4-5   pass   p50 1458ms, p95 1540ms against a 15000ms budget
  performance       gemini-2.5-flash   pass   p50 2368ms, p95 2393ms against a 15000ms budget
  response_quality  claude-haiku-4-5   pass   [1.00-1.00] n=6
  response_quality  gemini-2.5-flash   pass   [1.00-1.00] n=6

notes: ['candidates could not be separated: their intervals overlap']

Both models passed and the run still declined to name a winner. That note is the design working: ranking two identical intervals by a difference smaller than the noise would be inventing a result

Reading a report properly

Every score is an interval

At any temperature above zero the same prompt gives different answers, so a single sample is an anecdote. Each case runs samples_per_case times (three by default) and the score carries a mean and a spread. Comparisons are made between intervals, and where they overlap the run says so instead of ranking

The interval uses the small-sample t multiplier rather than the normal 1.96. At three samples that is the difference between an honest interval and one narrow enough to declare a winner nobody earned

The five states

State	Meaning
`queued`	Accepted, not started
`running`	In progress
`complete`	Every enabled dimension produced a score and a verdict
`failed`	The run could not finish. The partial report is kept, not discarded
`inconclusive`	The run finished but cannot tell: intervals overlap, or a percentile fell below its sample floor

inconclusive is a result, not a failure to produce one. It says this suite cannot separate these models

What is deliberately not scored

An ungradable sample is never scored zero. A judge that timed out, a guardrail refusal, and a model that answered badly are three different facts, and averaging the first two in would blame the candidate for the run’s own behaviour Latency comes only from calls that were answered. A failed call records the time it took to fail and a blocked one the time it took to refuse, and neither says anything about how fast the model answers Without a p95_latency_budget_ms the run reports latency and gives no verdict on it. A number with nothing to compare it to is not a verdict

The judge

Freeform cases need a model to read the answer, and that is the run’s main recurring expense. Set it on the control plane:

FORCEAI_EVALUATOR_JUDGE_MODEL=<a model id>

The judge is called through the gateway on your own key, exactly like a candidate, so judging is routed, guardrailed, billed and logged rather than being a second LLM client nobody can see

A judge that is also a candidate refuses the run. A model asked to grade itself scores itself higher, and by the time anyone reads the report the money is spent, so this is refused up front rather than reported as a caveat

If the judge merely shares a provider with a candidate the run proceeds and the report carries a caveat. Family is not reliably recoverable from a deployment name, and refusing would leave an organization standardized on one provider unable to grade anything A suite of only structured cases needs no judge at all

Scores and the badge

Once a canonical run exists, every model carries a badge:

passed, weakest response_quality 0.96
failed on hallucination
stale, 94 days
not evaluated

The badge is never an average. It shows the verdict and the weakest dimension, because where a model is weak is what matters while scanning a list. A failing dimension is named even when a passing one scored lower

Staleness is reported ahead of the verdict. Providers change models under a stable name, so a pass from six months ago is not evidence about today’s model, and showing passed with the age tucked elsewhere is how a gate stays satisfied forever on one old measurement

Scores resolve per model, not per run. A model the newest canonical run did not cover falls back to the most recent run that did, rather than reading as unevaluated

Enforcement, and how to turn it on safely

Level	Effect
`advisory`	Shows scores, marks unevaluated models, blocks nothing
`publication_gated`	Additionally refuses to publish a model that is not a fresh pass
`selection_gated`	Additionally keeps such a model away from non-admin keys

It ships in advisory, and you should leave it there until the dry run is empty. Switching straight to selection_gated would make every currently-published model unevaluated overnight and pull your working models out from under your users. That is an outage caused by a governance feature, which is the fastest way to have the governance feature switched off

curl -s "$FORCEAI_URL/v1/evaluations/policy" -H "Authorization: Bearer $FORCEAI_KEY"

The response carries the current level and exactly which models each stricter level would block, and why:

effective: advisory
publication_gated would block: (none)
selection_gated would block:
   bedrock-haiku-4-5      not evaluated
   forceai-auto           not evaluated
   forceai-auto-fallback  not evaluated

The two levels are reported separately because they do not block the same population. Publication-gated only reaches models that are published, so its blast radius is the AI Hub; selection-gated reaches every model Raise the level only once that list is empty or deliberately accepted:

curl -s -X PUT "$FORCEAI_URL/v1/evaluations/policy" \
  -H "Authorization: Bearer $FORCEAI_KEY" -H 'Content-Type: application/json' \
  -d '{"level":"publication_gated"}'

The effective level for a model is the strictest that applies, exactly how budgets already resolve. A workspace may tighten an organization’s rule and can never loosen it

Evaluating without spending

Passive profiles read the spend logs and call no model, so they cost nothing and need no suite. They describe every model already carrying traffic: latency percentiles, time to first token, cost per 1k tokens, failure rate and cache hit rate

curl -s "$FORCEAI_URL/v1/evaluations/passive?days=7" -H "Authorization: Bearer $FORCEAI_KEY"

A percentile below the sample floor is reported as insufficient_samples rather than as a number. A p95 over a handful of requests is one observation wearing a suit

Cache hits and failed requests are excluded from latency and cost. A cached response returns in single-digit milliseconds at no cost, and an errored request records a near-zero duration and no tokens, so leaving either in makes a model look faster and cheaper than it is

Endpoints

Method and path	What it does
`POST /v1/evaluations/suites`	Create a suite
`GET /v1/evaluations/suites`	List suites
`POST /v1/evaluations/suites/{suite_id}/canonical`	Make this the suite that produces org scores
`POST /v1/evaluations`	Start a run, returns a run id
`GET /v1/evaluations/{run_id}`	Read a run and its report
`GET /v1/evaluations/scores`	Badges for every model, or for named models
`GET /v1/evaluations/policy`	Current enforcement level plus the dry run
`PUT /v1/evaluations/policy`	Set the enforcement level
`GET /v1/evaluations/passive`	Spend-log profiles, no tokens spent

All require a proxy admin role. A run bills to your own key and can only reach models your key can already route to, so running an evaluation grants nobody any new access

A run stores scores and verdicts, and never the prompts sent or the completions returned. Persisting model output would make every evaluation a new prompt-retention surface, and a verdict carries no payload

Troubleshooting

Every model says 'not evaluated'

No canonical suite is designated, or no run has been made against it. Designate one under Suites, then start a run against it

A run returned 'failed' with gateway HTTP 401

The run reaches candidates through the gateway on your own key. Check the key is valid and can route to every candidate

The report says candidates could not be separated

The suite cannot tell these models apart at this sample count. Add harder cases, or raise samples_per_case to narrow the intervals

Performance is inconclusive with 'no budget configured'

Pass p95_latency_budget_ms to get a verdict rather than a bare number

A freeform suite is refused before it starts

Either no judge is configured, or the judge is also one of the candidates. Set FORCEAI_EVALUATOR_JUDGE_MODEL to a model outside the candidate set

A run id from the scores table returns 404

The table shows only the first segment of the uuid. Click it to open its report, or use the copy button beside it to get the whole value

Get started

LLM providers

A2A agents

Skills

Knowledge Catalog

Single sign-on

Admin settings

Tools

Two kinds of run, and only one counts

Case kinds

Walkthrough

Reading a report properly

Every score is an interval

The five states

What is deliberately not scored

The judge

Scores and the badge

Enforcement, and how to turn it on safely

Evaluating without spending

Endpoints

Troubleshooting

​Two kinds of run, and only one counts

​Case kinds

​Walkthrough

​Reading a report properly

​Every score is an interval

​The five states

​What is deliberately not scored

​The judge

​Scores and the badge

​Enforcement, and how to turn it on safely

​Evaluating without spending

​Endpoints

​Troubleshooting

Two kinds of run, and only one counts

Case kinds

Walkthrough

Reading a report properly

Every score is an interval

The five states

What is deliberately not scored

The judge

Scores and the badge

Enforcement, and how to turn it on safely

Evaluating without spending

Endpoints

Troubleshooting