# Evals

Evals run fixed cases against an agent and record whether its turns,
tools, events, and output meet a contract. Files under the project-root
`evals/` directory define cases with `@grok/agent-sdk/evals`; the runner
discovers their IDs, executes them, and reports every assertion. See the
[Evals guide](/docs/evals.md) for the regression workflow.

## Eval discovery and case IDs

Eval files live under the project-root `evals/` directory and end in
`.eval.ts` or `.eval.js`. The path under that directory becomes the
eval ID:

- `evals/readiness.eval.ts` becomes `readiness`.
- `evals/builds/api.eval.ts` becomes `builds/api`.
- `evals/builds/index.eval.ts` becomes `builds`.

Export one `defineEval` with either `test` or `cases`, not both. A case
ID appends one path segment to the file ID:

```ts
export default defineEval({
  tags: ["smoke"],
  cases: [
    {
      id: "checkout",
      async test(t) {
        await t.send("Review checkout PR 42.");
        t.succeeded();
      },
    },
    {
      id: "search",
      async test(t) {
        await t.send("Review search PR 7.");
        t.succeeded();
      },
    },
  ],
});
```

Those cases are `prs/checkout` and `prs/search` when the file is
`evals/prs.eval.ts`. Case IDs must be unique single path segments.

A file can also export an array of `defineEval` calls. The runner names
them with zero-padded indexes such as `sql/0000`. Named `cases` give
handwritten scenarios stable IDs; arrays fit loaded datasets.

`iterations` repeats one datapoint from 1 to 100 times. For
`iterations: 3`, `weather/nyc` expands to `weather/nyc/1`,
`weather/nyc/2`, and `weather/nyc/3`; selecting `weather/nyc` runs all
three.

## Eval configuration

Every running suite needs `evals/evals.config.ts` with
`maxConcurrency`:

```ts
import { defineEvalConfig } from "@grok/agent-sdk/evals";

export default defineEvalConfig({
  maxConcurrency: 20,
  timeoutMs: 180_000,
  judge: { model: "gpt-5.4-mini" },
});
```

| Option | Contract |
| --- | --- |
| `maxConcurrency` | Required; concurrent authored datapoints, from 1 to 200 |
| `timeoutMs` | Per-case timeout; case/file, CLI, then config precedence |
| `judge` | Default model for `t.judge` |
| `reporters` | Objects notified as cases and runs complete |
| `maxPlaygroundRuns` | Number of server-side batches kept in playground history; defaults to 20 |

A case can override `description`, `tags`, `timeoutMs`, `iterations`,
`judge`, `reporters`, and `metadata`. Case metadata merges over
file-level metadata; case reporters add to the file's reporters.

## Send turns in a case

`await t.send(message, options?)` runs one turn and waits until it
finishes or fails. Several sends in one test share the session.

The returned turn exposes its assistant `message`, `sessionId`,
`events`, ordered `toolCalls`, `ok`, and one-based `index`. Assertions
on the turn inspect only that turn; assertions on `t` inspect the whole
case. Run values include `t.reply`, `t.events`, `t.turns`,
`t.sessionId`, `t.iteration`, `t.iterations`, and the timeout
`t.signal`.

These options apply on the first send because they shape the session:

| Option | Contract |
| --- | --- |
| `workspaceFiles` | Relative path-to-content map seeded into the session workspace |
| `workspaceDir` | Absolute working directory for a new local session |

Use `turn.expectOk()` when later test steps depend on the turn
succeeding. It throws when the turn failed.

## Trajectory assertions

Assertions record failures without stopping the test, so one case
reports every violated contract.

| Assertion | Checks |
| --- | --- |
| `t.succeeded()` | Run finished without failure |
| `t.calledTool(name, matcher?)` | Matching tool request occurred |
| `t.notCalledTool(name)` | No request for that tool occurred |
| `t.loadedSkill(name)` | Agent opened a named skill |
| `t.toolOrder(names)` | Tool requests appeared in relative order |
| `t.usedNoTools()` | No tool was requested |
| `t.maxToolCalls(max)` | Tool-call count stayed under the bound |
| `t.noFailedActions()` | No tool result failed |
| `t.calledSubagent(name, matcher?)` | Matching subagent call occurred |
| `t.taggedArtifact(kind?, predicate?)` | Matching durable artifact was tagged |
| `t.event(type, matcher?)` | Matching session event occurred |
| `t.notEvent(type, matcher?)` | Matching session event did not occur |
| `t.eventOrder(matchers)` | Event groups appeared in relative order |
| `t.eventsSatisfy(label, predicate)` | Predicate accepted the event stream |
| `t.check(value, expectation)` | Value met an expectation builder |
| `t.score(name, value)` | Recorded a numeric score from 0 to 1 |

`t.requireToolCall`, `t.requireInputRequest`, and `t.require` return the
matched value and stop the remaining test body when no match exists.

## Tool and event matchers

Tool matchers can narrow by `input`, `output`, `status`, and `count`.
Values accept literals, regular expressions, or predicates. Object
literals partial-deep-match:

```ts
t.calledTool("inspect_pr", {
  input: { verdict: "review" },
  status: "completed",
  count: 1,
});
```

Tool status is `completed`, `failed`, `pending`, or `rejected`.
Subagent matchers accept `output`, `status`, `count`, and `callId`.
Event matchers accept `data` and `count`.

## Expectation builders

Import builders from `@grok/agent-sdk/evals`:

| Builder | Default severity |
| --- | --- |
| `includes(string | RegExp)` | Hard gate |
| `equals(value)` | Hard gate |
| `matches(schema)` | Hard gate |
| `satisfies(predicate, label)` | Hard gate |
| `similarity(expected)` | Soft score |

`matches` accepts Standard Schema implementations such as Zod.
`normalizedSimilarity(value, expected)` returns the same 0-to-1 value
used by `similarity`.

## Severity and verdicts

Every assertion returns a severity handle:

```ts
t.succeeded();
t.calledTool("search").soft();
t.check(t.reply, similarity("Expected answer")).atLeast(0.8);
t.judge.closedQA("names the failing step").gate(1);
```

- `.gate(threshold?)` fails the case when it misses.
- `.soft(threshold?)` records a value without failing.
- `.atLeast(threshold)` records a soft bar; a miss produces a `scored`
  verdict.

Case verdicts are `passed`, `failed`, `scored`, or `skipped`. The CLI
exits 1 for a failed gate. A scored result exits 0 unless `--strict` is
set. Exit 2 means the selection matched no cases.

## Judges

Use a judge for meaning that deterministic checks cannot express:

```ts
t.judge.factuality(expected).atLeast(0.8);
t.judge.summarizes(expected).atLeast(0.8);
t.judge.closedQA("names the root cause").gate(1);
t.judge.sql(expected).atLeast(0.9);
```

Each grades `t.reply` by default; pass `{ on }` to grade another value.
Judge configuration can live in the project config, eval, case, or
call, with the nearest value winning.

`t.judge.model(prompt)` returns a raw model reply for custom grading.
Treat agent output as untrusted input: wrap it with `fenceUntrusted` and
include `EVAL_JUDGE_INJECTION_GUARD` in the judge prompt.

## Judge host

A judge grades by calling a Cursor model. It needs a Cursor credential
(`agent-sdk login`, `CURSOR_API_KEY`, or `CURSOR_SERVICE_ACCOUNT_KEY`) and a
judge model. The grading session offers no tools and turns the local SDK
sandbox off, including any per-user sandbox config on the machine, so the
same case grades on macOS, a Linux cloud agent, and a Linux CI runner.

Without a credential, or when that credential belongs to a different
backend than the SDK will call, the case is skipped and the suite
continues. Without a judge model, the case fails and the assertion names
where to set one.

## Datasets and fixtures

Load committed JSON, JSONL, or YAML relative to the project root:

```ts
import { defineEval, includes } from "@grok/agent-sdk/evals";
import { loadYaml } from "@grok/agent-sdk/evals/loaders";

const rows = await loadYaml<
  Array<{ task: string; prompt: string; expected: string }>
>("evals/data/cases.yaml");

export default rows.map(row =>
  defineEval({
    description: row.task,
    async test(t) {
      await t.send(row.prompt);
      t.succeeded();
      t.check(t.reply, includes(row.expected));
    },
  })
);
```

## Reporters and results

Built-in reporters include `JUnit({ filePath, suiteName? })` and
`Artifacts({ dir })`. Custom reporters can expose `onRunStart`,
`onEvalComplete`, and `onRunComplete`.

JSON output contains run totals and one result per case. Results include
the case ID, verdict, assertions, session ID, inputs, final text, tools,
tool calls, metrics, logs, duration, metadata, tags, and any error or
skip reason.

## Select cases

```text
agent-sdk eval --dir . --list
agent-sdk eval --dir .
agent-sdk eval --dir . builds/checkout
agent-sdk eval --dir . --tag smoke
```

ID filters match an exact ID and its descendants. Repeated IDs use OR;
repeated tags use OR. When both are present, a case must match both
groups.

## Suite summary cases

Tag a case with `summary` when that case aggregates the other cases in
the same file. `EVAL_SUMMARY_TAG` is that string. Selecting the
aggregator by id, or with `--tag summary`, also runs every other case
from that file. A selection that includes no `summary` case stays as
you wrote it. `--list` shows the same set a run would execute.

```ts
import { defineEval, EVAL_SUMMARY_TAG } from "@grok/agent-sdk/evals";

export default defineEval({
  cases: [
    {
      id: "checkout",
      async test(t) {
        await t.send("Review checkout PR 42.");
        t.succeeded();
      },
    },
    {
      id: "overall",
      tags: [EVAL_SUMMARY_TAG],
      async test(t) {
        await t.send("Summarize the checkout review.");
        t.succeeded();
      },
    },
  ],
});
```

When that file is `evals/prs.eval.ts`, `agent-sdk eval --dir . prs/overall`
runs `prs/checkout` and `prs/overall`.

## Inspect run artifacts

Default run artifacts go under `evals/<stamp>/` in the project state
directory and include
`summary.json`, `results.jsonl`, and `evals/<case-id>.json`.
`--artifacts <dir>` chooses another destination; `--no-artifacts`
disables them. This location is independent of `--state-root`.

## Run evals against a server

The playground **Evals** view starts batches on its running server, and
`--url` targets a named running server:

```text
agent-sdk eval --url http://127.0.0.1:3000/vulnerability-scanner --tag smoke
agent-sdk eval get <eval-id> --url http://127.0.0.1:3000/vulnerability-scanner
agent-sdk eval cancel <eval-id> --url http://127.0.0.1:3000/vulnerability-scanner
```

Server-side batches use the target's discovered cases and return an eval
ID for `status` or `cancel`. Local reporter and concurrency flags don't
apply to `--url` runs.

## Related

- [Evals guide](/docs/evals.md)
- [CLI reference](/docs/reference/cli.md#eval)
- [Sessions](/docs/reference/sessions.md)
- [Artifacts](/docs/reference/artifacts.md)