# Evals

An eval sends a fixed input to the real agent and checks the resulting
trajectory: whether the turn succeeded, which tools ran, and what shape
the answer took. Use evals to protect behavior you already understand,
not to discover what the prompt should do.

Evals run real model turns and real tools. Guard external writes before
running a suite against an agent that can post, merge, or deploy.

## Protect a tool decision

This smoke case asks for a PR verdict, requires the read-only inspection
tool, and fails if the agent tries to approve. The CLI reports all three
decisions together instead of stopping at the first miss.

Use the [Jev extension](/docs/guides/jev.md) when host code needs a typed
choice, score, or boolean before it writes.

```ts
// evals/readiness.eval.ts
import { defineEval, includes } from "@grok/agent-sdk/evals";

export default defineEval({
  tags: ["smoke"],
  async test(t) {
    await t.send(
      "Is https://github.com/acme/checkout/pull/42 ready to approve?"
    );

    t.succeeded();
    t.calledTool("inspect_pr");
    t.notCalledTool("approve_pr");
    t.check(t.reply, includes(/ready|blocked|approve/i));
  },
});
```

Evals live in the project-root `evals/` directory, beside `bot/`.
Blank `agent-sdk init` writes `evals/smoke.eval.ts`, which uses the scaffolded
`echo` tool and asserts `t.succeeded()` and `t.calledTool("echo")`.
It also writes this concurrency config:

```ts
// evals/evals.config.ts
import { defineEvalConfig } from "@grok/agent-sdk/evals";

export default defineEvalConfig({
  maxConcurrency: 20,
});
```

```text
agent-sdk eval --dir . --list
agent-sdk eval --dir . --tag smoke
```

## Check answer shape or quality

Prefer a deterministic shape check when the output has a contract. The
failure tells you which field was wrong, and the case remains stable
when the model changes its wording.

```ts
import { matches } from "@grok/agent-sdk/evals";
import { z } from "zod";

const verdict = JSON.parse(t.reply ?? "{}");
t.check(
  verdict,
  matches(
    z.object({
      ready: z.boolean(),
      blockers: z.array(z.string()),
    })
  )
);
```

Use a judge when correctness depends on meaning that a schema, regex, or
known value cannot capture:

```ts
t.judge
  .factuality("The lint step failed on src/sidebar.ts.")
  .atLeast(0.8);
```

Judge checks are tracked scores until you give them a hard gate.
Configure the judge model in `evals.config.ts`. The
[Evals reference](/docs/reference/evals.md#judges) owns grader options and
severity rules.

## Test a conversation

Several `t.send` calls in one case share a session. Assertions on a
returned turn inspect only that turn, while assertions on `t` inspect
the whole conversation.

```ts
const first = await t.send("Inspect PR 42.");
first.calledTool("inspect_pr");

const followUp = await t.send("Now summarize only the blockers.");
followUp.usedNoTools();
t.check(followUp.message, includes(/blocker/i));
t.succeeded();
```

## Pin realistic inputs

A useful eval changes only when the agent changes. Freeze canonical chat
prompts, seed workspace evidence directly, and save webhook payloads
instead of depending on a developer's checkout or live external state.

```ts
await t.send("Review pr/diff.patch and report blockers.", {
  workspaceFiles: {
    "pr/diff.patch": [
      "diff --git a/app/routes/search.ts b/app/routes/search.ts",
      "+res.send(`<h1>Results for ${req.query.q}</h1>`);",
    ].join("\n"),
  },
});
```

For GitHub agents, snapshot a PR into committed fixtures:

```text
agent-sdk github replay acme/checkout#42 \
  --events '*' \
  --dry-run \
  --out fixtures/github
```

Keep a small, fast `smoke` set for every change. Put larger datasets and
drift-prone live cases behind an explicit selection. See
[Datasets and fixtures](/docs/reference/evals.md#datasets-and-fixtures) for
loaders and case expansion.

## Run and debug a regression suite

Run one case while iterating, then the smoke suite before review. A
failure artifact records the input, assertions, tool calls, and final
text, making it the first place to look before changing the prompt.

```text
agent-sdk eval --dir . readiness
agent-sdk eval --dir . --tag smoke
agent-sdk eval --dir . --json
```

By default, results land under `evals/<stamp>/` in the project state
directory. Open `evals/<case-id>.json` inside that run directory for the
failed case. This artifact location is independent of `--state-root`;
pass `--artifacts <dir>` to choose another destination.

Model turns need a credential. Resolution checks `CURSOR_API_KEY`,
`CURSOR_API_KEY_FILE`, `CURSOR_SERVICE_ACCOUNT_KEY`, then a saved
`agent-sdk login`. `eval --list` only discovers cases and needs none.

## Run evals in CI

Write machine-readable output and JUnit annotations, then let a failed
gate fail the job:

```text
# CURSOR_API_KEY comes from the CI secret store.
agent-sdk eval --dir . --json --no-stream \
  --junit reports/evals.xml \
  --artifacts reports/evals \
  > reports/evals.json
```

Run `--tag smoke` on pull requests and the full suite on a schedule when
cost or latency makes every-push coverage impractical. Use soft scores
for a new probabilistic benchmark until its threshold is trustworthy.

The [CLI reference](/docs/reference/cli.md#eval) covers selectors,
server-side batches, exit codes, and JSON output.

## Keep side effects out of eval sessions

Eval sessions carry `purpose: "eval"`. Check it at the deterministic
write boundary and return the decision without performing the external
action:

```ts
async execute({ findings }, ctx) {
  if (ctx.session.purpose === "eval") {
    return {
      posted: false,
      reason: "eval",
      count: findings.length,
    };
  }

  return await postFindings(findings);
}
```

This preserves the tool call in the trajectory, so the eval can still
assert that the agent chose to post.

## Evals, hooks, or hillclimbing?

Use an eval to gate one fixed input. Use a
[hook](/docs/reference/hooks.md) to observe every live session for metrics,
auditing, or alerts. Use [hillclimbing](/docs/hillclimbing.md) to decide
which source change improves a fixed set of inputs; every kept
hillclimb change should add an eval.

## Gates and scores

`t.succeeded()`, tool assertions, and deterministic checks are hard
gates by default. A failed gate fails the case. `.soft()` records a
measurement without blocking, while `.atLeast(threshold)` records a
score and marks a miss as `scored`.

Start with decisions and output shape: the intended tool, the tempting
wrong tool, and one structural check. Exact prose is rarely a stable
contract.

## Keep improvements with regression evals

Every kept improvement needs an eval that would have failed before the
change. A tool-choice gate, bounded `maxToolCalls`, or output-shape check
turns the improvement into a durable contract.

Never weaken an existing gate to make a new implementation pass. That
moves the freeze line instead of proving the change.

## Related

- [Evals reference](/docs/reference/evals.md): complete authoring and runner
  contracts
- [`skills/evals/SKILL.md`](/docs/package/skills/evals/SKILL.md): author and seed
  cases with a coding agent
- [Hillclimbing](/docs/hillclimbing.md): measure a change and lock the win
- [Sessions](/docs/reference/sessions.md): events available to trajectory
  assertions