# Hillclimbing

Hillclimbing improves an agent through controlled before-and-after
comparisons. Keep the inputs fixed, name one failure, change one lever,
and retain the change only when the same run gets better without
crossing the freeze line.

The companion [`hillclimb` skill](/docs/package/skills/hillclimb/SKILL.md) can run
this loop with you.

## Correct a wrong tool choice

Suppose a PR reviewer calls `approve_pr` before `inspect_pr`. Pin one PR
fixture, record the original tool order, then tighten the instruction
that owns the decision. The improved run should inspect first and leave
approval untouched.

```text
agent-sdk run --dir . \
  --message "Is https://github.com/acme/checkout/pull/42 ready?"
```

Lock the result with the intended and tempting-wrong paths:

```ts
t.succeeded();
t.calledTool("inspect_pr");
t.notCalledTool("approve_pr");
```

Do not change the tool implementation, prompt, and evidence together.
If tool choice improves, you should know which edit caused it.

## Reduce wandering or latency

A slow run often repeats reads, searches broad directories, or asks the
model to discover evidence the host already has. Compare the trajectory
by tool name and wall time, then move deterministic preparation ahead
of the turn or trim the evidence shape.

```text
before: 14 tool calls, 92 seconds, 6 repeated reads
after:   5 tool calls, 31 seconds, 0 repeated reads
```

Keep the change only if the answer still finds the same known issue.
Efficiency without a quality check can make an agent fast by making it
incomplete.

## Improve quality without changing the contract

For answer quality, freeze the public shape before editing: required
sections, API fields, side effects, and existing evals. Compare both
runs against a known-good answer or a small set of gold findings, then
change the instructions or skill that owns the missing judgment.

```text
fixture: saved checkout PR diff
success: report both gold findings with file paths
freeze:  findings JSON shape; no review post; existing smoke evals
```

A wording change is not a win by itself. Keep it when correctness rises
and every frozen behavior remains intact.

## Run one measured round

Agree on four inputs before editing:

1. **Target:** the agent path or slug.
2. **Fixtures:** one to three prompts, saved payloads, or pinned PRs.
3. **Success:** the metric or decision that should improve.
4. **Freeze line:** behavior that must not move.

Then run one loop:

1. Exercise the agent through the same surface a user would.
2. Read the trajectory and name the dominant failure in one sentence.
3. Change one lever.
4. Rerun the same fixtures.
5. Compare correctness, efficiency, and contract fit.
6. Keep, revert, or narrow the change.

Do not replace a fixture mid-round. A moving input makes the comparison
meaningless.

## Choose the smallest lever

Start with the earliest place that can remove the failure:

1. **Host preparation:** fetch deterministic evidence before the turn.
2. **Evidence shape:** trim, order, or label what the model receives.
3. **Instructions or skills:** clarify the decision procedure.
4. **Tool surface:** remove or gate tools that invite wandering.
5. **Framework:** change it only when the agent cannot express the fix.

Changing the host or evidence often improves both quality and latency
because the model spends less time searching for ground truth.

## Lock every kept win

Each retained change needs an eval that would have failed before it. Use
a tool-choice assertion, `maxToolCalls` bound, output-shape check, or
quality score that captures the measured improvement.

```text
agent-sdk eval --dir . --tag smoke
```

Never weaken an existing gate to make the round pass. The eval is the
record that the improvement survived after the fixture and source
change leave your working memory.

## Run the loop with a coding agent

Ask the coding agent to read
[`skills/hillclimb/SKILL.md`](/docs/package/skills/hillclimb/SKILL.md), then give
it the target, fixtures, success criterion, and freeze line. It will
measure the current run, propose one change, remeasure, and ask before
starting another round.

Use saved or local surfaces during a climb. Do not deploy or post to a
real external channel unless that side effect is part of an explicitly
approved fixture.

## Related

- [Evals](/docs/evals.md): author the regression check for a kept win
- [Evals reference](/docs/reference/evals.md): trajectory assertions and
  scores
- [Building agents with agents](/docs/building-with-agents.md): delegate
  creation and verification to a coding agent
- [Fix common agent problems](/docs/troubleshooting.md): diagnose a run
  that is broken instead of underperforming