# PagerDuty agents

A PagerDuty agent wakes when an incident triggers, investigates it, and
writes what it found back to the incident. `pagerdutyChannel()` receives
PagerDuty's v3 webhooks, verifies them, and keeps every later event for
the same incident in one session, so the agent sees an acknowledgement,
a note, or a responder's reply in the context of the investigation it
already started. Every setting has a default read from a well-known secret, so
the channel needs no configuration beyond its hooks; see
[Connect](#connect) for the variables.

## Investigate a new incident

When an incident triggers, the agent starts a turn with the incident's
title, service, urgency, and link. It reads whatever the investigation
needs, and with `deliverReplies: true` its reply is posted as a note on
the incident, where the person on call already looks.

```ts
// bot/channels/pagerduty.ts
import { defaultPagerDutyAuth, pagerdutyChannel } from "@grok/agent-sdk/channels/pagerduty";

export default pagerdutyChannel({
  deliverReplies: true,
  onIncident: (ctx, event) => {
    if (event.eventType !== "incident.triggered") {
      return null;
    }
    return {
      // `auth` starts a model turn running as the event's PagerDuty actor.
      auth: defaultPagerDutyAuth(ctx),
      message: `Investigate PagerDuty incident #${event.incident.number}: ${event.incident.title}`,
      context: [`Service: ${event.incident.service?.summary ?? "unknown"}`],
    };
  },
});
```

Reply notes are off unless you set `deliverReplies: true`, so the channel
writes nothing to PagerDuty on its own. An agent that writes to PagerDuty
through its own path, such as a service that audits each write, leaves it
off and posts only what it chooses, from `events` or `ctx.pagerduty`.

With `deliverReplies: true`, the channel posts each completed reply as a
note on the session's incident, and a short note if a turn fails, using
the `PAGERDUTY_TOKEN` secret. The note goes to the incident in the
verified event's state, so the model never chooses which incident it
writes to. Eval runs and agents without a token post nothing. Nothing is
posted until the channel knows the agent's own PagerDuty user (see
[Ignore your own actions](#ignore-your-own-actions)), because each note
comes back as a webhook event that only the echo skip keeps from starting
another turn. Handlers you write in `events` replace the built-in ones.

Hooks also get `ctx.pagerduty`, a REST client with the channel's
credentials, for reads and writes of their own:

```ts
onIncident: async (ctx, event) => {
  const incident = await ctx.pagerduty.getIncident(event.incident.id);
  // …
},
```

Hosted, the agent also declares PagerDuty's API host and the token:

```ts
// bot/agent.ts
import { defineAgent } from "@grok/agent-sdk";

export default defineAgent({
  architecture: "v2",
  hosting: {
    egressDomains: ["api.pagerduty.com"],
    secretNames: ["PAGERDUTY_TOKEN"],
  },
});
```

## Follow the incident in one session

People act on an incident while the agent investigates: someone
acknowledges it, adds a note, pulls in a responder, or resolves it.
Return a dispatch for those events too, and each one arrives in the
session that is already working on the incident, with the earlier
findings in context.

```ts
onIncident: (ctx, event) => {
  const followUps = ["incident.acknowledged", "incident.reassigned", "incident.resolved"];
  if (event.eventType !== "incident.triggered" && !followUps.includes(event.eventType)) {
    return null;
  }
  return {
    auth: defaultPagerDutyAuth(ctx),
    message: `${event.eventType} by ${ctx.agent?.summary ?? "PagerDuty"}`,
  };
},
onNote: (ctx, note) => {
  return {
    auth: defaultPagerDutyAuth(ctx),
    message: `Note on the incident: ${note.content}`,
  };
},
```

Add these to the same `pagerdutyChannel()` declaration as the
investigation handler.

## Ignore your own actions

PagerDuty sends the agent's own changes back as webhook events: the
note it posted, or an acknowledgement or resolve made through its
PagerDuty user. The channel acknowledges those events without running a
hook or starting a turn. The same action by a person is still delivered.

The channel finds the agent's user in this order, on each delivery:

1. The `selfUserIds` option, when you pass one. Nothing below is used.
2. The `PAGERDUTY_USER_ID` secret: one PagerDuty user id.
3. The user `PAGERDUTY_TOKEN` acts as, which PagerDuty reports at
   `GET /users/me`.

When none of these names a user, echo skip is off. `agent-sdk deploy` warns
while neither `PAGERDUTY_USER_ID` nor `PAGERDUTY_TOKEN` is set, and the
agent logs a warning when its token names no single user.

Inference depends on the kind of token:

| `PAGERDUTY_TOKEN` | Inferred user |
| --- | --- |
| User token REST API key | The key's own user |
| OAuth user token (Classic User OAuth or Scoped OAuth), written as `Bearer <token>` | The user who authorized the app |
| General access (account-level) REST API key | The user whose email is exactly `PAGERDUTY_FROM_EMAIL`, if set |
| Scoped OAuth app token (client credentials) | The user whose email is exactly `PAGERDUTY_FROM_EMAIL`, if set |

PagerDuty serves
[`GET /users/me`](https://developer.pagerduty.com/api-reference/0b6f8b0c3e8d1-get-the-current-user)
only to a user-level API key or a token from an OAuth flow, not to an
account-level key. An app token
[acts as the app](https://docs.pagerduty.com/developer/private-apps),
not as a user. With either, the channel looks up `PAGERDUTY_FROM_EMAIL`,
the `From` user the token writes as, through `GET /users?query=` and uses
the one user whose email matches exactly. The inferred id is kept in the
agent's memory for an hour per token, never in agent storage, and an
unresolved answer is retried after five minutes. Hosted, the lookup needs
`api.pagerduty.com` (or your `PAGERDUTY_API_BASE_URL` host) in
`hosting.egressDomains`, the same entry write-backs need.

A read-only agent with no token sets the id directly:

```bash
agent-sdk secret set incident-responder PAGERDUTY_USER_ID
```

`PAGERDUTY_USER_ID` holds one id because a deployment has one token
and so acts as one PagerDuty user. To skip several users, or to keep the
id in code, pass `selfUserIds`:

```ts
export default pagerdutyChannel({
  selfUserIds: ["PBOT001"],
  onIncident: (ctx) => ({ auth: defaultPagerDutyAuth(ctx) }),
});
```

`selfUserIds: { secretName: "OTHER_NAME" }` reads one id from another
secret and never falls back to inference.

## Events

Each hook receives one family of v3 events:

| Hook | Events |
| --- | --- |
| `onIncident` | `incident.triggered`, `.acknowledged`, `.unacknowledged`, `.escalated`, `.reassigned`, `.delegated`, `.resolved`, `.reopened`, `.priority_updated`, `.incident_type.changed` |
| `onNote` | `incident.annotated` |
| `onResponder` | `incident.responder.added`, `incident.responder.replied` |
| `onEvent` | Everything else, such as status updates and workflow events |

Without `onIncident`, a triggered or reopened incident starts a turn and
other lifecycle events are acknowledged. Notes, responder events, and
the rest are acknowledged until you add their hook. Each hook returns
one of three decisions:

| Return | What happens |
| --- | --- |
| `{ auth }` | Start or continue a model turn as the event's actor |
| `{ task }` | Acknowledge the webhook, then run host code that may also call `send` |
| `null` | Skip the event |

## Incident affinity

Every event for one incident joins one session, keyed on the incident
id. Lifecycle events carry the incident, and notes and responder events
name it by reference, so all of them reach the same session.

A re-trigger after a resolve is a new PagerDuty incident, so it starts a
new session. `ctx.incidentKey` carries the dedup key of the alert that
opened an incident (its `incident_key`, set per alert by a Datadog
monitor, a Sentry issue, or a Grafana rule), for an agent that wants to
recall what it found the last time the same alert fired.

## Connect

Create a v3 webhook subscription in PagerDuty for the services that
should wake the agent, and point it at
`https://<your-host>/<slug>/v1/channels/pagerduty`. Then set the
channel's secrets on the deployment. Locally the same names are
environment variables.

| Variable | Used for | Override |
| --- | --- | --- |
| `PAGERDUTY_WEBHOOK_SECRET` | Verifying `X-PagerDuty-Signature`. Hosted, the platform checks it before any compute starts and answers a bad delivery with `401` | `webhookSecret: { secretName }` |
| `PAGERDUTY_USER_ID` | Skipping the agent's own actions: its PagerDuty user id. Optional when `PAGERDUTY_TOKEN` names a user | `selfUserIds: [ids]` or `{ secretName }` |
| `PAGERDUTY_TOKEN` | Reply notes (with `deliverReplies: true`), `ctx.pagerduty`, and inferring the agent's user: a REST API key, or an OAuth token as `Bearer <token>` | `token: { secretName }` |
| `PAGERDUTY_FROM_EMAIL` | The `From` user PagerDuty requires for writes made with an account-level key or app token, and the user inferred for those tokens | `from: { secretName }` |
| `PAGERDUTY_API_BASE_URL` | The REST host, such as `https://api.eu.pagerduty.com`. Defaults to `https://api.pagerduty.com` | `apiBaseUrl` |

Set `token`, `from`, and `apiBaseUrl` on `pagerdutyChannel()`; they cover
reply notes, `ctx.pagerduty`, and inference. `createPagerDutyClient()`
takes the same options for code outside the channel.

```bash
agent-sdk deploy --dir . --slug incident-responder
agent-sdk secret set incident-responder PAGERDUTY_WEBHOOK_SECRET PAGERDUTY_TOKEN
agent-sdk secret list incident-responder
```

`agent-sdk deploy` warns until `PAGERDUTY_WEBHOOK_SECRET` is set, and until
either `PAGERDUTY_USER_ID` or `PAGERDUTY_TOKEN` is. The next delivery
uses a new value without another deploy.

### Several subscriptions

PagerDuty gives each webhook subscription its own signing secret, so an
agent fed by several subscriptions (one per service, say) needs all of
them. Put them in `PAGERDUTY_WEBHOOK_SECRET`, one per line:

```text
<signing secret of subscription 1>
<signing secret of subscription 2>
```

A delivery is admitted when any `v1=` signature in its
`X-PagerDuty-Signature` header matches any line. Each line is trimmed,
blank lines are ignored, and a value without a line break is one secret,
used as-is. The value holds at most 32 secrets. Past that, every delivery
is rejected rather than checked against only some of them.

Keep the list in a file, since `agent-sdk secret list` never shows values,
and set it from stdin, which keeps the line breaks:

```bash
agent-sdk secret set incident-responder PAGERDUTY_WEBHOOK_SECRET < pagerduty-webhook-secrets.txt
```

Rotate by adding before removing. Add the new secret as a line before
PagerDuty starts signing with it, for example before creating the
replacement subscription. Remove the old line only once nothing signs
with the old secret, for example after deleting the subscription it
replaced. While both are live, a delivery signed with either secret
verifies, including one whose header carries both signatures. Removing
the old line first rejects those deliveries until the new one is added.

The same format works for every `hmacSignatureAuth()` secret, including
`GITHUB_WEBHOOK_SECRET` and `BITBUCKET_WEBHOOK_SECRET`. It needs
`@grok/agent-sdk` 0.2.25 or later. Earlier versions check the signature again
in the agent with the whole value as one secret, so they reject every
delivery once the value has a second line.

Only signature checks read the value as a list. Everywhere else it is
used whole: a `sharedSecretAuth()` header compares the entire value, and
a `${NAME}` template in an MCP connection's `headers` would put the line
breaks into the header. Keep a list secret for webhook signing only.

Until `PAGERDUTY_WEBHOOK_SECRET` is set, the route accepts only local
traffic: loopback deliveries under `agent-sdk dev`, nothing in production.

## Test locally

`agent-sdk pagerduty replay` posts synthesized v3 deliveries for one incident
to the running agent, in order. With `PAGERDUTY_WEBHOOK_SECRET` unset,
`agent-sdk dev` admits them unsigned from loopback; with it set, replay signs
them the way PagerDuty does:

```bash
agent-sdk dev &
agent-sdk pagerduty replay --events triggered,acknowledged,annotated,resolved \
  --comment "Rolled back the 09:30 deploy."
```

The trigger comes from PagerDuty itself, and the later events come from a
person, so each one joins the session the trigger started. Pass an
incident id to start another session, `--dry-run` to print the bodies,
or `--out fixtures/pagerduty` to save them as fixtures for
[evals](/docs/evals.md). `agent-sdk pagerduty events` lists the channels and the
events each accepts.

To post a real delivery saved from the webhook's delivery log, sign it
with the secret:

```bash
export PAGERDUTY_WEBHOOK_SECRET=local-test-secret
agent-sdk dev &

body=fixtures/pagerduty/incident-triggered.json
signature=$(openssl dgst -sha256 -hmac "$PAGERDUTY_WEBHOOK_SECRET" -hex < "$body" | awk '{print $NF}')
curl -X POST http://127.0.0.1:3000/v1/channels/pagerduty \
  -H 'content-type: application/json' \
  -H "x-pagerduty-signature: v1=$signature" \
  --data-binary @"$body"
```

Set `event.agent.id` in a saved body to the agent's own user id to see
the delivery skipped as an echo.

## Practices

- Write back from host code with the incident in channel state or
  `ctx`, never from an incident id the model produced.

- Make sure the agent knows its own user before it acts on incidents, so
  its own notes and resolves do not wake it again: a user-level token, a
  `PAGERDUTY_FROM_EMAIL` with an account-level key, or `PAGERDUTY_USER_ID`.

- Scope the webhook subscription to the services the agent responds to;
  every delivery it sends reaches the channel.

## Related

- [Channels reference](/docs/reference/channels.md): channel hooks,
  dispatch decisions, and session behavior
- [Webhooks and custom channels](/docs/guides/webhooks.md): direct webhook auth
  and reply delivery
- [Evals](/docs/evals.md): turn saved PagerDuty payloads into regression
  fixtures