PagerDuty agents

A PagerDuty agent wakes when an incident triggers, investigates it, and writes what it found back to the incident. pagerdutyChannel() receives PagerDuty's v3 webhooks, verifies them, and keeps every later event for the same incident in one session, so the agent sees an acknowledgement, a note, or a responder's reply in the context of the investigation it already started. Every setting has a default read from a well-known secret, so the channel needs no configuration beyond its hooks; see Connect for the variables.

Investigate a new incident

When an incident triggers, the agent starts a turn with the incident's title, service, urgency, and link. It reads whatever the investigation needs, and with deliverReplies: true its reply is posted as a note on the incident, where the person on call already looks.

// bot/channels/pagerduty.ts
import { defaultPagerDutyAuth, pagerdutyChannel } from "@grok/agent-sdk/channels/pagerduty";

export default pagerdutyChannel({
  deliverReplies: true,
  onIncident: (ctx, event) => {
    if (event.eventType !== "incident.triggered") {
      return null;
    }
    return {
      // `auth` starts a model turn running as the event's PagerDuty actor.
      auth: defaultPagerDutyAuth(ctx),
      message: `Investigate PagerDuty incident #${event.incident.number}: ${event.incident.title}`,
      context: [`Service: ${event.incident.service?.summary ?? "unknown"}`],
    };
  },
});

Reply notes are off unless you set deliverReplies: true, so the channel writes nothing to PagerDuty on its own. An agent that writes to PagerDuty through its own path, such as a service that audits each write, leaves it off and posts only what it chooses, from events or ctx.pagerduty.

With deliverReplies: true, the channel posts each completed reply as a note on the session's incident, and a short note if a turn fails, using the PAGERDUTY_TOKEN secret. The note goes to the incident in the verified event's state, so the model never chooses which incident it writes to. Eval runs and agents without a token post nothing. Nothing is posted until the channel knows the agent's own PagerDuty user (see Ignore your own actions), because each note comes back as a webhook event that only the echo skip keeps from starting another turn. Handlers you write in events replace the built-in ones.

Hooks also get ctx.pagerduty, a REST client with the channel's credentials, for reads and writes of their own:

onIncident: async (ctx, event) => {
  const incident = await ctx.pagerduty.getIncident(event.incident.id);
  // …
},

Hosted, the agent also declares PagerDuty's API host and the token:

// bot/agent.ts
import { defineAgent } from "@grok/agent-sdk";

export default defineAgent({
  architecture: "v2",
  hosting: {
    egressDomains: ["api.pagerduty.com"],
    secretNames: ["PAGERDUTY_TOKEN"],
  },
});

Follow the incident in one session

People act on an incident while the agent investigates: someone acknowledges it, adds a note, pulls in a responder, or resolves it. Return a dispatch for those events too, and each one arrives in the session that is already working on the incident, with the earlier findings in context.

onIncident: (ctx, event) => {
  const followUps = ["incident.acknowledged", "incident.reassigned", "incident.resolved"];
  if (event.eventType !== "incident.triggered" && !followUps.includes(event.eventType)) {
    return null;
  }
  return {
    auth: defaultPagerDutyAuth(ctx),
    message: `${event.eventType} by ${ctx.agent?.summary ?? "PagerDuty"}`,
  };
},
onNote: (ctx, note) => {
  return {
    auth: defaultPagerDutyAuth(ctx),
    message: `Note on the incident: ${note.content}`,
  };
},

Add these to the same pagerdutyChannel() declaration as the investigation handler.

Ignore your own actions

PagerDuty sends the agent's own changes back as webhook events: the note it posted, or an acknowledgement or resolve made through its PagerDuty user. The channel acknowledges those events without running a hook or starting a turn. The same action by a person is still delivered.

The channel finds the agent's user in this order, on each delivery:

1. The selfUserIds option, when you pass one. Nothing below is used.
2. The PAGERDUTY_USER_ID secret: one PagerDuty user id.
3. The user PAGERDUTY_TOKEN acts as, which PagerDuty reports at GET /users/me.

When none of these names a user, echo skip is off. agent-sdk deploy warns while neither PAGERDUTY_USER_ID nor PAGERDUTY_TOKEN is set, and the agent logs a warning when its token names no single user.

Inference depends on the kind of token:

PAGERDUTY_TOKEN	Inferred user
User token REST API key	The key's own user
OAuth user token (Classic User OAuth or Scoped OAuth), written as Bearer <token>	The user who authorized the app
General access (account-level) REST API key	The user whose email is exactly PAGERDUTY_FROM_EMAIL, if set
Scoped OAuth app token (client credentials)	The user whose email is exactly PAGERDUTY_FROM_EMAIL, if set

PagerDuty serves GET /users/me only to a user-level API key or a token from an OAuth flow, not to an account-level key. An app token acts as the app, not as a user. With either, the channel looks up PAGERDUTY_FROM_EMAIL, the From user the token writes as, through GET /users?query= and uses the one user whose email matches exactly. The inferred id is kept in the agent's memory for an hour per token, never in agent storage, and an unresolved answer is retried after five minutes. Hosted, the lookup needs api.pagerduty.com (or your PAGERDUTY_API_BASE_URL host) in hosting.egressDomains, the same entry write-backs need.

A read-only agent with no token sets the id directly:

agent-sdk secret set incident-responder PAGERDUTY_USER_ID

PAGERDUTY_USER_ID holds one id because a deployment has one token and so acts as one PagerDuty user. To skip several users, or to keep the id in code, pass selfUserIds:

export default pagerdutyChannel({
  selfUserIds: ["PBOT001"],
  onIncident: (ctx) => ({ auth: defaultPagerDutyAuth(ctx) }),
});

selfUserIds: { secretName: "OTHER_NAME" } reads one id from another secret and never falls back to inference.

Events

Each hook receives one family of v3 events:

Hook	Events
onIncident	incident.triggered, .acknowledged, .unacknowledged, .escalated, .reassigned, .delegated, .resolved, .reopened, .priority_updated, .incident_type.changed
onNote	incident.annotated
onResponder	incident.responder.added, incident.responder.replied
onEvent	Everything else, such as status updates and workflow events

Without onIncident, a triggered or reopened incident starts a turn and other lifecycle events are acknowledged. Notes, responder events, and the rest are acknowledged until you add their hook. Each hook returns one of three decisions:

Return	What happens
{ auth }	Start or continue a model turn as the event's actor
{ task }	Acknowledge the webhook, then run host code that may also call send
null	Skip the event

Incident affinity

Every event for one incident joins one session, keyed on the incident id. Lifecycle events carry the incident, and notes and responder events name it by reference, so all of them reach the same session.

A re-trigger after a resolve is a new PagerDuty incident, so it starts a new session. ctx.incidentKey carries the dedup key of the alert that opened an incident (its incident_key, set per alert by a Datadog monitor, a Sentry issue, or a Grafana rule), for an agent that wants to recall what it found the last time the same alert fired.

Connect

Create a v3 webhook subscription in PagerDuty for the services that should wake the agent, and point it at https://<your-host>/<slug>/v1/channels/pagerduty. Then set the channel's secrets on the deployment. Locally the same names are environment variables.

Variable	Used for	Override
PAGERDUTY_WEBHOOK_SECRET	Verifying X-PagerDuty-Signature. Hosted, the platform checks it before any compute starts and answers a bad delivery with 401	webhookSecret: { secretName }
PAGERDUTY_USER_ID	Skipping the agent's own actions: its PagerDuty user id. Optional when PAGERDUTY_TOKEN names a user	selfUserIds: [ids] or { secretName }
PAGERDUTY_TOKEN	Reply notes (with deliverReplies: true), ctx.pagerduty, and inferring the agent's user: a REST API key, or an OAuth token as Bearer <token>	token: { secretName }
PAGERDUTY_FROM_EMAIL	The From user PagerDuty requires for writes made with an account-level key or app token, and the user inferred for those tokens	from: { secretName }
PAGERDUTY_API_BASE_URL	The REST host, such as https://api.eu.pagerduty.com. Defaults to https://api.pagerduty.com	apiBaseUrl

Set token, from, and apiBaseUrl on pagerdutyChannel(); they cover reply notes, ctx.pagerduty, and inference. createPagerDutyClient() takes the same options for code outside the channel.

agent-sdk deploy --dir . --slug incident-responder
agent-sdk secret set incident-responder PAGERDUTY_WEBHOOK_SECRET PAGERDUTY_TOKEN
agent-sdk secret list incident-responder

agent-sdk deploy warns until PAGERDUTY_WEBHOOK_SECRET is set, and until either PAGERDUTY_USER_ID or PAGERDUTY_TOKEN is. The next delivery uses a new value without another deploy.

Several subscriptions

PagerDuty gives each webhook subscription its own signing secret, so an agent fed by several subscriptions (one per service, say) needs all of them. Put them in PAGERDUTY_WEBHOOK_SECRET, one per line:

<signing secret of subscription 1>
<signing secret of subscription 2>

A delivery is admitted when any v1= signature in its X-PagerDuty-Signature header matches any line. Each line is trimmed, blank lines are ignored, and a value without a line break is one secret, used as-is. The value holds at most 32 secrets. Past that, every delivery is rejected rather than checked against only some of them.

Keep the list in a file, since agent-sdk secret list never shows values, and set it from stdin, which keeps the line breaks:

agent-sdk secret set incident-responder PAGERDUTY_WEBHOOK_SECRET < pagerduty-webhook-secrets.txt

Rotate by adding before removing. Add the new secret as a line before PagerDuty starts signing with it, for example before creating the replacement subscription. Remove the old line only once nothing signs with the old secret, for example after deleting the subscription it replaced. While both are live, a delivery signed with either secret verifies, including one whose header carries both signatures. Removing the old line first rejects those deliveries until the new one is added.

The same format works for every hmacSignatureAuth() secret, including GITHUB_WEBHOOK_SECRET and BITBUCKET_WEBHOOK_SECRET. It needs @grok/agent-sdk 0.2.25 or later. Earlier versions check the signature again in the agent with the whole value as one secret, so they reject every delivery once the value has a second line.

Only signature checks read the value as a list. Everywhere else it is used whole: a sharedSecretAuth() header compares the entire value, and a ${NAME} template in an MCP connection's headers would put the line breaks into the header. Keep a list secret for webhook signing only.

Until PAGERDUTY_WEBHOOK_SECRET is set, the route accepts only local traffic: loopback deliveries under agent-sdk dev, nothing in production.

Test locally

agent-sdk pagerduty replay posts synthesized v3 deliveries for one incident to the running agent, in order. With PAGERDUTY_WEBHOOK_SECRET unset, agent-sdk dev admits them unsigned from loopback; with it set, replay signs them the way PagerDuty does:

agent-sdk dev &
agent-sdk pagerduty replay --events triggered,acknowledged,annotated,resolved \
  --comment "Rolled back the 09:30 deploy."

The trigger comes from PagerDuty itself, and the later events come from a person, so each one joins the session the trigger started. Pass an incident id to start another session, --dry-run to print the bodies, or --out fixtures/pagerduty to save them as fixtures for evals. agent-sdk pagerduty events lists the channels and the events each accepts.

To post a real delivery saved from the webhook's delivery log, sign it with the secret:

export PAGERDUTY_WEBHOOK_SECRET=local-test-secret
agent-sdk dev &

body=fixtures/pagerduty/incident-triggered.json
signature=$(openssl dgst -sha256 -hmac "$PAGERDUTY_WEBHOOK_SECRET" -hex < "$body" | awk '{print $NF}')
curl -X POST http://127.0.0.1:3000/v1/channels/pagerduty \
  -H 'content-type: application/json' \
  -H "x-pagerduty-signature: v1=$signature" \
  --data-binary @"$body"

Set event.agent.id in a saved body to the agent's own user id to see the delivery skipped as an echo.

Practices

• Write back from host code with the incident in channel state or ctx, never from an incident id the model produced.

• Make sure the agent knows its own user before it acts on incidents, so its own notes and resolves do not wake it again: a user-level token, a PAGERDUTY_FROM_EMAIL with an account-level key, or PAGERDUTY_USER_ID.

• Scope the webhook subscription to the services the agent responds to; every delivery it sends reaches the channel.

Related

• Channels reference: channel hooks, dispatch decisions, and session behavior
• Webhooks and custom channels: direct webhook auth and reply delivery
• Evals: turn saved PagerDuty payloads into regression fixtures