# OpenTelemetry

Agent SDK sends traces, metrics, and logs to an OTLP collector when `OTEL_*` is set. Record with `@opentelemetry/api`, and there is no `defineOtel`.

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.example.com
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20$OTEL_TOKEN"
export OTEL_SERVICE_NAME=incident-agent
agent-sdk serve --dir . --dev
```

`http/protobuf` is the default, `http/json` works, `grpc` warns and falls back to `http/protobuf`, metrics use delta temporality, and Grafana Cloud rejects delta, so set `OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative` for Grafana Cloud or Prometheus. Providers load before `bot/agent.ts`, so a module-scope meter, tracer, or logger binds to them, and with no endpoint there are no exporters, `fetch` has no `traceparent`, and the session collector stays off.

## Variables

| Variable | Effect |
| --- | --- |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | Base URL. Appends `/v1/traces`, `/v1/metrics`, `/v1/logs`. |
| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | Traces URL, used as written. Overrides the base URL. |
| `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` | Metrics URL, used as written. Overrides the base URL. |
| `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` | Logs URL, used as written. Overrides the base URL. |
| `OTEL_EXPORTER_OTLP_HEADERS` | Sent on every export. URL-encode values. Reread on each export. |
| `OTEL_EXPORTER_OTLP_PROTOCOL` | `http/protobuf` (default) or `http/json`. |
| `OTEL_SERVICE_NAME` | `service.name`. Defaults to the agent name, then `unknown_service:node`. |
| `OTEL_RESOURCE_ATTRIBUTES` | `service.namespace`, `service.version`, `deployment.environment.name`. |
| `OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE` | `delta` (default), `cumulative`, or `lowmemory`. |
| `OTEL_LOGS_EXPORTER` | `otlp` exports logs. Unset, `none`, and `console` do not. |
| `OTEL_SDK_DISABLED` | `true` turns export off. |
| `OTEL_TRACES_EXPORTER` | `none` turns traces off. |
| `OTEL_METRICS_EXPORTER` | `none` turns metrics off. |
| `OTEL_LOG_USER_PROMPTS` | `1` adds prompt text to log records. |
| `OTEL_LOG_TOOL_CONTENT` | `1` adds tool payloads to log records. |

## Hosted agents

```ts
export default defineAgent({
  hosting: {
    egressDomains: ["otlp-gateway-prod-us-east-0.grafana.net"],
    secretNames: [
      "OTEL_EXPORTER_OTLP_ENDPOINT",
      "OTEL_EXPORTER_OTLP_HEADERS",
      "OTEL_SERVICE_NAME",
    ],
  },
});
```

`egressDomains` is a hostname with no scheme or path. Secrets arrive with the first request that has an endpoint, and that export uses those headers and `service.name`. A later `agent-sdk secret set` updates headers on the next export, including a rotated key. A URL, protocol, or signal-list change replaces the exporter. `service.name` is fixed on the first export that attaches, so redeploy to change it.

## What you can record

```ts
import { metrics, trace } from "@opentelemetry/api";
import { logs } from "@opentelemetry/api-logs";
import { currentConversationId } from "@grok/agent-sdk/otel";

const tracer = trace.getTracer("incident-agent");
const lookups = metrics.getMeter("incident-agent").createCounter("incident.lookups");
const logger = logs.getLogger("incident-agent");

lookups.add(1, { "cursor.conversation.id": currentConversationId() });
```

Record after secrets load. A span or log that ends before exporters attach is still sent, up to 2048 of each, a metric recorded before the first collect is included, and HTTP, token, and tool spans carry `cursor.conversation.id`, while a span started during a turn or a tool call gets that attribute, a value you set is kept, the id is not propagated, and a metric gets it only when you set it.

## Tag a turn

```ts
async execute({ key }, ctx) {
  ctx.telemetry.setTraceAttributes({ "work.item": key });
  ctx.telemetry.setMetricAttributes({ "work.item": key });
  ctx.telemetry.setLogAttributes({ "work.item": key });
  return await jira.getIssue(key);
}
```

`ctx.telemetry` writes the current turn's store. A span started in that turn, including a span Agent SDK starts, gets the trace keys. A log record Agent SDK emits gets the log keys. `cursor.token.usage` recorded for that turn gets the metric keys. The same store is still there for a later tool in the turn. With export on, a later turn of the session keeps it. With export off, the next turn does not.

```ts
import { telemetry } from "@grok/agent-sdk/telemetry";

telemetry.setTraceAttributes({ "work.item": ticket });
```

The global methods read the active context. Outside a turn they do nothing. Tags stay in this process. They are not written on the `baggage` header.

```promql
sum by (work_item) (cursor_token_usage_total)
```

Grafana renames `.` to `_`, and each distinct metric value is its own series, including a new id on every turn. Keys match `[A-Za-z][A-Za-z0-9._-]*` and are at most 128 characters, values are 1 to 256 characters, a session holds 32 trace keys, 32 metric keys, and 32 log keys, a later call replaces a key, and a metric value must not be a secret, an email, or a raw user id.

Outbound calls send `traceparent` only. With export off, the tags last for that turn, a span in the turn sees them, the next turn does not, and nothing is exported.

`ctx.host.otel` is deprecated, still runs this release, and logs one warning per process. `setAttributes` writes one map onto spans and token usage, and the calls above replace it.

Trace and log keys are accepted as written. A log record keeps a key Agent SDK already set. `setMetricAttributes` rejects `cursor.token.type`, `agent_sdk.channel`, `cursor.tool.name`, `cursor.tool.kind`, `cursor.tool.status`, `agent_sdk.turn_status`, `agent_sdk.subagent_name`, `agent_sdk.sdk_agent_id`, and `agent_sdk.sdk_agent_url`. The call throws and leaves the runtime value in place.

Prompt text and tool payloads stay off log records unless `OTEL_LOGS_EXPORTER=otlp` and `OTEL_LOG_USER_PROMPTS` or `OTEL_LOG_TOOL_CONTENT` is set.

## Limits

Custom span processors, samplers, views, your own SDK, library auto-instrumentation, and a `register()` hook are not supported, and `OTEL_TRACES_SAMPLER` and `OTEL_METRIC_EXPORT_INTERVAL` are read when the provider is built, before hosted secrets load, so set them and restart, because a later value only logs a warning. If a module replaces a provider Agent SDK registered, the global API uses that provider and Agent SDK shuts down a provider it created only when it is not the registered one, and if a context manager is already registered, Agent SDK logs a warning and leaves it, so load Agent SDK first or the turn store does not stay on the active context.

## Migrating from serve({ otel })

`serve({ otel })` is removed, so delete that option, `bot/otel.ts`, `agent/otel.ts`, and `bot/instrumentation.ts`, and do not call `defineOtel`. Each signal has one URL, and a second exporter for the same signal has no variable.

| Old option | Set this instead |
| --- | --- |
| `serviceName` | `OTEL_SERVICE_NAME` |
| `serviceVersion` | `service.version` in `OTEL_RESOURCE_ATTRIBUTES` |
| `resourceAttributes` | `OTEL_RESOURCE_ATTRIBUTES`, comma-separated, URL-encoded values |
| `exporters[].url` | `OTEL_EXPORTER_OTLP_ENDPOINT` (appends `/v1/traces`, `/v1/metrics`, `/v1/logs`) |
| `exporters[].url` with `signals` | `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`, or `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`, used as written |
| `exporters[].headers` | `OTEL_EXPORTER_OTLP_HEADERS`, comma-separated, URL-encoded values |
| `exporters[].protocol` | `OTEL_EXPORTER_OTLP_PROTOCOL`, or `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL`, `OTEL_EXPORTER_OTLP_METRICS_PROTOCOL`, `OTEL_EXPORTER_OTLP_LOGS_PROTOCOL` |
| `exporters[].signals` | The per-signal URL. `OTEL_TRACES_EXPORTER=none`, `OTEL_METRICS_EXPORTER=none`, or `OTEL_LOGS_EXPORTER=none` stops that signal. |
| `traces: false` | `OTEL_TRACES_EXPORTER=none` |
| `traces: true` | Leave `OTEL_TRACES_EXPORTER` unset. Traces start when an endpoint is set. |
| `traces.sampleRatio` | `OTEL_TRACES_SAMPLER=parentbased_traceidratio` and `OTEL_TRACES_SAMPLER_ARG`, before process start |
| `metrics: false` | `OTEL_METRICS_EXPORTER=none` |
| `metrics: true` | Leave `OTEL_METRICS_EXPORTER` unset. Metrics start when a metrics URL is set. |
| `metrics.exportIntervalMillis` | `OTEL_METRIC_EXPORT_INTERVAL`, milliseconds, before process start |
| `logs: false` | Unset `OTEL_LOGS_EXPORTER`, or set `none` |
| `logs: true` | `OTEL_LOGS_EXPORTER=otlp` |
| `logs.userPrompts` | `OTEL_LOGS_EXPORTER=otlp` and `OTEL_LOG_USER_PROMPTS=1` |
| `logs.toolContent` | `OTEL_LOGS_EXPORTER=otlp` and `OTEL_LOG_TOOL_CONTENT=1` |
| `otel: false` | No endpoint, or `OTEL_SDK_DISABLED=true` |

Put those names in `hosting.secretNames` and the collector host in `hosting.egressDomains`.

```bash
agent-sdk secret set my-agent OTEL_SERVICE_NAME OTEL_EXPORTER_OTLP_ENDPOINT OTEL_EXPORTER_OTLP_HEADERS
```