OpenTelemetry

Agent SDK sends traces, metrics, and logs to an OTLP collector when OTEL_* is set. Record with @opentelemetry/api, and there is no defineOtel.

export OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.example.com
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20$OTEL_TOKEN"
export OTEL_SERVICE_NAME=incident-agent
agent-sdk serve --dir . --dev

http/protobuf is the default, http/json works, grpc warns and falls back to http/protobuf, metrics use delta temporality, and Grafana Cloud rejects delta, so set OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative for Grafana Cloud or Prometheus. Providers load before bot/agent.ts, so a module-scope meter, tracer, or logger binds to them, and with no endpoint there are no exporters, fetch has no traceparent, and the session collector stays off.

Variables

Variable	Effect
OTEL_EXPORTER_OTLP_ENDPOINT	Base URL. Appends /v1/traces, /v1/metrics, /v1/logs.
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT	Traces URL, used as written. Overrides the base URL.
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT	Metrics URL, used as written. Overrides the base URL.
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT	Logs URL, used as written. Overrides the base URL.
OTEL_EXPORTER_OTLP_HEADERS	Sent on every export. URL-encode values. Reread on each export.
OTEL_EXPORTER_OTLP_PROTOCOL	http/protobuf (default) or http/json.
OTEL_SERVICE_NAME	service.name. Defaults to the agent name, then unknown_service:node.
OTEL_RESOURCE_ATTRIBUTES	service.namespace, service.version, deployment.environment.name.
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE	delta (default), cumulative, or lowmemory.
OTEL_LOGS_EXPORTER	otlp exports logs. Unset, none, and console do not.
OTEL_SDK_DISABLED	true turns export off.
OTEL_TRACES_EXPORTER	none turns traces off.
OTEL_METRICS_EXPORTER	none turns metrics off.
OTEL_LOG_USER_PROMPTS	1 adds prompt text to log records.
OTEL_LOG_TOOL_CONTENT	1 adds tool payloads to log records.

Hosted agents

export default defineAgent({
  hosting: {
    egressDomains: ["otlp-gateway-prod-us-east-0.grafana.net"],
    secretNames: [
      "OTEL_EXPORTER_OTLP_ENDPOINT",
      "OTEL_EXPORTER_OTLP_HEADERS",
      "OTEL_SERVICE_NAME",
    ],
  },
});

egressDomains is a hostname with no scheme or path. Secrets arrive with the first request that has an endpoint, and that export uses those headers and service.name. A later agent-sdk secret set updates headers on the next export, including a rotated key. A URL, protocol, or signal-list change replaces the exporter. service.name is fixed on the first export that attaches, so redeploy to change it.

What you can record

import { metrics, trace } from "@opentelemetry/api";
import { logs } from "@opentelemetry/api-logs";
import { currentConversationId } from "@grok/agent-sdk/otel";

const tracer = trace.getTracer("incident-agent");
const lookups = metrics.getMeter("incident-agent").createCounter("incident.lookups");
const logger = logs.getLogger("incident-agent");

lookups.add(1, { "cursor.conversation.id": currentConversationId() });

Record after secrets load. A span or log that ends before exporters attach is still sent, up to 2048 of each, a metric recorded before the first collect is included, and HTTP, token, and tool spans carry cursor.conversation.id, while a span started during a turn or a tool call gets that attribute, a value you set is kept, the id is not propagated, and a metric gets it only when you set it.

Tag a turn

async execute({ key }, ctx) {
  ctx.telemetry.setTraceAttributes({ "work.item": key });
  ctx.telemetry.setMetricAttributes({ "work.item": key });
  ctx.telemetry.setLogAttributes({ "work.item": key });
  return await jira.getIssue(key);
}

ctx.telemetry writes the current turn's store. A span started in that turn, including a span Agent SDK starts, gets the trace keys. A log record Agent SDK emits gets the log keys. cursor.token.usage recorded for that turn gets the metric keys. The same store is still there for a later tool in the turn. With export on, a later turn of the session keeps it. With export off, the next turn does not.

import { telemetry } from "@grok/agent-sdk/telemetry";

telemetry.setTraceAttributes({ "work.item": ticket });

The global methods read the active context. Outside a turn they do nothing. Tags stay in this process. They are not written on the baggage header.

sum by (work_item) (cursor_token_usage_total)

Grafana renames . to _, and each distinct metric value is its own series, including a new id on every turn. Keys match [A-Za-z][A-Za-z0-9._-]* and are at most 128 characters, values are 1 to 256 characters, a session holds 32 trace keys, 32 metric keys, and 32 log keys, a later call replaces a key, and a metric value must not be a secret, an email, or a raw user id.

Outbound calls send traceparent only. With export off, the tags last for that turn, a span in the turn sees them, the next turn does not, and nothing is exported.

ctx.host.otel is deprecated, still runs this release, and logs one warning per process. setAttributes writes one map onto spans and token usage, and the calls above replace it.

Trace and log keys are accepted as written. A log record keeps a key Agent SDK already set. setMetricAttributes rejects cursor.token.type, agent_sdk.channel, cursor.tool.name, cursor.tool.kind, cursor.tool.status, agent_sdk.turn_status, agent_sdk.subagent_name, agent_sdk.sdk_agent_id, and agent_sdk.sdk_agent_url. The call throws and leaves the runtime value in place.

Prompt text and tool payloads stay off log records unless OTEL_LOGS_EXPORTER=otlp and OTEL_LOG_USER_PROMPTS or OTEL_LOG_TOOL_CONTENT is set.

Limits

Custom span processors, samplers, views, your own SDK, library auto-instrumentation, and a register() hook are not supported, and OTEL_TRACES_SAMPLER and OTEL_METRIC_EXPORT_INTERVAL are read when the provider is built, before hosted secrets load, so set them and restart, because a later value only logs a warning. If a module replaces a provider Agent SDK registered, the global API uses that provider and Agent SDK shuts down a provider it created only when it is not the registered one, and if a context manager is already registered, Agent SDK logs a warning and leaves it, so load Agent SDK first or the turn store does not stay on the active context.

Migrating from serve({ otel })

serve({ otel }) is removed, so delete that option, bot/otel.ts, agent/otel.ts, and bot/instrumentation.ts, and do not call defineOtel. Each signal has one URL, and a second exporter for the same signal has no variable.

Old option	Set this instead
serviceName	OTEL_SERVICE_NAME
serviceVersion	service.version in OTEL_RESOURCE_ATTRIBUTES
resourceAttributes	OTEL_RESOURCE_ATTRIBUTES, comma-separated, URL-encoded values
exporters[].url	OTEL_EXPORTER_OTLP_ENDPOINT (appends /v1/traces, /v1/metrics, /v1/logs)
exporters[].url with signals	OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, OTEL_EXPORTER_OTLP_METRICS_ENDPOINT, or OTEL_EXPORTER_OTLP_LOGS_ENDPOINT, used as written
exporters[].headers	OTEL_EXPORTER_OTLP_HEADERS, comma-separated, URL-encoded values
exporters[].protocol	OTEL_EXPORTER_OTLP_PROTOCOL, or OTEL_EXPORTER_OTLP_TRACES_PROTOCOL, OTEL_EXPORTER_OTLP_METRICS_PROTOCOL, OTEL_EXPORTER_OTLP_LOGS_PROTOCOL
exporters[].signals	The per-signal URL. OTEL_TRACES_EXPORTER=none, OTEL_METRICS_EXPORTER=none, or OTEL_LOGS_EXPORTER=none stops that signal.
traces: false	OTEL_TRACES_EXPORTER=none
traces: true	Leave OTEL_TRACES_EXPORTER unset. Traces start when an endpoint is set.
traces.sampleRatio	OTEL_TRACES_SAMPLER=parentbased_traceidratio and OTEL_TRACES_SAMPLER_ARG, before process start
metrics: false	OTEL_METRICS_EXPORTER=none
metrics: true	Leave OTEL_METRICS_EXPORTER unset. Metrics start when a metrics URL is set.
metrics.exportIntervalMillis	OTEL_METRIC_EXPORT_INTERVAL, milliseconds, before process start
logs: false	Unset OTEL_LOGS_EXPORTER, or set none
logs: true	OTEL_LOGS_EXPORTER=otlp
logs.userPrompts	OTEL_LOGS_EXPORTER=otlp and OTEL_LOG_USER_PROMPTS=1
logs.toolContent	OTEL_LOGS_EXPORTER=otlp and OTEL_LOG_TOOL_CONTENT=1
otel: false	No endpoint, or OTEL_SDK_DISABLED=true

Put those names in hosting.secretNames and the collector host in hosting.egressDomains.

agent-sdk secret set my-agent OTEL_SERVICE_NAME OTEL_EXPORTER_OTLP_ENDPOINT OTEL_EXPORTER_OTLP_HEADERS