Your APM stack fails AI workloads because Datadog, CloudWatch, and Dynatrace were built to observe deterministic request-response services, and agent chains are neither deterministic nor single-step. A user request that fans out into six LLM calls, three vector queries, two tool invocations, and a model fallback shows up in your trace viewer as one opaque span with a duration and an HTTP 200. The fix is not a new vendor SDK. It is OpenTelemetry GenAI semantic conventions layered onto a four-tier AWS reference architecture that correlates traces, token cost, vector query performance, and semantic quality without replacing what you already run.
This article walks through the specific blind spots, the numbers behind them, the OTel contract that lets platform and ML teams share one truth, and a phased migration path that keeps your existing observability investment intact.
The 2am Trace That Explains Nothing
It is 2am. Your on-call phone buzzes because a customer support agent endpoint is taking 47 seconds per response instead of the usual 4. You open Datadog, find the trace, and stare at a single 47-second span labeled POST /agent/invoke with a green 200 status. That is the entire story your APM stack can tell you.
Was it the LLM inference? A slow vector query against OpenSearch? A tool call to a downstream CRM API that timed out and retried three times? A model fallback from Claude to a cheaper backup that changed the whole reasoning path? The trace does not know. It measured wall-clock time on one HTTP handler and called it a day.
Compare that to a normal microservice trace. When your checkout service is slow, you see predictable spans: auth-service at 12ms, inventory-check at 40ms, payment-gateway at 800ms. The fan-out is fixed. Each span has a known parent, a known operation, and a duration you can attribute. You find the payment gateway is slow, you page that team, you go back to sleep.
Agent chains break every assumption baked into that model. The number of steps varies per request because the agent decides how many tools to call. Retries happen inside the reasoning loop, not just at the network layer. A single logical request can hit three different foundation models. The output is non-deterministic, so two identical inputs produce different span trees. Request-response instrumentation collapses all of that into a black box, and you are left reading tea leaves at 2am.
Four Blind Spots Your APM Stack Can't See
The gap is not one missing feature. It is four categories of signal that traditional APM was never designed to capture.
- Latency attribution across steps. When an agent request is slow, you need to know which step caused it: inference, retrieval, tool execution, or retry loops. CPU and memory metrics on the container tell you nothing about which of the six LLM calls stalled.
- Cost allocation per request, tenant, and agent. Traditional cost tools bill you on instance-hours and memory. AI cost is dominated by token spend, which varies wildly per request. Your APM has zero visibility into how many input and output tokens a single customer consumed.
- Failure diagnosis on non-deterministic output. HTTP 200 does not mean the agent gave a correct answer. It might have hallucinated a policy, cited a wrong document, or produced valid JSON that is semantically garbage. Status codes lie for AI workloads. You need quality signals, not just error codes.
- Retry and fallback amplification. A single user action can silently trigger three model retries and a fallback to a different provider. Your APM sees one request. Your bill sees four inference calls.
Here is what each tooling tier actually captures versus what agent workloads require.
| Signal | CloudWatch | Datadog APM | AI Workload Needs |
|---|---|---|---|
| Wall-clock latency | Yes | Yes | Per-step attribution across LLM, vector, tool calls |
| Token consumption | No | No | Input/output tokens per span, joined to pricing |
| Semantic correctness | No | No | LLM-as-judge quality score attached to trace |
| Retry amplification | Partial | Partial | Retry and fallback count as first-class span events |
| Vector query health | No | Limited | Recall, index freshness, query latency per call |
| Per-tenant cost | Instance-level | Instance-level | Token cost by tenant and agent type |
The pattern is consistent. Existing tools measure infrastructure. AI workloads need to measure reasoning, cost, and quality, three dimensions that live above the infrastructure layer.
The Numbers Behind the Gap
The operational cost of these blind spots is measurable, and it is worse than most platform teams assume.
The MTTR number is the one that keeps platform leads up at night. When a failure is non-deterministic, the first thing an engineer does is try to reproduce it, and with agents that reproduction often fails because the model took a different path the second time. You cannot debug what you cannot reproduce, so teams burn hours chasing ghosts.
The 17x cost variance is the finding that breaks budgets. Two users ask functionally the same question. One triggers a clean single-pass answer. The other triggers a retry loop, a fallback to a premium model, and a longer completion. Same feature, 17 times the cost, and no dashboard shows you the difference.
OpenTelemetry GenAI: The Contract Between Platform and ML Teams
The reason AI observability fails organizationally is that platform engineering owns the APM stack and ML teams own the models, and neither team's tooling speaks the other's language. OpenTelemetry GenAI semantic conventions are the shared contract that fixes this. They define standard span attributes like gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, and gen_ai.usage.output_tokens that any tool can read.
This matters because it is vendor-neutral. Both teams instrument against the same open specification, and the data flows into whatever backend you already run. Here is what manual instrumentation of a single agent step looks like in Python:
from opentelemetry import trace
tracer = trace.get_tracer("agent.runtime")
def call_model(prompt, model="claude-3-5-sonnet"):
with tracer.start_as_current_span("chat.completions") as span:
span.set_attribute("gen_ai.system", "anthropic")
span.set_attribute("gen_ai.request.model", model)
span.set_attribute("gen_ai.operation.name", "chat")
response = client.messages.create(model=model, messages=prompt)
span.set_attribute("gen_ai.usage.input_tokens",
response.usage.input_tokens)
span.set_attribute("gen_ai.usage.output_tokens",
response.usage.output_tokens)
span.set_attribute("gen_ai.response.finish_reason",
response.stop_reason)
return responseThat span now carries model identity, token counts, and finish reason in a format CloudWatch, Datadog, Grafana, and any OTel-compatible backend can ingest. You are not locked to one vendor's proprietary GenAI SDK, and you are not maintaining three parallel instrumentation layers. This portability is the same principle that makes our agentic AI systems observable regardless of which foundation model provider a client standardizes on.
A Four-Layer Reference Architecture on AWS
You do not need a new platform. You need four layers that sit on top of your existing AWS footprint and feed the signals your APM cannot generate.
Layer 1, instrumentation. Run the AWS Distro for OpenTelemetry (ADOT) as a sidecar on ECS and EKS, or as a Lambda layer for serverless agents. Your application code emits GenAI semantic convention spans. This is the only layer that touches application code.
Layer 2, collection and routing. An OpenTelemetry Collector receives every span and fans it out. Traces go to X-Ray or Datadog. Token-bearing spans get routed to a cost pipeline. Metrics go to CloudWatch. One collector config, multiple destinations, no duplicate instrumentation.
Layer 3, storage and correlation. This is where trace-to-cost joins happen. Spans with token counts get enriched against a model pricing table, session IDs roll up multi-turn conversations, and vector query metrics land alongside inference spans so you can attribute the 38% of latency that retrieval now consumes.
Layer 4, evaluation and quality signals. An asynchronous LLM-as-judge process scores a sample of completions for correctness, relevance, and safety, then writes those scores back onto the original traces. Now a trace tells you not just how long and how much, but whether the answer was actually good.
| Layer | Purpose | AWS Service | Open Source Component |
|---|---|---|---|
| 1. Instrumentation | Emit GenAI spans | ADOT on ECS/EKS/Lambda | OpenTelemetry SDK |
| 2. Collection & routing | Fan-out to backends | Kinesis, EventBridge | OTel Collector |
| 3. Storage & correlation | Trace-to-cost joins | X-Ray, S3, Athena | ClickHouse or Grafana Tempo |
| 4. Evaluation | Quality scoring | Bedrock, Lambda, Step Functions | Ragas, custom judge prompts |
The design principle: instrument once with an open standard, route everywhere, correlate cost and quality into the same trace context.
Instrumenting the Three Hardest Signals
Three signals separate real AI observability from dashboards that look impressive and tell you nothing.
Token cost attribution
Tag every inference span with input and output token counts, then join those counts against a pricing table in your correlation layer. A span with 4,000 input tokens and 800 output tokens on Claude 3.5 Sonnet becomes a dollar figure. Roll those figures up by trace ID for per-request cost, by tenant tag for per-customer cost, and by agent name for per-feature cost.
Vector DB observability
Retrieval is now a first-class latency contributor, so instrument it like one. For OpenSearch or pgvector, capture query latency, the number of results returned, recall against a known ground-truth set, and index freshness (how stale the embedded documents are). A slow agent is often a slow retrieval step, and without this you will blame the model for a problem in your index.
Session-level tracing
Multi-turn agent conversations must roll into a single parent trace with tool calls as children. A support conversation spanning eight turns should be one trace you can read top to bottom, not eight disconnected requests. Propagate a session ID through the context so every span links back to the conversation.
Here is a cost-per-successful-outcome metric derived from spans plus evaluation scores:
def cost_per_successful_outcome(traces):
total_cost = 0.0
successful = 0
for t in traces:
cost = sum(s.token_cost for s in t.spans if s.token_cost)
total_cost += cost
# quality_score written by Layer 4 async judge
if t.quality_score and t.quality_score >= 0.8:
successful += 1
if successful == 0:
return float("inf")
return round(total_cost / successful, 4)This single number, dollars spent per genuinely good answer, is the metric that should replace p99 latency on your AI operations dashboard.
The Migration Path That Doesn't Require Ripping Out Datadog
You keep Datadog. You keep CloudWatch. You add the four layers incrementally and let them coexist. This is a phased rollout, not a rebuild.
| Phase | Focus | Duration | Exit Criteria |
|---|---|---|---|
| 1. Instrument | Add GenAI spans to top 3 agent endpoints | Week 1-2 | Token counts visible per span |
| 2. Correlate | Join spans to pricing, roll up sessions | Week 3-4 | Per-request cost queryable |
| 3. Quality signals | Async LLM-as-judge scoring | Week 5-7 | Traces carry a quality score |
| 4. AI-native SLOs | Define and alert on new metrics | Week 8+ | Cost-per-outcome SLO live |
The new SLOs replace latency-only targets. Track cost per successful outcome by agent type, semantic error rate (percentage of responses scoring below your quality threshold), and retry amplification factor (inference calls per user request). These are the numbers that predict both your bill and your customer satisfaction.
Organizationally, this closes the gap between platform engineering and ML ops because both teams read the same enriched traces. The platform team owns the collector and storage layers. The ML team owns the instrumentation attributes and the judge prompts. The OTel standard is the seam where those responsibilities meet. This same telemetry feeds AI FinOps reporting and agentic control-plane observability, which pairs naturally with a broader cloud modernization strategy rather than requiring a separate initiative.
What to Do in the Next 30 Minutes
Pick one agent endpoint, the one that pages you most often, and add the six gen_ai.* span attributes from the code example above using your existing OpenTelemetry Collector. You do not need to touch the other endpoints yet. You need one trace that shows token counts and per-step timing so you can prove the value before you scale it.
This week, start tracking one metric: cost per successful outcome, broken down by agent type. Even a rough version using a sampled quality score will surface which agents are quietly burning budget on retries.
FAQ
Do I have to replace Datadog or CloudWatch? No. OpenTelemetry GenAI spans flow into both. The four-layer architecture adds cost and quality correlation alongside your existing APM, not instead of it.
How do I control storage costs from all this trace data? Never log full prompts and completions by default. Store token counts, model IDs, and content hashes. Sample full text at 1% and capture it fully only on flagged failures.
Which foundation model providers does this work with? All of them. GenAI semantic conventions define provider-neutral attributes like gen_ai.system, so Bedrock, OpenAI, Anthropic, and self-hosted models all emit the same span shape.
Do platform or ML teams own this? Both. Platform owns collection and storage. ML owns instrumentation attributes and evaluation prompts. The OTel standard is the shared contract.
Remember that opaque 47-second span from 2am. With GenAI instrumentation, that same incident shows a vector query timing out, triggering two retries, then a fallback to a premium model that tripled the token cost. The trace tells the whole story now, and you fix the retrieval index instead of guessing until sunrise.