06 / AI Reliability / EvaluationR&D + product architecture

Agent TraceReliability Intelligence

My roleAI Reliability Architect

ContextProduction agent systems

System classAI Reliability / Evaluation

Portfolio evidenceArchitecture · decisions · production

01 / Brief

The problem

Production agent systems can be technically available while functionally wrong. Raw traces expose every span but do not reveal recurring shapes such as empty retrieval, wrong-source selection, malformed tool output, looping plans or silent fallback degradation.

02 / Constraints

Why it was hard

01Signals combine infrastructure, application, LLM and agent-semantic telemetry.

02Failures are non-deterministic and often require comparison across many traces.

03The output must help engineers decide what to fix or whether a release should proceed.

04Evaluation costs and latency must remain bounded enough for continuous operation.

03 / Architecture

System shape

The architecture is expressed as operational layers: experience and identity at the edge, bounded orchestration in the middle, governed data and tools underneath, and evidence across the entire path.

Agent Trace / logical architectureSanitized portfolio view

01IngestOTel + agent traces

02NormalizeEvents, tools, retrieval, costs

03ScanSpecialized evaluators

04ClusterRecurring failure shapes

05ActRCA, alert, release gate

04 / Decisions

The engineering judgment

Use multiple scanners, not one score

Grounding, retrieval, tool behavior, policy, latency and cost fail differently and need separate evidence.

Cluster traces before human review

Compression turns thousands of spans into a smaller set of recurring operational problems.

Connect evaluation to deployment policy

Quality, cost and latency regressions can block or canary a release instead of becoming passive analytics.

Preserve the original trace as evidence

Every cluster and automated RCA links back to source spans so engineers can verify the conclusion.

05 / Production

Hardening the system

01Semantic SLOs for groundedness, tool success and agent completion.

02Golden datasets and regression runs in CI/CD.

03Online sampling, drift detection and incident thresholds.

04Cost attribution by model, workflow and tenant.

05Runbooks for hallucination spikes, retrieval failures and provider degradation.

OpenTelemetryLangfuseOpik conceptsLLM evaluationTrace analysisFailure clusteringRCASLOsQuality gatesAI incident managementCost telemetryAI SRE

06 / Outcome

A design for moving from “we have traces” to we can detect, explain and prevent recurring agent failures.

Next case study / 01