LangSmith: Close the Loop Between Shipped and Working

Watch (4:31)

Overview

LangSmith's agent engine closes the gap between shipped and working: it clusters production failures into named issues, traces each back to the commit that introduced it, and drafts a fix that waits for your approval.

Full transcript (from the video)

Your latency dashboard looks clean. Error rate is zero. Somewhere in production right now, your agent is answering the wrong question and nobody knows yet. Now, finding it means five serial steps.

Trace, spot the gap, edit, build a data set, run an experiment. That cycle takes days and resets with every new incident. The interesting part is the LangSmith engine automates that entire cycle. It clusters failures into named issues, diagnoses root causes against live code, and drafts fixes, handing off only at the approval step.

Now it watches five signal types: explicit errors, evaluator failures, trace anomalies, negative user feedback, and questions the agent was never designed to answer. The interesting part is what it does with those signals. Instead of flooding you with individual failing traces, it collapses them into clusters. Each one gets a name, a severity score, and an estimate of how many sessions were actually affected.

So rather than scrolling through dozens of broken spans, you might see three named issues. Tool call timeout at high severity. Touching 32% of sessions, missing context window at medium, affecting eight. Triage becomes a decision instead of an excavation.

You pick the issue with the biggest footprint, fix it, and that percentage number tells you exactly how much ground you recover when you do. Now, every regression has a first commit, and the engine finds it. Cross referencing each issue cluster against your deployment history, it surfaces the exact release where the pattern first appeared. That is what turns a vague quality complaint into a bounded solvable problem.

one commit, a diff you can actually read, a decision you can reverse or double down on. Instead of interviewing engineers or combing through change logs trying to reconstruct a timeline, you are looking at a single pinpointed change. The regression has a birthday. The fix has a clear concrete target.

Now consider what that birthday looks like in practice. A customer support bot. A subscription cancellation flow gets flagged at high severity, not for slow responses, not for errors, for a pattern. 12% of support sessions that week collapsing into the same cluster of confused looping exchanges.

Dig in and the thread leads to a specific deploy 4 days earlier. A latency monitor would have shown nothing. Response times were fine. A token count alert would have shown nothing either.

The only signal was semantic. Users asking to cancel kept landing in a conversation that failed them in a way no dashboard metric was designed to detect. That is the gap a behavioral layer fills. Now here is where the diagnosis gets precise.

Once a failure cluster is identified, the platform reaches into the repository itself reading the actual prompt templates and tool definitions that ran during those sessions. Not a summary, not a reconstructed guess. the live source. From that, it produces a structured root cause explanation that points to a specific template line or a missing branch in a tool's logic.

That grounding matters because speculative diagnosis send engineers chasing the wrong variable. When the explanation names the exact instruction that caused the model to misroute a cancellation intent, the fix becomes obvious. Rewrite that instruction, rerun, and watch the cluster shrink. Now, one diagnosis yields three outputs.

A pull request with a fix, new evaluation examples, and an online evaluator that flags the same regression if it reappears. The interesting part is that nothing about your current setup needs to change. If your team already sends traces there, the engine simply layers on top. Now, tools like Weights & Biases, Arize, and HoneyHive will surface the failure, but that's where they stop.

The engine diagnoses the root cause and drafts the fix. One catch. Anthropic, OpenAI, and Google are each pulling observability into their own platforms. Enterprises running agents across all three may still need a neutral layer, a gap this service doesn't fully fill.

Traces flowing. Connect the repo. Flip engine on and let the first clustering pass