LangSmith

What a platform adds to logs you already have

You can log every model call yourself. Request, response, latency, token counts, a request ID. That is a few lines of code and a table.

What you get is a list of rows. What you wanted, at the moment something breaks, is the shape of one request: the prompt as assembled after retrieval, the tool the model chose, what came back, the second model call that followed, and where the eleven seconds went. Reconstructing that from rows means joining on a correlation ID and reading JSON in a terminal. Tracing and logging covers why a trace is the right structure for this and what instrumenting costs.

LangSmith offers "full visibility into your LLM application: from individual traces to production-wide performance metrics." That much is true of every platform in this category. The part specific to this one is the sentence after it.

Traces, and what they cost to produce

"Traces are the record of what your agents did in production. Use them to debug failures, monitor quality, and build the datasets you evaluate against."

Note the third clause. The other tools on this site treat traces as something to read; this product treats them as raw material for the thing you test against next. That framing is the product's own, and it is the reason to write about it separately from the others.

Producing traces costs what it always costs: an SDK in your request path, calls that can fail or add latency, and your prompts and outputs leaving your infrastructure for a hosted service. That last one is a genuine constraint for regulated workloads, and it is the general consequence of sending traces anywhere hosted rather than a claim about this vendor's retention policy, I did not verify that, and neither should you take it on faith from an article.

Turning a bad trace into a saved case

Here is the loop the product is organized around, and it is worth describing as a practice even if you use a different tool.

Something goes wrong in production. You find the trace. Normally that is where investigation ends: you understand the failure, you fix the prompt, you move on, and the knowledge lives in a commit message.

The alternative is one more step. Before fixing anything, promote the trace to a saved case: its inputs, and what the right answer would have been. Now the failure is an artifact rather than an anecdote. It runs against every future change, and the fix is verified rather than assumed.

Do that for six months and you have something you could not have designed up front: a test set made entirely of things that broke, weighted by what your real users do. Nobody sitting down to write test cases produces that. It only comes from failures you kept.

Running the saved cases against the next change

The saved cases are only worth their storage if something runs them.

That is regression testing, which is this loop from the vendor-neutral side: how to compare runs, how to read a diff of failures rather than an aggregate score, and how to gate a change when the same input does not give the same output twice. The mechanics belong there.

What a platform adds is that the three pieces sit in one system. The trace that exposed the problem, the case it became, and the run that proves it is fixed are linked, so six months later you can answer "why is this case here" by opening the trace it came from. Assembled out of separate tools, that link is a convention someone has to maintain, and it is usually the first thing to rot.

Using LangSmith without LangChain

The name suggests a prerequisite. It is not one.

The documentation states that LangSmith "works with many frameworks and providers", listing OpenAI, Anthropic, CrewAI, the Vercel AI SDK and Pydantic AI. So an application calling a provider SDK directly, or one running the loop from manual implementation, can be traced without adopting the framework.

Worth checking rather than assuming, which is why this section exists. A tool built by a framework's authors tends to be best documented, best tested, and least surprising on that framework, and "supported" and "first-class" are not the same word. If you are not on LangChain, read the integration page for your stack before deciding, and expect to write a little more instrumentation than the tutorials show.

Choosing between LangSmith and the alternatives

The observability section on this site is crowded, which is itself informative: these products overlap heavily and differentiate at the edges.

Arize comes from machine learning monitoring and brings that analysis with it. PostHog joins model calls to product analytics, so you can ask what a user did after a bad answer. Langfuse is the open-source option. This one's edge is the trace-to-dataset-to-evaluation path being one path.

Pick on which join you need. If your recurring question is "what happened in this request", any of them works and you should choose on price and deployment. If it is "does this change make things worse than last week", the loop described here is the one to look for, under whatever name your chosen tool gives it.

Further reading

• LangChain, LangSmith documentation: the framework support list and the trace-to-dataset framing quoted above.
• Tracing and logging: what a trace is, and what instrumenting costs, without a vendor attached.
• Regression testing: the loop above, described as a practice rather than a product.
• LLM evaluations: the tiers a saved case set fits into.