Helicone

A gateway rather than an SDK

The other observability tools here instrument your code. Langfuse, LangSmith and Arize all sit inside your application and send data out.

Helicone can sit in front of it instead. Its AI Gateway is an OpenAI-compatible endpoint, so the integration is a base URL and a key:

baseURL: 'https://ai-gateway.helicone.ai'
apiKey: process.env.HELICONE_API_KEY

Requests appear in the platform within seconds, and the gateway fronts a hundred or so models behind one API. That position is what the rest of this page is about, because standing in the request path buys capabilities an SDK cannot have and costs you something an SDK does not.

The choice that decides your availability

Helicone offers both integrations, and picking between them is the real decision.

Async gives up prompt auto-formatting, bucket caching, retries and custom rate limiting. The documentation is direct that it "cannot offer the same suite of tools as we can with the proxy".

So the question is whether you want a dependency in your critical path in exchange for those features. A team that would not accept a new single point of failure in front of their model calls should take async and treat Helicone as a logger. A team that wants caching and rate limiting without building them should take the proxy and understand what they have added.

Neither answer is wrong. Choosing without noticing the trade is.

What the gateway position buys

Three things the SDK-based tools cannot do, because they are not in the path.

Caching. Repeated requests are served without reaching the provider. Worth real money for a workload with recurring prompts, and the gateway can also cache context across providers.

Custom rate limits. Limits by request count, by cost, or by a custom property. Capping spend per customer is a product requirement that is awkward to build and trivial to configure here.

Retries. Handled for you rather than written into every call site.

All three are proxy-only, which is the same trade stated a second way.

Segmenting the data

Custom properties are the mechanism worth setting up first. Tag each request with the feature, the customer, the plan, the experiment, and the cost and latency questions people ask become answerable without new instrumentation.

Sessions group a multi-turn conversation, which is the unit a support complaint arrives as, and a Request API lets you pull session data back out for analysis. User metrics roll the same data up per end user.

Prompt management and scoring sit on top, so a prompt is a versioned object and a trace can carry feedback. The same pieces Langfuse offers, reached from a different architectural starting point.

Running it yourself

Self-hosting is documented for both Docker and Kubernetes, which puts Helicone in the same category as Langfuse on the question that usually decides these tools: whether prompts and completions can leave your network.

With one difference worth thinking through. Self-hosting a proxy means you now operate the thing in your request path, so its uptime is your uptime. That is a heavier commitment than self-hosting a logger, and it is the argument for running async if you are going to host it yourself.

Whether to choose it

Reach for Helicone when you want caching or per-customer rate limiting and would rather configure than build, or when an OpenAI-compatible base URL is the only change you can get merged.

Look elsewhere when nothing may sit in your request path, or when you have already instrumented with OpenTelemetry, since LLM observability is right that settled attribute names are worth keeping.

Further reading

• Helicone, Proxy vs async: the trade-off, and what async gives up.
• Helicone, Quick start: the gateway URL and the two lines of setup.
• Langfuse: the self-hostable SDK platform.
• LLM observability: what any of these tools should be recording.