Helicone: observability that sits in the request path

Helicone is an Apache-2.0 licensed LLM gateway and observability platform, and the review position is that the architecture is the whole story: Helicone works by sitting between the application and the model provider, which buys an enormous amount of functionality for one changed line of code and costs a network hop, a data-residency question and a vendor on the critical path. For teams that want a dashboard on Monday, that trade is good. For teams with a latency budget measured in milliseconds or a rule that prompts cannot leave the network, it is the wrong shape.

It competes in two directions at once. Against pure tracing backends such as Langfuse, Arize Phoenix and LangSmith, it wins on time-to-first-dashboard and on the gateway features bolted to the proxy, and loses on OpenTelemetry-native instrumentation and on evaluation tooling. Against API routers such as LiteLLM or OpenRouter, it is observability first with routing attached.

What it is

Two products share one codebase. The gateway is an OpenAI-compatible endpoint in front of 100 plus providers, reached by pointing the base URL at ai-gateway.helicone.ai. The observability platform is what records the requests that pass through, stores them in ClickHouse, and answers questions about cost, latency, sessions and users through a dashboard, a SQL-like query language called HQL, alerts, reports and webhooks.

• Licence Apache 2.0, about 6,200 GitHub stars, and a note that Helicone joined Mintlify in 2025.
• One-line integration: change the base URL, or send an async log through the OpenLLMetry instrumentation.
• Cost tracking computed from token usage and a public price database covering more than 300 models and providers.
• Gateway features on the request path: edge cache, custom rate limits by request count, cost or property, and automatic fallbacks.
• Operational surface: sessions, per-user metrics, custom properties, scores, datasets, webhooks and an MCP server over the data.
• Self-hosting that runs five services: a web frontend, a Cloudflare Worker, the Jawn collector, Supabase and ClickHouse, plus MinIO for bodies.

How it works

The request path is deliberately thin. Unless a header enables a feature, the worker forwards the request and returns the response untouched; after the response is complete, the proxy ships logs to Kafka for a separate service to consume. The availability documentation states the design intent directly: all business logic falls back to plain proxying on any error, so a bug in observability degrades to a working relay.

One consequence is worth stating plainly: in proxy mode every prompt and every completion transits a third-party edge network by default. The self-hosted deployment removes that, but it also removes the part of the product that makes Helicone easy, because a self-hosted Jawn no longer proxies at all — the docs record that the gateway routes were removed and the AI Gateway must be deployed separately.

Getting started

The whole integration is a base URL. The example below also switches on the edge cache, which is the fastest way to see the platform do something visible: the second identical request is served from Cloudflare's KV store rather than from the provider.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://ai-gateway.helicone.ai",
    api_key=os.environ["HELICONE_API_KEY"],
    default_headers={
        "Helicone-Cache-Enabled": "true",
        "Cache-Control": "max-age=3600",
        "Helicone-Cache-Seed": "user-123",
        "Helicone-Property-Env": "production",
    },
)

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Summarise the refund policy."}],
)

print(response.choices[0].message.content)
# The next identical call returns Helicone-Cache: HIT and skips the provider.

The cache key is a hash of the cache seed, the request URL, the whole request body and the relevant headers, which makes it exact-match rather than semantic: change one word or one temperature and it misses. Durations run from an hour to a 365-day maximum, buckets up to 20 stored responses are available for non-deterministic prompts, and Helicone-Cache-Ignore-Keys excludes JSON fields such as a request id or a provider prompt_cache_key from the key.

Proxy or async, pick one deliberately

This is the decision that shapes everything else. Helicone documents both modes and is unusually honest about the gap between them, which makes the choice easy to reason about even though the answer is rarely comfortable.

The reasonable conclusion is that these are two products sharing a name. The proxy is a gateway with analytics attached; the async integration is a logging library with a web UI. Teams that take the proxy get caching, rate limiting and fallbacks that they would otherwise build, and pay for it with an extra hop and a data-residency decision. Teams that take the async path keep their latency budget and their network boundary, and build the gateway themselves or go without it.

Latency and the cost of a hop

The published benchmark is small but unusually well specified: 500 requests with unique prompts interleaved between OpenAI and Helicone inside the same one-second window, alternating which endpoint was called first, with the prompt context maximised, on text-ada-001, logging round-trip latency for both sets.

Read the maximum row carefully. A 200 millisecond difference on a three-and-a-half second response is 6 per cent, and the same absolute hop applied to a 300 millisecond model call or to a 40 millisecond tool call would dominate it. The benchmark is also a vendor test on a retired model, so it demonstrates that the overhead is bounded, not what it will be for a given workload.

Pricing and what the meter is

The free Hobby plan covers 10,000 requests a month, one seat, one organisation, one gigabyte and seven days of retention. Pro is $79 a month for unlimited seats, one organisation, alerts, reports, HQL, one month of retention and 1,000 ingested logs per minute. Team is $799 for five organisations, SOC 2 and HIPAA options, three months of retention and 15,000 logs per minute. Enterprise adds unlimited organisations, SAML SSO and on-prem deployment.

The number to watch is the one nobody prices prominently: ingestion. A Hobby workspace ingests 10 logs a minute, Pro 1,000, Team 15,000. A production application can emit several logs per user request — a model call, a retrieval step, a tool call — so a moderately busy product exhausts the Pro ceiling before it exhausts the request allowance, and the plan that fixes it costs 799 dollars a month. Storage is metered separately above the first gigabyte.

Self-hosting the whole stack

The Apache-2.0 repository contains the entire platform, and the README is direct about the operational shape: five services, and a manual deployment that it marks as not recommended in favour of the Docker compose file or an Enterprise Helm chart.

• Web: the dashboard frontend, a Next.js application.
• Worker: the proxy, deployed as Cloudflare Workers in the hosted version.
• Jawn: the log collector, an Express service with Tsoa-generated routes.
• Supabase: the application database and authentication.
• ClickHouse: the analytics store that answers cost and latency questions.
• MinIO: object storage for request and response bodies.

Where it falls short

The honest weaknesses are structural, not cosmetic. Helicone's data model is built around requests that pass through its gateway, so a team that already traces with OpenTelemetry spans has to choose between two instrumentation stacks. The evaluation story is thin next to Langfuse or Braintrust: prompts, playground and scores exist, but there is no first-class experiments workflow. And the ownership question is live — Helicone joined Mintlify in 2025, which removes the open-core lock-in risk but adds the usual vendor dependency.

Langfuse is MIT-licensed, was acquired by ClickHouse in January 2026 with the licence and self-hosting explicitly unchanged, and stores traces in ClickHouse as well; if the deciding factor is a permissive licence with a real evaluation stack, it is the stronger tool. Arize Phoenix is built on OpenInference conventions and runs anywhere, including air-gapped, but its Elastic License 2.0 is not a permissive one. Helicone's argument is not capability, it is time: the distance between a repository and a cost dashboard measured in requests per user is one line of code.

Verdict

Helicone is the right tool when nobody can answer what the LLM spend was last Tuesday, and the wrong tool when latency, data residency or tracing standards decide the architecture. Its engineering is unremarkable in the best sense: the proxy is thin, logs go out after the response, and the product is Apache-2.0 from the gateway to the dashboard. Its weakness is equally plain: it wants to be your gateway, and everything that makes it comfortable is a consequence of that.

1. Use it when the team needs per-request cost, latency and error visibility within a week and has no tracing stack yet.
2. Use it when the edge cache, custom rate limits and automatic fallbacks are worth having on the request path.
3. Use the async OpenLLMetry path when you want Helicone's analytics without a proxy hop, and accept losing the gateway features.
4. Avoid it when prompts must not leave your network or when the proxy hop eats a hard latency budget.
5. Avoid it when the platform already traces with OpenTelemetry and a second instrumentation model would fragment the data.
6. Reconsider it above roughly 1,000 ingested logs a minute, or when the request allowance is large enough that the usage meter needs its own line in the budget.

Sources

1. Helicone quickstart
2. Helicone AI Gateway overview
3. Helicone: latency impact and benchmark
4. Helicone: proxy versus async integration
5. Helicone: LLM caching
6. Helicone: self-hosting with Docker
7. Helicone: how we calculate cost
8. Helicone pricing
9. Helicone repository on GitHub