Arize Phoenix review: LLM tracing and evals you host yourself

Arize Phoenix is an open-source server that records what an LLM application did — every prompt, retrieval step, tool call and token — and then scores it. It starts from a single command on a laptop or lands in your own Kubernetes cluster, and the licence puts no meter on any of it. The position of this review is blunt: Phoenix is the shortest path to keeping trace data inside your network, and what you pay for that is the job of running an observability stack yourself.

It sits between the application SDK and the dashboard, competing with Langfuse, Braintrust and LangSmith for that slot, and with Arize's own AX cloud for teams that would rather send an invoice than operate a database. What it usually replaces is worse: a pile of log dumps, a dashboard nobody reads and a spreadsheet of evaluation results.

What it is

Phoenix is a containerised application in three parts — a web interface, a trace collector and a SQL backend — released by Arize under the Elastic License 2.0. The Python package is arize-phoenix, at version 20.19.0 in early October 2026 and requiring Python 3.11 or newer, and the repository holds a little over 11,700 stars. Everything is in the free build: tracing, annotation, datasets, experiments, a prompt IDE and LLM-as-a-judge evaluation.

• Licence: Elastic License 2.0 (ELv2), free to self-host with no usage limits and no feature gates
• Ingestion: OTLP with OpenInference semantic conventions, plus auto-instrumentation packages for providers and frameworks
• Storage: SQLite by default, PostgreSQL 14 or newer for production, both behind the same SQL schema
• Deployment: terminal, Docker, Kubernetes, Helm, CloudFormation and one-click templates for Railway, Render, Cloud Run and Azure
• Scope: traces and sessions, annotations, datasets, experiments, prompt versioning, code scorers and LLM-as-a-judge
• Tenancy: one tenant per instance, with OAuth2, LDAP, local accounts and role-based access control in the same build
• Cloud counterpart: Arize AX, which is where support, an uptime commitment and multi-tenancy live

For a team whose customer or regulator will not accept traces leaving the country, that list is the shortlist on its own. For everyone else it is a trade: the whole product costs nothing, and the jobs a vendor would otherwise do — retention, backups, upgrades, and the answer to who is paged when the collector stops — become yours.

How it works

The instrumented process emits spans to an OpenTelemetry exporter, Phoenix accepts them over OTLP on port 6006, stores them in SQL and groups them into projects, sessions and traces. Each span carries OpenInference attributes for tokens, cost, model, retrieval payloads and tool arguments, which is what later lets a scorer read the run rather than only its final string.

The loop after that is the one Arize publishes: observe, annotate, hypothesise, experiment, measure. A production trace becomes a dataset row, a candidate prompt runs against that dataset as an experiment, and scorers — built-in, code-based or judge-based — attach scores to the records the interface already shows. The documentation also describes PXI, an agent interface for investigating issues and running experiments over captured traces, plus an agent-assisted setup command, px setup, which waits for a real trace before it reports success.

Getting started

Two commands. The server is one Python invocation, uvx arize-phoenix serve, which answers on http://localhost:6006 with an empty project. The client side is arize-phoenix-otel plus the OpenInference instrumentor for the SDK the application already uses:

# Terminal 1: uvx arize-phoenix serve  ->  UI on http://localhost:6006
# pip install "arize-phoenix-otel>=0.16.0" openinference-instrumentation-openai
from phoenix.otel import register

register(
    project_name="support-agent",
    auto_instrument=True,
    endpoint="http://localhost:6006/v1/traces",
)

from openai import OpenAI

client = OpenAI()
reply = client.responses.create(
    model="gpt-5-mini",
    input="Summarise this ticket in one sentence.",
)
print(reply.output_text)

What comes back is a project with real spans: input, output, model, token counts, latency and the nesting of tool calls under the agent turn that triggered them. The register() call reads PHOENIX_COLLECTOR_ENDPOINT when it is set, so the same code reaches a laptop, a shared staging server or an air-gapped deployment without an edit.

Self-hosting in practice

The documentation claims an air-gapped deployment: nothing in the open build talks to Arize, and traces, prompts and datasets stay inside your infrastructure. The footprint behind that claim is one container and one database, and the working directory or PHOENIX_SQL_DATABASE_URL is the only stateful part that deserves a backup schedule.

• Storage: SQLite in the working directory by default; setting PHOENIX_SQL_DATABASE_URL switches to PostgreSQL, minimum supported version 14
• Images: arizephoenix/phoenix on Docker Hub with latest, pinned version, nonroot and debug tags, plus a separate arizephoenix/phoenix-helm chart
• Authentication: OAuth2, LDAP and local accounts, with role-based access control and retention policies per project
• Scale-out: several instances behind a load balancer on one database, or one instance per team with its own database
• Isolation: a schema setting shares one database between teams without sharing rows

Two limits are worth knowing before the first production week. A single instance is one tenant, so team isolation means several deployments, and group-based multi-tenancy sits in the issue tracker as a 2026 item rather than a shipped feature. SQLite is also the development backend: the architecture page routes production traffic to PostgreSQL, which puts backups, migrations and connection pooling in your column.

What it costs

Phoenix itself has no price. The self-hosting page lists no licence fees, no usage limits and no feature gates, so the monthly cost is the machine, the database and whoever owns them. The commercial surface is Arize AX, a separate managed product whose free tier is enough to evaluate and whose Pro tier is the first real bill:

Both of the first two tiers are hosted by Arize, and the pricing table lists self-hosted deployment against AX Enterprise only. Against the field the free self-hosted path is the outlier: Langfuse caps its free cloud at 50,000 units a month, Braintrust at 1 GB and LangSmith at 5,000 traces, while an unlimited Phoenix costs an instance. The catch sits at the far end — dedicated support and an uptime commitment are Enterprise lines, so a team that needs someone else accountable for availability cannot buy that for $50.

Where it falls short

Phoenix asks you to be your own observability vendor, and it shows. Retention is whatever you configure, ingestion volume is whatever the disk takes, and nothing in the open build arrives with a support contract or an uptime commitment. The SQL backend is not an analytical engine either: the architecture page routes high-volume, sub-second OLAP work to adb, Arize's proprietary database, which exists only inside AX. The project also ships quickly, which is good for features and awkward for pinning an upgrade window, and the polished hosted extras — managed agents, issue detection, repository access — are AX rows rather than Phoenix rows.

Choose by constraint rather than by feature count. If data residency is the requirement, Phoenix or self-hosted Langfuse is the answer and Braintrust is not on the list until sales is involved. If the bill matters more than the data plane, Langfuse Core at $29 undercuts AX Pro and carries prompt management with it. If evaluation in CI is the actual job, a scorer library and a ready-made eval action are a shorter route than assembling the parts here.

You may not provide the software to third parties as a hosted or managed service, where the service provides users with access to any substantial set of the features or functionality of the software. — Elastic License 2.0, limitations

Verdict

Phoenix is the right default for a team that cannot export traces and has someone who enjoys running Postgres. It is the wrong default for a team that wants an evaluation platform this week with no infrastructure conversation, because storage, retention, access control and upgrades are real work even though the licence is free.

1. Choose it when trace data must stay in your own network, including air-gapped environments: the open build sends nothing to Arize.
2. Choose it when the cost model matters more than the feature checklist: no span caps, no seat fee and no ingestion meter.
3. Choose it when OpenTelemetry is already the house standard, since ingestion is OTLP and the exporter can be exchanged without touching the application.
4. Do not choose it when you need multi-tenant hosting with an uptime commitment at an entry price: AX Pro is hosted only, and contracts start at Enterprise.
5. Do not choose it when evaluation in CI is the primary job, or when nobody on the team wants to own a database.

Sources

1. Arize Phoenix: open-source AI observability and evaluation
2. Phoenix documentation: self-hosting
3. Phoenix documentation: licence under the Elastic License 2.0
4. Phoenix documentation: architecture, storage and scaling
5. Phoenix documentation: setup tracing
6. Phoenix documentation: OpenTelemetry SDK setup
7. Arize pricing: AX Free, AX Pro and AX Enterprise
8. Langfuse pricing: cloud plans and billable units
9. LangChain pricing: LangSmith plans