Third Eye: Observability for LLM Apps

An open source tracing and prompt testing layer — real traces, saved cases, and a diff you can put in a pull request.

Every team shipping on top of an LLM hits the same wall. The demo works. Then production happens, a user says the answer was wrong, and you've got nothing to go on — not the exact prompt, not the model, not the parameters, not the response, not the latency, not the cost. Worse, you can't tell whether last week's prompt edit helped or made things worse. Third Eye is the tool I built to fix that. It's an open source observability and prompt testing tool for LLM apps.

LLM failures are quiet

Normal systems fail loudly: a stack trace, a 500, a red line on a graph. LLM systems fail politely. They hand back fluent, confident, wrong answers, and because no two inputs are identical you can't just replay the request that broke. The only durable fix is to treat every call as an event worth recording — the full prompt, the rendered messages, the model and parameters, the response, token counts, latency, cost, and whatever business metadata you attach.

Third Eye is two pieces. A small TypeScript SDK wraps your calls, and a Hono server ingests and serves the traces. The constraint that shaped the SDK is that tracing must never slow down or break the request it's watching. Recording is fire-and-forget. If the collector is down, your app still answers the user.

import { thirdEye } from '@third-eye/sdk';

const trace = thirdEye.trace('summarize-ticket');

const res = await client.messages.create({ /* ... */ });

trace.record({
  model: 'claude-opus-5',
  prompt,
  response: res,
  metadata: { ticketId, userTier },
});

Traces are half of it

Capturing what happened is the easy half. The half that changes how you work is testing a change before it ships. You save real traces as cases, run a prompt revision across the whole set, and diff the results. Did quality move? Did cost? Did latency? Did any single case regress?

$ third-eye eval prompts/summarize@v7 --against cases/support-tickets
  quality      0.82 -> 0.86   (+0.04)
  cost / req   $0.0031 -> $0.0029
  regressions  1 of 214

A prompt change should show up in review with a before and an after, the same as any other diff.

Why open source

Two practical reasons. Prompts and completions are often a team's most sensitive data, and plenty of orgs simply can't ship them to an outside vendor — self-hosting isn't a tier for them, it's the only version they can use. Second, this category is turning into infrastructure, and infrastructure earns trust by being inspectable and forkable. I'd rather build the thing a team can own than one more dashboard they rent.

It's the same instinct behind everything else I build. The trading agent keeps a ledger. The desktop app appends every action to a log. Third Eye points that at the model layer. I don't trust software I can't observe, and I trust it least when it's acting on its own.