Observatory: what we track and why

We've been running Observatory on an agent fleet of 20 active agents, varied models, and a mix of task types from document handling to customer service. After eight weeks we have a good sense of what's actually useful to track, what's noise, and where thresholds should sit so the flow holds.

The first thing that becomes clear is that observability isn't the same as monitoring. Monitoring is something turning red when it goes wrong. Observability is being able to answer "why did this happen" without guessing. The difference is real, and Observatory is built for the second.

What we track continuously

Latency per agent call. The simplest one, and it surfaces a surprising amount. When an agent that normally responds in 2 seconds starts taking 8, it's usually not the model. It's either the context window growing, or a tool call that's gotten slower. Latency as a time series shows this clearly.

Token consumption per task. This is the cost signal, and it varies more than people assume. The same task type can use 1,500 tokens one day and 3,500 the next, because the agent chose to pull more context. If we don't see this, we burn money without knowing.

Tool-call frequency and failure rate. How often does the agent reach out to tools? How often does the tool fail? If a particular tool endpoint has a 12 percent failure rate, we need to know before the agents have accumulated 50 failed sessions.

Eval scores over time. We run automated evaluations on output for critical task types. Score over time surfaces drift no human review would catch manually. If a KYC classification drops from 94 percent to 88 percent over two weeks, the alarm rings before it becomes a compliance issue.

Human override rate. Every time an operator overrides an agent recommendation, that's signal. How often does it happen? Which task type has the highest override rate? Which model configuration is involved? It's the most direct feedback on agent quality, and it's gold for improvement.

What we've learned to ignore

Token use as an absolute number without context. An agent using a lot of tokens isn't bad by definition. It might be solving a complex task. The relevant signal is tokens-per-outcome, not tokens in the absolute.

Latency spikes that last under five minutes. Model providers have normal jitter. Alerting on anything past the 95th percentile drowns us in noise without cause. We alert on spikes that last over 15 minutes or on a clear time-series shift.

General "hallucination rate" estimates from the model provider. These are too coarse to be actionable. We measure on our own evaluation for the task types that actually matter.

How we set thresholds

The most common mistake is setting thresholds in absolute values. "Alert if latency over 5 seconds." It doesn't work because 5 seconds can be normal for a complex task and too much for a simple one. We set relative thresholds against baseline. Alert if median latency for this agent has shifted 50 percent over 24 hours.

On evaluations we set two thresholds. The first is a gradual drift alarm catching decline over weeks. The second is an acute alarm if a run drops more than 10 percent over two days. The first gives us time to investigate. The second pauses the agent until we've checked.

On override rate we set sector-specific thresholds. KYC tasks above 15 percent override is unacceptable. Customer service at 30 percent is normal. These get defined per agent and per task type, never as global values.

What still needs human judgement

When an alarm fires, it's still a human's job to decide whether the problem is the agent, the data, the infrastructure, or the business. Observatory points at where in the flow the signal originates, but the diagnosis is human.

Decisions to pause an agent permanently. Observatory can suggest it based on acute signals, but an operator with authority makes the call. Automatic pause is a concrete risk if thresholds are mis-tuned.

Assessment of new task types where baseline isn't established. You don't have history to set good thresholds, and a period of close human oversight is necessary before Observatory can stand alone.

What Observatory isn't

Observatory isn't a BI tool. It isn't for monthly reports to leadership. It's an operational tool to keep the fleet up, make debugging possible, and catch drift before it becomes loss.

Observatory isn't a replacement for evaluation culture. If nobody in the company knows the difference between good and bad agent output, Observatory delivers no value. It measures what you ask it to measure. If you haven't defined what good quality looks like, no tool will tell you whether you have it.

Who should use Observatory

Companies that have or are planning multiple agents in production, where those agents make decisions or do work with consequences for customers, operations or compliance. If you have one agent answering FAQs, you don't need Observatory. If you have a fleet of five or more working in regulated processes, Observatory is operationally necessary.

After eight weeks our verdict is that Observatory delivers what's at the core of mature agent operations: early warning, traceability and explanatory power. It isn't spectacular. It's solid and reliable, and that's exactly what's needed to keep an agent fleet out of trouble.