Where does time go in an AI agent? Tracing inference, tools, and orchestration

At 9:00 am, an alert reports a rising error rate on your service. An on-call agent sends the alert to a model, which decides to check recent error logs. A tool searches those logs. The model then reads the results and writes an explanation for the engineer. The answer arrives 18 seconds later.

Teams with a slow or unpredictable agent often reach for a faster model first. In this run, that would barely help. Twelve of the 18 seconds went to the log tool, including one query that timed out. Even if every model call returned instantly, the answer would still take more than 12 seconds. In any agent, the delay can come from model calls, tool calls, the application’s coordination between steps, retries or executing more steps than the task needs.

Tracing a complete task helps you find the steps holding up the answer and decide where to focus your engineering effort. The hypothetical example below shows how to measure the time from alert to answer and separate time spent on model calls, tools, retries, and waits. You’ll also check whether the task succeeded, so you can tell whether a change delivered a useful answer sooner or simply made the agent stop earlier. If a model request is the slow step, Parasail’s guide to reducing LLM latency in production explains how to investigate it.

Measure end-to-end agent latency, including orchestration

End-to-end agent latency is the time from when a task arrives to when the complete answer reaches the person waiting for it. In this example, the clock starts when your system receives the alert and stops when the complete answer reaches the on-call channel. This measures how long the agent takes to finish a task, end-to-end. The total should include the wait for an agent worker to pick up the task, as well as any formatting and delivery after the final model response.

Orchestration is the work your application does between model and tool calls. It starts requests, passes results to the next step, and decides whether to retry or stop. In this example, the model chooses the log query and the application handles the retry. Timing both shows which part held up the task.

A trace is a record of every step in one agent task, showing when each step started and how long it took. A trace ID is the unique identifier that links those steps. Use it to find a run’s model requests, tool attempts and application events together. In OpenTelemetry's tracing model, a span is one timed operation inside a trace, such as a single model request or tool attempt. Trace context is the trace ID and parent span that travel with each request. When services pass it along, every span stays attached to the same trace.

Before comparing timings, decide what a useful answer needs to contain. In our example, the agent must return relevant log evidence, explain what it suggests, and recommend a next check. It should separate observations from hypotheses. It doesn’t need to resolve the incident. An answer without supporting evidence fails, even if every request completes without an error.

Read an agent trace step-by-step

Our example run has trace ID 7e24c8a60f194d23b5a981c047d6e302. It starts at 09:00:00.000 and finishes at 09:00:18.000. The task succeeded after recovering from one tool timeout.

Read the table from top to bottom. The start time shows when each step began and the duration shows how long it lasted. Each step here waits for the previous one, so no recorded intervals overlap.

The logs show timeout errors from a downstream dependency. The final answer reports those errors and recommends checking that service's health. It also notes that the root cause is unconfirmed. That gives the engineer evidence and a next step.

Group the rows by the work they represent.

• 4.1 seconds in model calls
• 12 seconds in tool attempts
• 1.4 seconds in waits
• 0.5 seconds in delivery

Together they account for all 18 seconds. Tool attempts take up about 67% of the run. That means the log search and its timeout are the first places to investigate.

The trace shows where the time went. It doesn’t show why the query timed out. The log service could have been slow, or the request could have been delayed before reaching it. The outer tool timer can’t tell those apart.

Follow the critical path that sets the finish time

To see which delays affect when the answer arrives, ask what each step needs before it can start. The agent needs the model's query before searching logs, then needs the log results before asking the model to explain them. Each step in this run holds up everything that follows it.

This sequence is the task's critical path. The critical path is the chain of work and waits that determines when a task can finish. Because these steps happen one after another, their durations add up. The failed attempt and retry delay alone consume 9 seconds before the successful log query begins.

When an agent runs calls one after another, their durations add up. When it runs independent calls at the same time, the next step waits only for the slowest one. Suppose another hypothetical run starts an independent status check and log query together. They take 3 seconds and 5 seconds respectively, and the next step can’t start without both results. When run simultaneously, the agent can proceed after 5 seconds, plus any overhead for starting the calls and collecting results. Adding the calls would incorrectly count 8 seconds of elapsed time.

Making the 3-second status check faster wouldn't shorten that stage. The agent would still be waiting for the 5-second log query.

Check that the timestamps show actual overlap before treating calls as parallel. Their inputs also have to permit it. If the status check needs a service name found in the logs, it must wait for the log query to finish.

Timers that contain smaller timers are another source of double counting. If a log-search span covers both attempts and the retry wait, it lasts 13 seconds (8 + 1 + 4). These inner timers are called nested spans and are already accounted for in the 13 seconds. Use the whole-task timer to establish total elapsed time, then use the smaller spans to explain it.

Count calls, retries, and waits to find repeated work

A task can be slow because one call takes too long or because the agent makes too many calls. Record counts alongside durations to see which explanation fits.

Our example agent requested one log search, but the tool tried it twice. Record that as 1 logical tool call, 2 attempts, 1 retry, and 1 failed attempt. The model was called twice. Keep the failed attempt in the record even though the task succeeded, or you'll lose the 8-second timeout that explains much of the delay.

Record input and output tokens for each model call when they’re available. Input includes the alert and any log results and output includes the query or final answer.

Suppose the first call used 1,200 input tokens and produced 120 output tokens, while the second used 2,800 and produced 260. That's 4,000 input tokens and 380 output tokens across the run. Context sent in both requests counts in both totals. Mark unavailable usage as unknown rather than zero.

Next, ask what each extra call contributed. Did it interpret new evidence, or repeat a plan without learning anything? Check the stopping rules, which are the conditions that tell the agent it has enough information to answer or should give up. Several short calls can add substantial time if the agent keeps repeating work.

Record why each retry happened and which part of the application requested it. The agent and the client library that sends a tool request may have separate retry settings. The Amazon Builders’ Library explains how retries at several layers can multiply downstream requests. One visible tool invocation may contain several attempts.

Also inspect pauses between recorded steps. The task might be waiting for an available worker, a retry delay, or application processing you haven't timed yet. Label known waits and leave unexplained gaps visible until you can account for them. A pause between model calls doesn't mean the model was running.

Decide whether to inspect the model, tool, or agent logic

The trace should tell you what to check next. For example, a long tool call tells you where to look more closely. It doesn't yet tell you whether the tool's server or the connection caused the delay.

If a model request is holding up the task, split its time into the input and output phases. Parasail’s explainer on prefill vs. decode in LLM inference shows how input size drives time to first token and how output length drives generation time. Keep the model request’s timer separate from the whole-task timer. Your application’s wait for a model response can include more than model computation.

For a slow tool, compare the duration seen by your application with the time recorded by the tool's server. A large gap points to the network, a connection queue or the client. A small gap points to the server’s own processing. If you only have the outer tool timer, collect more detail before choosing a fix.

In this example, start with the timed-out log request. Did the log service receive it, and how long did the service spend processing it? Then check the client's timeout and retry settings. A successful retry by itself doesn't establish whether the timeout was too short or whether retries should be removed.

Check answer quality before saying your agent is faster

After making a change, compare complete tasks under similar conditions. Keep the task definition and answer-quality criteria consistent, and record the model and agent configuration. Compare similar alerts with one another so a shift toward easier tasks doesn't look like a speed improvement.

Include successful and failed runs, and report their timings separately. Google's guidance on monitoring distributed systems explains why this matters for request latency. Failed requests can return quickly and make a combined latency number look better. For an agent, check the answer too. A task can complete without an error and still fail the quality check.

An agent that gives up on the log search sooner might finish faster while returning no evidence. Removing the final model call could save time while leaving the engineer with raw logs instead of the requested explanation. Review the answer alongside its trace to confirm that the faster run still meets the original goal.

Collect multiple runs before deciding what usually slows your agent down. Compare slow and failed tasks with ordinary successful ones, looking at their step counts, waits, and tool errors. Our hypothetical example teaches you how to read a trace, but it can't establish your team's typical bottleneck.

Record these details for every agent trace

Capture these details for each complete task.

• Trace ID and task boundaries. Record when the alert arrived, when the answer was delivered, and total elapsed time.
• Step boundaries and durations. Identify model calls, tool attempts, waits, delivery, and any actual overlap.
• Task outcome. Record success or failure against a defined quality check, with a reason.
• Call and token counts. Count model calls and logical tool calls, with input and output tokens where available.
• Retries and tool failures. Preserve attempt counts, retry counts, error reasons, and recovered failures.

Keep this record for successful and failed runs. Use it to choose the next investigation. Then check that the change delivers a useful answer sooner. If the trace points at the model call itself, test other models on the same task using Parasail’s model catalog.