> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orq.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# LLM traces for debugging
> Explore step-by-step details of every LLM generation. Debug RAG pipelines, evaluators, guardrails, and caching with full workflow visibility and cost tracking.
## What are Traces
**Traces** record every AI request handled by **Orq.ai**, from a single model call to a multi-step **Agent** run. Each trace shows the full workflow of one request as a tree of steps (spans), with inputs, outputs, latency, token usage and cost for each step.
Traces are created for:
* Model calls sent through the [AI Gateway](/ai-gateway/using-the-router)
* [Agent](/ai-studio/ai-engineering/run-agents) runs, including sub-agent and tool calls
* [Deployment](/ai-studio/ai-engineering/deployments) invocations
* Coding sessions from [code assistants](/ai-studio/integrations/code-assistants/overview) connected to **Orq.ai**
* Application code instrumented with [OpenTelemetry frameworks](/ai-studio/integrations/frameworks/overview) or the [`@traced` decorator](#custom-tracing-using-the-traced-decorator)
A trace contains a step for each operation in the request, such as:
* LLM calls, including [Retries and Fallbacks](/ai-studio/ai-engineering/deployments#primary-model-retries-and-fallback)
* Tool and sub-agent calls
* [Evaluators](/ai-studio/optimize/evaluators) and [Guardrails](/ai-studio/ai-engineering/deployments#evaluators-and-guardrails)
* Retrieval from a [Knowledge Base](/ai-studio/ai-engineering/knowledge-bases), including [Embeddings](/ai-studio/ai-engineering/knowledge-bases#embedding-models)
* [Caching](/ai-studio/ai-engineering/deployments#cache)
## Creating Traces
Traces are generated automatically for requests through the **AI Gateway**, **Agents** and **Deployments**. Create custom traces for application code using framework instrumentation or the Orq.ai SDK.
### Framework Instrumentation
For popular AI frameworks and libraries, use automatic instrumentation with OpenTelemetry:
Explore automatic instrumentation for OpenAI, LangChain, LlamaIndex, CrewAI, Autogen, and 15+ other frameworks with OpenTelemetry integration.
### Custom Tracing using the @traced decorator
For custom functions or application workflows, use the `@traced` decorator from the [Python SDK](https://github.com/orq-ai/orq-python). It works with both synchronous and async functions:
```python Sync theme={"theme":{"light":"github-light","dark":"github-dark"}}
import os
from orq_ai_sdk import Orq
from orq_ai_sdk.traced import traced
@traced(name="process_document", type="agent")
def process_document(doc_id: str):
result = {"status": "processed", "doc_id": doc_id}
return result
@traced(name="fetch_context", type="retrieval")
def fetch_relevant_context(query: str):
return {"context": "relevant information", "sources": ["doc1", "doc2"]}
@traced(name="format_response", type="function")
def format_response(raw_output: str, metadata: dict):
return {"formatted": raw_output.strip(), "metadata": metadata}
@traced(name="document_pipeline", type="agent")
def document_pipeline(orq, doc_id: str):
doc = process_document(doc_id)
context = fetch_relevant_context("user query")
response = orq.responses.create(
model="agent/simple-agent",
input=f"Summarize this document: {doc['status']}. Context: {context['context']}",
)
return format_response(response.output[0]["content"][0]["text"], {"doc_id": doc_id})
with Orq(api_key=os.getenv("ORQ_API_KEY")) as orq:
result = document_pipeline(orq, "doc-123")
print(result)
```
```python Async theme={"theme":{"light":"github-light","dark":"github-dark"}}
import os
import asyncio
from orq_ai_sdk import Orq
from orq_ai_sdk.traced import traced
@traced(name="process_document", type="agent")
async def process_document(doc_id: str):
result = {"status": "processed", "doc_id": doc_id}
return result
@traced(name="fetch_context", type="retrieval")
async def fetch_relevant_context(query: str):
return {"context": "relevant information", "sources": ["doc1", "doc2"]}
@traced(name="format_response", type="function")
async def format_response(raw_output: str, metadata: dict):
return {"formatted": raw_output.strip(), "metadata": metadata}
@traced(name="document_pipeline", type="agent")
async def document_pipeline(orq, doc_id: str):
doc = await process_document(doc_id)
context = await fetch_relevant_context("user query")
response = await orq.responses.create_async(
model="agent/simple-agent",
input=f"Summarize this document: {doc['status']}. Context: {context['context']}",
)
return await format_response(response.output[0]["content"][0]["text"], {"doc_id": doc_id})
async def main():
async with Orq(api_key=os.getenv("ORQ_API_KEY")) as orq:
result = await document_pipeline(orq, "doc-123")
print(result)
asyncio.run(main())
```
The result is a single trace with each decorated function appearing as a nested span:
The `@traced` decorator supports the following span types:
| Type | Description |
| - | - |
| `agent` | A high-level orchestration step or agent workflow |
| `embedding` | An embedding generation operation |
| `function` | A general-purpose function or processing step |
| `llm` | A direct LLM API call |
| `retrieval` | A retrieval or knowledge lookup operation |
| `tool` | An external tool call |
Capture or suppress inputs/outputs with `capture_input` and `capture_output`, and attach custom metadata with the `attributes` parameter:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@traced(
name="process_document",
type="agent",
capture_input=False,
capture_output=False,
attributes={"version": "1.0", "env": "production"}
)
def process_document(doc_id: str):
...
```
## How to Lookup Traces
To find traces, open **Traces** in the **Observability** section of the sidebar.
The page has three parts:
* **Query bar**: search, filters, time range, and the Views, Fields, Review and Reload buttons.
* **Activity chart**: the number of **Successful** and **Errors** traces over the selected time range. Click a bar to zoom into that time window, then click to go back to the previous range.
* **Traces table**: one row per trace, newest first. Scroll down to load more traces.
Select a row to open the trace in a side panel, showing the **hierarchy** of events in order of execution and the details of the selected step. With a trace open, press J and K to move to the next or previous trace.
Turn on **Live** to refresh the list every 10 seconds. Live pauses while a trace is open.
Search, filters, columns and time range are stored in the page URL. Copy the URL to share the exact list with a teammate.
### Trace Views
Each trace can be inspected in three views to suit different debugging and analysis needs.
The **Trace** view shows the full execution tree for a single run. Each step is displayed hierarchically, including LLM calls, tool invocations, knowledge retrievals, and memory interactions. Use this view to inspect inputs, outputs, token usage, and latency at every step.
For multi-agent runs, the hierarchy renders as an **agent graph**: parent agents and their sub-agent calls are shown as a nested tree, making it easy to follow how work was delegated across agents and where time was spent. See [Agent Graphs](/ai-studio/observability/agent-graphs) for how graphs are produced and how to debug delegation paths.
Erroring Traces are shown with a icon.
Manually evaluate responses using **Annotations** directly on individual spans. Annotations defined in the project are available on all spans automatically. Learn more in [Annotations](/ai-studio/observability/annotations).
The **Thread** view presents the execution as a conversation thread, showing the sequence of messages exchanged between the user, the agent, and any tools. Use this view to follow the narrative flow of a task and understand how the agent reasoned through a problem.
The **Timeline** view shows execution steps plotted against time, making it easy to identify bottlenecks, measure step durations, and understand which operations ran sequentially or in parallel.
### Search and Time Range
The search box matches part of a trace ID, thread ID or session ID, and the name, span ID, model, provider, agent name or tool name of any span in the trace. Matching is case-insensitive.
Select the time range picker to choose a preset from **5 mins** to **90 days**, or set a **Start date** and **End date** for a custom range. Presets longer than the workspace's data retention are disabled. The last preset used is remembered for the next visit.
Switch the query bar from **Builder** to **OQL** to write the query as a pipeline, for example `fetch traces | filter status == "error" | sort end_time desc | limit 100`. See [Trace and log selection with OQL](/ai-studio/observability/oql).
### Viewing Errors
Traces that encountered an error show a red **error** status badge in the list, and are counted as **Errors** in the activity chart. Add a **Status** filter with the value `error` to scope the list to errors only.
The **Status** filter combines with any other filter, for example:
* Filter by **Status** `error` and an **Identity** to see all errors for a specific user.
* Combine with **Project** or a metadata field to narrow down failures to a particular environment or deployment.
Save the filter as a [view](#creating-custom-views) to return to it in one click.
### Filtering Traces
1. Click Filter in the query bar.
2. Select a field. Search the list to find it faster. Common fields such as **Status**, **Model** and **Trace name** are listed first, followed by the attributes and metadata fields found in the workspace's traces.
3. Select an operator, for example `==`, `!=`, `contains`, `in`, `between` or `is empty`.
4. Pick a suggested value or type one, then click Filter.
The first filter applies immediately. Each filter then appears as a chip below the query bar:
* Click a chip to change its operator or value, or click to remove it.
* Click to add another filter.
* With two or more filters, choose **and** to match all of them or **or** to match any of them.
* Changes to the filters wait until **Run** is clicked. **Clear** removes all filters at once.
When working with [Agents](/ai-studio/ai-engineering/run-agents#traces), access traces directly from the Agent page with automatic filtering for that specific agent.
### Filter grammar
The **Search traces** and **Search logs** APIs take a `filters` array alongside the free-text `query`. Each entry is one condition:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"field": "status",
"op": "in",
"values": ["error"]
}
```
| Key | Value |
| - | - |
| `field` | Field name, up to 128 characters of letters, digits, and `_ . : -` |
| `op` | Comparison operator, from the set below |
| `values` | String array, up to 100 entries of up to 4096 characters each. Numeric and timestamp values are sent as strings |
`filter_operator` combines the entries with `and` (default) or `or`. A request accepts up to 20 filters. `from` and `to` are required and bound the search window; `sort`, `limit`, and `page_token` follow the [Search traces reference](/reference/traces/search-traces) for traces and the [Search logs reference](/reference/logs/search-logs) for logs.
The `op` values are `eq`, `neq`, `in`, `not_in`, `gt`, `gte`, `lt`, `lte`, `between`, `contains`, `exists`, and `not_exists`. The table lists the operators each type commonly accepts; a specific field may accept fewer or more:
| Field type | Operators | Example |
| - | - | - |
| String | `eq`, `neq`, `in`, `not_in`, `contains`, `exists`, `not_exists` | `{"field": "name", "op": "contains", "values": ["checkout"]}` |
| Number | `eq`, `neq`, `gt`, `gte`, `lt`, `lte`, `between`, `exists` | `{"field": "tokens.total", "op": "gt", "values": ["1000"]}` |
| Timestamp | `eq`, `neq`, `gt`, `gte`, `lt`, `lte`, `between`, `exists`, `not_exists` | `{"field": "start_time", "op": "gte", "values": ["2026-10-01T00:00:00Z"]}` |
| Boolean | `eq`, `neq`, `exists`, `not_exists` | `{"field": "eval_passed", "op": "eq", "values": ["true"]}` |
| Enum | `eq`, `neq`, `in`, `not_in` | `{"field": "status", "op": "in", "values": ["error"]}` |
`between` takes exactly two values, the lower bound first. `exists` and `not_exists` take no values.
Field-level support varies, so call `GET /v3/traces/fields` for the filter fields, their types, and the operators each one accepts. Log search uses the same filter object and operator vocabulary.
### Configuring Columns
Click Fields to open the **Fields** panel and choose which columns the table shows. Every attribute and metadata field found in the workspace's traces can be shown as a column, next to the built-in columns such as **Start time**, **Status**, **Trace name**, **Duration**, **Total tokens**, **Total cost** and **Trace ID**.
* **Show or hide**: check or uncheck a field. Search the panel to find a field by name.
* **Reorder**: switch to **Selected** and drag a field by to move its column. Reordering only works while all selected fields are visible, so clear the search box first.
* **Resize**: drag the right edge of a column header.
* **Rename**: open on an attribute or metadata column header and select **Edit name** to give it a display name.
* **Remove**: open on a column header and select **Remove** to hide it.
Column widths and order are remembered in the browser. To keep a column layout and share it with the workspace, save it in a view.
### Creating Custom Views
A view saves the current filters, columns and column names under a name, to reopen them later from the Views dropdown.
To create a view:
1. Set the filters and columns.
2. Open Views and select **Create new view**.
3. Enter a **Name**.
4. Optionally check **Make this view private for myself**. Views are shared with workspace members by default.
5. Click **Create View**.
To update a view, select it, change the filters or columns, then open **Save** next to the filter chips and select **Save to this view**. When only the columns changed, click **Update view** in the query bar instead.
Hover a view in the dropdown to manage it:
* : set or unset it as the default view. The default view opens automatically on the Traces page.
* > **Edit**: change its name or privacy.
* > **Share**: copy a link that opens the view with its filters.
* > **Delete**: delete the view.
Select **Default view** to return to the standard columns with no saved filters.
Views saved on the previous Traces page were moved to the new page. Filters that the new page cannot express were adjusted or removed.
Manage saved Views programmatically through the [Views API](/ai-studio/observability/views-api).
## Review Traces
Click Review in the toolbar to open the review screen and step through traces one by one. Applying one or more [filters](#filtering-traces) first narrows the review queue to matching traces; with no filter active, Review steps through the current list.
The screen follows the same three-panel layout as [Annotation Queues](/ai-studio/observability/annotation-queues#use-annotation-queues):
* **Left**: Inputs and Metrics for the selected span.
* **Center**: Full trace conversation.
* **Right**: [Annotation](/ai-studio/observability/annotations) controls for manual review. Select in the Annotations panel header to [create a new annotation](/ai-studio/observability/annotations#create-annotations) without leaving the screen.
Annotations applied here are written back to the span and are queryable via the [**Orq MCP**](/ai-studio/integrations/code-assistants/orq-mcp).
## Correct an Evaluator Result
Hover an Evaluator result to reveal a , and select it to open a popover with:
* **Value**: the corrected result, matching the Evaluator's own output type (a toggle for boolean Evaluators, a text or number field otherwise).
* **Explanation**: an optional comment.
Select **Correct** to submit, or **Update** to replace an existing correction. The correction appears under the original result as **Corrected to \**, along with the explanation and the reviewer's name.
Corrections are written back to the Trace like any other annotation. This gives a basis to compare an Evaluator's own results against reviewer corrections over time, using that signal (for example through the [**Orq MCP**](/ai-studio/integrations/code-assistants/orq-mcp)) to see where an Evaluator's configuration may need adjustment.
This corrects the Evaluator's *result*, not the AI response text. For rewriting the response itself, use the Logs [**Add a Text Correction**](/ai-studio/observability/annotations#use-annotations) flow.
See [Correct an Evaluator Result](/ai-studio/observability/annotations#use-annotations) in the API & SDK tab for the API/SDK equivalent.
## Threads
Visualize conversation history as Threads to follow the full sequence of messages across an agent session.
## Reference
How token usage is captured and cost is calculated per LLM request.
Complete reference for all `orq.*` span attributes emitted on traces, webhook payloads, and trace exports.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.