LLM traces for debugging

​

What are Traces

• Model calls sent through the AI Gateway
• Agent runs, including sub-agent and tool calls
• Deployment invocations
• Coding sessions from code assistants connected to Orq.ai
• Application code instrumented with OpenTelemetry frameworks or the @traced decorator

• LLM calls, including Retries and Fallbacks
• Tool and sub-agent calls
• Evaluators and Guardrails
• Retrieval from a Knowledge Base, including Embeddings
• Caching

​

Creating Traces

​

Framework Instrumentation

​

Custom Tracing using the @traced decorator

import os
from orq_ai_sdk import Orq
from orq_ai_sdk.traced import traced

@traced(name="process_document", type="agent")
def process_document(doc_id: str):
    result = {"status": "processed", "doc_id": doc_id}
    return result

@traced(name="fetch_context", type="retrieval")
def fetch_relevant_context(query: str):
    return {"context": "relevant information", "sources": ["doc1", "doc2"]}

@traced(name="format_response", type="function")
def format_response(raw_output: str, metadata: dict):
    return {"formatted": raw_output.strip(), "metadata": metadata}

@traced(name="document_pipeline", type="agent")
def document_pipeline(orq, doc_id: str):
    doc = process_document(doc_id)
    context = fetch_relevant_context("user query")

    response = orq.responses.create(
        model="agent/simple-agent",
        input=f"Summarize this document: {doc['status']}. Context: {context['context']}",
    )

    return format_response(response.output[0]["content"][0]["text"], {"doc_id": doc_id})

with Orq(api_key=os.getenv("ORQ_API_KEY")) as orq:
    result = document_pipeline(orq, "doc-123")
    print(result)

import os
import asyncio
from orq_ai_sdk import Orq
from orq_ai_sdk.traced import traced

@traced(name="process_document", type="agent")
async def process_document(doc_id: str):
    result = {"status": "processed", "doc_id": doc_id}
    return result

@traced(name="fetch_context", type="retrieval")
async def fetch_relevant_context(query: str):
    return {"context": "relevant information", "sources": ["doc1", "doc2"]}

@traced(name="format_response", type="function")
async def format_response(raw_output: str, metadata: dict):
    return {"formatted": raw_output.strip(), "metadata": metadata}

@traced(name="document_pipeline", type="agent")
async def document_pipeline(orq, doc_id: str):
    doc = await process_document(doc_id)
    context = await fetch_relevant_context("user query")

    response = await orq.responses.create_async(
        model="agent/simple-agent",
        input=f"Summarize this document: {doc['status']}. Context: {context['context']}",
    )

    return await format_response(response.output[0]["content"][0]["text"], {"doc_id": doc_id})

async def main():
    async with Orq(api_key=os.getenv("ORQ_API_KEY")) as orq:
        result = await document_pipeline(orq, "doc-123")
        print(result)

asyncio.run(main())

@traced(
    name="process_document",
    type="agent",
    capture_input=False,
    capture_output=False,
    attributes={"version": "1.0", "env": "production"}
)
def process_document(doc_id: str):
    ...

​

How to Lookup Traces

• Query bar: search, filters, time range, and the Views, Fields, Review and Reload buttons.
• Activity chart: the number of Successful and Errors traces over the selected time range. Click a bar to zoom into that time window, then click to go back to the previous range.
• Traces table: one row per trace, newest first. Scroll down to load more traces.

​

Trace Views

• Trace
• Thread
• Timeline

The Trace view shows the full execution tree for a single run. Each step is displayed hierarchically, including LLM calls, tool invocations, knowledge retrievals, and memory interactions. Use this view to inspect inputs, outputs, token usage, and latency at every step.For multi-agent runs, the hierarchy renders as an agent graph: parent agents and their sub-agent calls are shown as a nested tree, making it easy to follow how work was delegated across agents and where time was spent. See Agent Graphs for how graphs are produced and how to debug delegation paths.

Erroring Traces are shown with a icon.

Manually evaluate responses using Annotations directly on individual spans. Annotations defined in the project are available on all spans automatically. Learn more in Annotations.

The Thread view presents the execution as a conversation thread, showing the sequence of messages exchanged between the user, the agent, and any tools. Use this view to follow the narrative flow of a task and understand how the agent reasoned through a problem.

The Timeline view shows execution steps plotted against time, making it easy to identify bottlenecks, measure step durations, and understand which operations ran sequentially or in parallel.

​

Search and Time Range

​

Viewing Errors

• Filter by Status error and an Identity to see all errors for a specific user.
• Combine with Project or a metadata field to narrow down failures to a particular environment or deployment.

​

Filtering Traces

1. Click Filter in the query bar.
2. Select a field. Search the list to find it faster. Common fields such as Status, Model and Trace name are listed first, followed by the attributes and metadata fields found in the workspace’s traces.
3. Select an operator, for example ==, !=, contains, in, between or is empty.
4. Pick a suggested value or type one, then click Filter.

• Click a chip to change its operator or value, or click to remove it.
• Click to add another filter.
• With two or more filters, choose and to match all of them or or to match any of them.
• Changes to the filters wait until Run is clicked. Clear removes all filters at once.

​

Filter grammar

{
  "field": "status",
  "op": "in",
  "values": ["error"]
}

​

Configuring Columns

• Show or hide: check or uncheck a field. Search the panel to find a field by name.
• Reorder: switch to Selected and drag a field by to move its column. Reordering only works while all selected fields are visible, so clear the search box first.
• Resize: drag the right edge of a column header.
• Rename: open on an attribute or metadata column header and select Edit name to give it a display name.
• Remove: open on a column header and select Remove to hide it.

​

Creating Custom Views

1. Set the filters and columns.
2. Open Views and select Create new view.
3. Enter a Name.
4. Optionally check Make this view private for myself. Views are shared with workspace members by default.
5. Click Create View.

• : set or unset it as the default view. The default view opens automatically on the Traces page.
• > Edit: change its name or privacy.
• > Share: copy a link that opens the view with its filters.
• > Delete: delete the view.

​

Review Traces

• Left: Inputs and Metrics for the selected span.
• Center: Full trace conversation.
• Right: Annotation controls for manual review. Select in the Annotations panel header to create a new annotation without leaving the screen.

​

Correct an Evaluator Result

• Value: the corrected result, matching the Evaluator’s own output type (a toggle for boolean Evaluators, a text or number field otherwise).
• Explanation: an optional comment.

​

Threads

​

Reference