> ## Documentation Index
> Fetch the complete documentation index at: https://docs.genesiscomputing.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Tracing

> Configure OpenTelemetry distributed traces for Genesis, including span semantics, attribute schemas, and privacy guidance.

## Overview

Genesis emits OpenTelemetry distributed traces for every conversation
thread. A "trace" is a tree of timed spans that captures everything
that happens between a user prompt and the final response — the LLM
calls Genesis made, the tools the model invoked, how long each step
took, and where errors occurred.

Traces are exported over standard **OTLP/gRPC** to any compatible
backend: Jaeger, Tempo, Honeycomb, Datadog APM, New Relic, Grafana
Cloud, Lightstep, Splunk Observability, AWS X-Ray (via collector), etc.
Genesis ships traces through the standard OpenTelemetry SDK, so any
backend that speaks OTLP works without Genesis-specific glue.

## Configuration

| Setting | Type | Default | Description |
| - | - | - | - |
| `observability.tracing.enabled` | boolean | `true` | Master switch for tracing. When `false`, Genesis emits no spans and skips OTLP setup entirely. |
| `observability.tracing.endpoint` | string | `http://localhost:4317` | OTLP/gRPC endpoint URL. This is also the fallback for `observability.metrics.endpoint` when metrics export does not set its own endpoint. |
| `observability.tracing.headers` | string | empty | Bearer token value sent on every span export request. Genesis prepends `Bearer ` automatically. See [Headers and authentication](#headers-and-authentication) below. |
| `observability.tracing.console_export` | boolean | `false` | Development helper: when `true` and no OTLP endpoint is reachable, Genesis prints spans to the console instead of dropping them. |

Genesis performs a short startup availability check against the OTLP
endpoint before enabling the exporter. If the collector is unreachable,
tracing is disabled cleanly and Genesis continues running. Set
`OTEL_TRACING_FORCE_ENABLE=true` only when you need to skip that
preflight (for example, when the collector starts after Genesis).

<Note>
  **Restart required.** All `observability.tracing.*` settings are read
  once at startup. After changing any of them — including enabling
  tracing for the first time, swapping the endpoint, or rotating the auth
  token — restart Genesis for the change to take effect.
</Note>

### Headers and authentication

`observability.tracing.headers` is treated as the **opaque value**
portion of an `Authorization: Bearer <value>` header. Genesis prepends
`Bearer ` itself, so the setting should contain the bare token only.

The single most common gotcha is the **space inside `Bearer ` tokens**:
the Genesis Secrets UI strips literal spaces from values to prevent
accidental copy-paste damage. If your auth value itself contains a
space (it usually does not for a JWT, but can for some proxy
configurations), encode it as `%20` — the value is URL-decoded before
being sent.

| You want to send | Enter in `observability.tracing.headers` |
| - | - |
| `Authorization: Bearer eyJhbGciOi…` | `eyJhbGciOi…` (Genesis adds the `Bearer ` prefix) |
| A token with an embedded space | encode the space as `%20` |

If your collector requires headers other than `Authorization: Bearer
…` (for example, a `DD-API-KEY` for Datadog, or a tenant-id alongside
the bearer), use the **Metrics** export's
`observability.metrics.headers` setting for those custom headers on
the metrics path, and route traces through an OpenTelemetry Collector
that injects the headers your APM backend expects. The traces path
itself currently emits only the `Authorization: Bearer …` header.

## What gets traced

Genesis groups every span by the conversation `thread_id`. The
`thread_id` is deterministically converted to a 128-bit trace ID, so
every span emitted while handling that thread shares the same trace.
You'll see one trace per conversation, with each LLM call and tool call
as a child span — useful for end-to-end latency analysis, error
diagnosis, and cost attribution.

### Global resource attributes

Every span carries these resource attributes:

| Attribute | Value |
| - | - |
| `service.name` | `genesis-bots` |
| `service.version` | Genesis service version identifier |
| `hostname` | The host the Genesis process is running on |
| `environment` | From `server.environment` (defaults to `production`) |

### Span inventory

#### Root spans

Created at the start of a job. The duration of this span is the
wall-clock duration of the whole job.

| Span name | When created | Key attributes |
| - | - | - |
| `job.execution` (default) or a job-specific name like `blueprint.execution`, `program.execution`, `chat.execution` | Once per job/conversation | `span.type=root`, `job.type` (e.g. `blueprint`, `program`, `chat`, `unknown`), `thread_id`, `session_id`, `bot_id`, optional `user_id` |

#### LLM call spans

Each LLM API call emits **two** spans:

| Span name | When emitted | Why |
| - | - | - |
| `llm.start.<model>` | Immediately when the call is dispatched, closed instantly | Lets you see that a call was attempted even if it later hangs or never returns |
| `llm.completed.<model>` | When the call finishes (success or failure) | Tracks the actual wall-clock duration of the LLM call |

Both spans carry the same baseline attributes:

| Attribute | Description |
| - | - |
| `llm.model` | Model name (e.g. `gpt-5`, `claude-sonnet-4-6`) |
| `llm.type` | `chat` |
| `llm.system` | `genesis-bots` |
| `llm.message_count` | Number of messages in the conversation context |
| `llm.reasoning_effort` | Reasoning effort parameter when forwarded (OpenAI GPT-5 / o1 / o3 / o4 families only) |
| `llm.temperature`, `llm.max_tokens`, `llm.top_p`, `llm.stream` | Request parameters, when provided |
| `llm.last_message.role` | Role of the most recent non-system message (e.g. `user`, `assistant`) |
| `llm.last_message.content` | **Text content** of the most recent non-system message. See [Privacy](#privacy) below. |
| `llm.parent_thread_id` | Present when the call belongs to a sub-thread (`handoff`, etc.) |

The completed span adds, when the call returns successfully:

| Attribute | Description |
| - | - |
| `llm.duration_ms` | Wall-clock LLM call duration in milliseconds |
| `llm.success` | `true` |
| `llm.usage.prompt_tokens` | Input tokens consumed |
| `llm.usage.completion_tokens` | Output tokens emitted |
| `llm.usage.total_tokens` | Sum reported by the provider |
| `llm.tool_calls_count` | Number of tool calls the model requested in its response |

On failure, the completed span instead carries `llm.success=false`,
`llm.error_type`, and `llm.error_message`, and the span status is set
to `ERROR`.

#### Tool call spans

Every tool invocation emits a span named `tool.<tool_function_name>`
(e.g. `tool.git_action`, `tool.search_threads`, `tool.manage_context`).

Baseline attributes:

| Attribute | Description |
| - | - |
| `tool.name` | Tool function name |
| `tool.groups` | Comma-separated tool group names (e.g. `git_tools`) |
| `tool.module` | Python module the tool is defined in (debugging aid) |
| `tool.bot_id`, `tool.thread_id`, `tool.session_id` | Conversation context |
| `tool.parent_thread_id` | Present when this call is part of a sub-thread |
| `tool.param_count` | Number of parameters passed (not the values themselves) |
| `tool.param_names` | Comma-separated parameter names (not values) |
| `tool.duration_ms` | Tool execution duration in milliseconds |
| `tool.success` | `true` when the tool returned without error and without a `success=false` result; otherwise `false` |
| `tool.result_type` | Python type name of the returned value |
| `tool.result_keys` | Number of keys in the result dict (when applicable) |
| `tool.result_length` | Length of the result list/tuple (when applicable) |

When present in the tool arguments, the following **operational
parameters** are also captured (values are recorded because they
describe *what* the tool is doing, not the data it is acting on):
`action`, `operation`, `command`, `method`, `mode`, `type`, `task`,
`workflow`, `step`, `phase`, plus blueprint-related fields
(`blueprint_name`, `demo_name`, `exit_ticket_from_previous`, `todo_id`,
`project_id`) and path-related fields (`file_path`, `path`,
`filename`, `directory`, `repo_id`, `git_path`).

When the tool fails — either by raising an exception or by returning a
result with `success=false` or a non-empty `error` field — the span
records `tool.success=false`, `tool.error_type`, `tool.error_message`
(or a `tool.result.error` event), and the span status is set to
`ERROR`.

### Typical trace shape

```
Trace (derived from thread_id "abc123")
├─ job.execution                          ← root span, total wall-clock time
│  ├─ llm.start.gpt-5                     ← model dispatch
│  ├─ llm.completed.gpt-5                 ← model returned, tokens recorded
│  ├─ tool.search_threads                 ← model asked to call a tool
│  ├─ tool.git_action                     ← another tool call
│  ├─ llm.start.gpt-5                     ← second LLM call
│  └─ llm.completed.gpt-5                 ← final answer
```

This shape lets you answer questions like:

* *"What's the p95 end-to-end latency for chat threads on this
  instance?"* → look at `job.execution` duration distribution.
* *"Where is the time going inside slow threads?"* → expand
  `job.execution` and compare the `llm.completed.*` and `tool.*`
  durations.
* *"Which tool fails most often?"* → group span errors by `tool.name`.
* *"Are any LLM calls hanging?"* → look for `llm.start.<model>` spans
  with no matching `llm.completed.<model>` sibling.

## Privacy

Tracing carries more conversation context than metrics. Make sure your
collector retention policy is appropriate for the data described below.

* **`llm.last_message.content` contains text content** of the most
  recent non-system message (typically the user's prompt or the
  assistant's tool-call request). System-injected context messages are
  filtered out, but ordinary user prompts are exported. If this is
  unacceptable, set `observability.tracing.enabled=false` or have your
  collector strip the `llm.last_message.content` attribute on ingest.
* **`tool.thread_id` and `tool.session_id`** are the raw Genesis
  thread identifiers. They are not user identifiers, but they let an
  observer correlate spans within the same conversation.
* **`tool.param_count` and `tool.param_names` are captured; parameter
  *values* are NOT** (with the explicit exception of the operational
  and path fields listed above, which describe what the tool is
  doing rather than the data it is acting on).
* **`user_id` is captured as-is when it's attached to a span** (root
  spans created with an explicit `user_id`). Unlike metrics, traces do
  not hash this value. Consider stripping `user_id` at collector
  ingest if your retention policy requires it.
* **Tool error messages may contain operational detail** (exception
  text, error strings from downstream services). Treat error
  attributes as you would log lines.

## Volume and sampling

Genesis currently exports **every** span — there is no tail-based or
head-based sampling. Each conversation typically produces one root
span, two spans per LLM call (`start` + `completed`), and one span per
tool invocation. High-volume deployments should size their OTLP
collector and APM backend storage accordingly, or run a sampling
processor in their collector pipeline (e.g. the
[tail\_sampling processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor))
before forwarding to long-term storage.

## Verifying it works

After enabling tracing and restarting Genesis, look in the server logs
for one of these messages at startup:

| Log line | Meaning |
| - | - |
| `OpenTelemetry tracing enabled with OTLP exporter: <endpoint>` | Tracing is on and the exporter was created successfully. Spans should begin arriving at your backend within seconds of the first conversation. |
| `OTLP endpoint <endpoint> is not available, tracing disabled` | The startup availability check failed. The endpoint is wrong, the collector is down, or the network path is blocked. Genesis continues running with tracing disabled until the next restart. |
| `Failed to setup OTLP exporter, tracing disabled: <error>` | The exporter could not be initialized (auth rejected, TLS error, etc.). Check the error detail. |
| `OpenTelemetry tracing disabled via observability.tracing.enabled` | `observability.tracing.enabled` is `false`. |

If your APM backend supports a "service map" or "service list" view,
look for `genesis-bots` after Genesis has handled at least one
conversation.

## Disabling tracing

Set `observability.tracing.enabled=false` and restart. No spans are
emitted, the OTLP exporter is not configured, and no trace data leaves
the host. You can set this through environment as
`GENESIS__OBSERVABILITY__TRACING__ENABLED=false`.

## Related pages

* [Metrics](/docs/Setup/Metrics) — OTLP metrics export, configured the same way
* [Pulse Monitoring](/docs/User-Guide/Pulse-Monitoring)
* [Advanced Settings](/docs/Setup/Advanced-Settings)