Skip to main content

​
Overview

Genesis emits OpenTelemetry distributed traces for every conversation thread. A “trace” is a tree of timed spans that captures everything that happens between a user prompt and the final response — the LLM calls Genesis made, the tools the model invoked, how long each step took, and where errors occurred. Traces are exported over standard OTLP/gRPC to any compatible backend: Jaeger, Tempo, Honeycomb, Datadog APM, New Relic, Grafana Cloud, Lightstep, Splunk Observability, AWS X-Ray (via collector), etc. Genesis ships traces through the standard OpenTelemetry SDK, so any backend that speaks OTLP works without Genesis-specific glue.

​
Configuration

Genesis performs a short startup availability check against the OTLP endpoint before enabling the exporter. If the collector is unreachable, tracing is disabled cleanly and Genesis continues running. Set OTEL_TRACING_FORCE_ENABLE=true only when you need to skip that preflight (for example, when the collector starts after Genesis).
Restart required. All observability.tracing.* settings are read once at startup. After changing any of them — including enabling tracing for the first time, swapping the endpoint, or rotating the auth token — restart Genesis for the change to take effect.

​
Headers and authentication

observability.tracing.headers is treated as the opaque value portion of an Authorization: Bearer <value> header. Genesis prepends Bearer itself, so the setting should contain the bare token only. The single most common gotcha is the space inside Bearer tokens: the Genesis Secrets UI strips literal spaces from values to prevent accidental copy-paste damage. If your auth value itself contains a space (it usually does not for a JWT, but can for some proxy configurations), encode it as %20 — the value is URL-decoded before being sent. If your collector requires headers other than Authorization: Bearer … (for example, a DD-API-KEY for Datadog, or a tenant-id alongside the bearer), use the Metrics export’s observability.metrics.headers setting for those custom headers on the metrics path, and route traces through an OpenTelemetry Collector that injects the headers your APM backend expects. The traces path itself currently emits only the Authorization: Bearer … header.

​
What gets traced

Genesis groups every span by the conversation thread_id. The thread_id is deterministically converted to a 128-bit trace ID, so every span emitted while handling that thread shares the same trace. You’ll see one trace per conversation, with each LLM call and tool call as a child span — useful for end-to-end latency analysis, error diagnosis, and cost attribution.

​
Global resource attributes

Every span carries these resource attributes:

​
Span inventory

​
Root spans

Created at the start of a job. The duration of this span is the wall-clock duration of the whole job.

​
LLM call spans

Each LLM API call emits two spans: Both spans carry the same baseline attributes: The completed span adds, when the call returns successfully: On failure, the completed span instead carries llm.success=false, llm.error_type, and llm.error_message, and the span status is set to ERROR.

​
Tool call spans

Every tool invocation emits a span named tool.<tool_function_name> (e.g. tool.git_action, tool.search_threads, tool.manage_context). Baseline attributes: When present in the tool arguments, the following operational parameters are also captured (values are recorded because they describe what the tool is doing, not the data it is acting on): action, operation, command, method, mode, type, task, workflow, step, phase, plus blueprint-related fields (blueprint_name, demo_name, exit_ticket_from_previous, todo_id, project_id) and path-related fields (file_path, path, filename, directory, repo_id, git_path). When the tool fails — either by raising an exception or by returning a result with success=false or a non-empty error field — the span records tool.success=false, tool.error_type, tool.error_message (or a tool.result.error event), and the span status is set to ERROR.

​
Typical trace shape

This shape lets you answer questions like:
  • “What’s the p95 end-to-end latency for chat threads on this instance?” → look at job.execution duration distribution.
  • “Where is the time going inside slow threads?” → expand job.execution and compare the llm.completed.* and tool.* durations.
  • “Which tool fails most often?” → group span errors by tool.name.
  • “Are any LLM calls hanging?” → look for llm.start.<model> spans with no matching llm.completed.<model> sibling.

​
Privacy

Tracing carries more conversation context than metrics. Make sure your collector retention policy is appropriate for the data described below.
  • llm.last_message.content contains text content of the most recent non-system message (typically the user’s prompt or the assistant’s tool-call request). System-injected context messages are filtered out, but ordinary user prompts are exported. If this is unacceptable, set observability.tracing.enabled=false or have your collector strip the llm.last_message.content attribute on ingest.
  • tool.thread_id and tool.session_id are the raw Genesis thread identifiers. They are not user identifiers, but they let an observer correlate spans within the same conversation.
  • tool.param_count and tool.param_names are captured; parameter values are NOT (with the explicit exception of the operational and path fields listed above, which describe what the tool is doing rather than the data it is acting on).
  • user_id is captured as-is when it’s attached to a span (root spans created with an explicit user_id). Unlike metrics, traces do not hash this value. Consider stripping user_id at collector ingest if your retention policy requires it.
  • Tool error messages may contain operational detail (exception text, error strings from downstream services). Treat error attributes as you would log lines.

​
Volume and sampling

Genesis currently exports every span — there is no tail-based or head-based sampling. Each conversation typically produces one root span, two spans per LLM call (start + completed), and one span per tool invocation. High-volume deployments should size their OTLP collector and APM backend storage accordingly, or run a sampling processor in their collector pipeline (e.g. the tail_sampling processor) before forwarding to long-term storage.

​
Verifying it works

After enabling tracing and restarting Genesis, look in the server logs for one of these messages at startup: If your APM backend supports a “service map” or “service list” view, look for genesis-bots after Genesis has handled at least one conversation.

​
Disabling tracing

Set observability.tracing.enabled=false and restart. No spans are emitted, the OTLP exporter is not configured, and no trace data leaves the host. You can set this through environment as GENESIS__OBSERVABILITY__TRACING__ENABLED=false.