Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.tracectrl.ai/llms.txt

Use this file to discover all available pages before exploring further.

​
Overview

The TraceCtrlSpanProcessor is an OpenTelemetry SpanProcessor that runs on every span. It reads framework-set attributes and adds TraceCtrl security context — agent identity, tool risk classification, session correlation, and prompt integrity hashing.

​
Enrichment Pipeline

For each span, the processor runs these enrichment steps:
1

Session ID (on span start)

On span start, attaches the current tracectrl.session_id from the in-process context var. This ensures every span in a session is correlated. On span end, falls back to Agno’s session.id if no explicit session was set.
2

Ingress detection (on span start)

If the span has no valid parent span context, it is marked with tracectrl.ingress = true and tracectrl.trigger_type is inferred from the span name (email, upload, webhook, scheduled, manual).
3

Agent Identity

For AGENT spans, derives tracectrl.agent.id and tracectrl.agent.name from framework-specific attributes (Agno’s agno.agent.id / agno.team.id, OpenInference’s agent.name). If only a name exists, the ID is derived as name.lower().replace(' ', '-'). The framework is inferred from span naming conventions: invoke_agent … → strands, openclaw.… → openclaw, Agno IDs → agno, otherwise unknown.
4

Tool Category and Direction

For spans with a tool.name attribute, classifies the tool into one of 8 risk categories and sets tracectrl.tool.category. Also stamps tracectrl.tool.direction (input, output, or internal).
5

System Prompt Hash

For spans with llm.system (the system prompt), computes a SHA-256 hash (truncated to 16 hex chars / 64 bits) and stores it as tracectrl.system_prompt_hash. A change in this hash between observations indicates prompt drift or tampering.
6

Span Sequence

Sets tracectrl.span_sequence to a monotonic counter (stored as a string), reflecting the chronological order of span creation. This enables reconstructing the exact execution order even when timestamps have insufficient resolution.
7

Memory Operation

For RETRIEVER spans the operation is set to read; for spans whose inferred tool category is memory_write the operation is set to write.
8

Input Source

Sets tracectrl.input.source based on provenance:
  • "external" — tool category is external_api or email
  • "memory" — tool category is memory_read or span is a RETRIEVER
  • "agent" — tracectrl.caller.agent_id is present
  • "user" — default (direct human input)
9

Memory Write Provenance

For memory_write operations, sets tracectrl.memory.write_provenance to the resolved input.source. This enables detecting memory poisoning (e.g., external input being written to a shared knowledge base).

​
Installation

Unlike SystemPromptStamper (which is auto-installed by configure()), TraceCtrlSpanProcessor must be added to your TracerProvider explicitly: 
from tracectrl.processor import TraceCtrlSpanProcessor
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider

provider = TracerProvider()
provider.add_span_processor(TraceCtrlSpanProcessor())
trace.set_tracer_provider(provider)
When using tracectrl.configure(...), the provider it creates is set as the global tracer provider; add TraceCtrlSpanProcessor to that global provider after configuration: 
import tracectrl
from opentelemetry import trace
from tracectrl.processor import TraceCtrlSpanProcessor

tracectrl.configure(endpoint="...", api_key="...")
trace.get_tracer_provider().add_span_processor(TraceCtrlSpanProcessor())

​
System Prompt Stamping

System prompts are surfaced into spans through a complementary flow:
  1. The user calls tag_agent(agent, system_prompt=...) (or tag_agents(...)) once per agent. This registers the prompt in an in-process registry keyed by agent name.
  2. configure() auto-installs a SystemPromptStamper span processor on the global TracerProvider.
  3. On every span, the stamper looks up the agent name and writes llm.system onto the span. The TraceCtrlSpanProcessor then hashes that value into tracectrl.system_prompt_hash.
This indirection means user code never has to wire the stamper directly — calling tag_agent after configure() is enough for prompt hashes to start appearing on spans.

​
Prompt Drift Detection

The tracectrl.system_prompt_hash attribute enables detecting when an agent’s system prompt changes unexpectedly: 
-- Find agents whose system prompt hash changed
SELECT
  agent_id,
  groupArray(DISTINCT SpanAttributes['tracectrl.system_prompt_hash']) AS hashes,
  count(DISTINCT SpanAttributes['tracectrl.system_prompt_hash']) AS unique_hashes
FROM tracectrl.otel_traces FINAL
WHERE SpanAttributes['tracectrl.system_prompt_hash'] != ''
GROUP BY SpanAttributes['tracectrl.agent.id'] AS agent_id
HAVING unique_hashes > 1
A changing system prompt hash doesn’t always mean an attack — it could be a legitimate deployment update. But combined with other signals (e.g., input.source = "external"), it’s a strong indicator of prompt injection.