Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.tracectrl.ai/llms.txt

Use this file to discover all available pages before exploring further.

​
Overview

The TraceCtrl Engine is a FastAPI application that:
  1. Runs a background pipeline every PIPELINE_INTERVAL_SECONDS to process spans.
  2. Serves a REST API under /api/v1 for the dashboard and external integrations.
  3. Maintains cumulative state in ClickHouse using ReplacingMergeTree tables.

​
Pipeline

Scheduled via APScheduler at the interval set by PIPELINE_INTERVAL_SECONDS (default: 60). Each tick re-reads all spans — there is no watermark on the main path; ReplacingMergeTree handles deduplication via background merges.
1

Fetch spans

Read every span from otel_traces (via fetch_all_spans()).
2

Update agent inventory

Group spans by tracectrl.agent.id and upsert agent records. Tracks cumulative observation_count, run_count, tools_observed, and computes maturity (LEARNING → MATURE after 10 observations).
3

Build topology

Create agent-to-agent edges (via tracectrl.caller.agent_id or parent span resolution) and agent-to-tool edges. Edges carry cumulative counts and confidence levels (LOW → MEDIUM → HIGH).
4

Ingest guardrail violations

Scan otel_traces for tracectrl.guardrail.evaluation spans with decision in ('fail', 'error') and insert into guardrail_violations. A 5-minute lookback off the latest observed_at keeps it from rescanning the whole table every tick.
5

Ingest guardrail registry

Scan otel_traces for tracectrl.guardrail.registered spans and upsert into guardrail_registry. A health-override pass flips health='error' for any guardrail with a decision=error violation in the last hour.
6

Attack graph + risk

Evaluate TAGAAI rules, build multi-hop attack paths, score risk, and write attack_paths, agent_risk_scores, system_risk.

​
TAGAAI Rules

The engine ships four detection rules. Rule names and base CVSS come from engine/rules/.
Rule (rule_name)OWASPBase CVSSTrigger
vulnerableToPromptInjectionASI017.2Agent has a tool edge with external call-context > 0 AND no human_interaction tool.
vulnerableToExcessiveAgencyASI028.1Agent has access to code_execution or email tools without a human_interaction guardrail.
vulnerableToDataLeakageASI01 + ASI026.8Agent chain where one reads from memory_read and another writes to external_api or email.
ingressToEndpointAttackPathASI01/ASI02 (impact-dependent)varies (5.0–9.0)Multi-hop ingress → high-impact endpoint chain. Base CVSS varies by impact category.

​
Guardrails Pipeline

The engine has two ingestion paths feeding the Alerts and Guardrails UI pages, both driven from spans the SDK already emits.

​
Registration ingestion

update_guardrail_registry() reads tracectrl.guardrail.registered spans and writes one row per (agent_id, guardrail_name) to guardrail_registry. The ReplacingMergeTree version column is last_seen_at, so re-emitting the same registration just refreshes the row. Spans carry: severity, mode (monitoring/blocking), timing (pre_input/post_output), judge_model, description, judge_prompt, health (active/error/disabled), and provider (judge_llm or protector_plus).

​
Violation ingestion

update_violations() reads tracectrl.guardrail.evaluation spans with decision in ('fail', 'error') and writes them to guardrail_violations. The violation_id is the eval span id, so the ReplacingMergeTree dedupes re-inserts. Each row carries reason, evidence, severity, judge_model, and provider.

​
ClickHouse tables (guardrails)

TableEnginePurpose
guardrail_registryReplacingMergeTree(last_seen_at) ORDER BY (agent_id, guardrail_name)One row per registered guardrail
guardrail_violationsReplacingMergeTree() ORDER BY (observed_at, violation_id)One row per failing/erroring evaluation
protector_plus_configReplacingMergeTree(updated_at) ORDER BY (id)Single global row: endpoint, API key, enabled guardrails

​
Configuration

VariableDefaultDescription
CLICKHOUSE_HOSTlocalhost (clickhouse in compose)ClickHouse host
CLICKHOUSE_PORT9000ClickHouse native port
CLICKHOUSE_DBtracectrlDatabase name
PIPELINE_INTERVAL_SECONDS60Pipeline cycle interval

​
Running Locally

cd tracectrl
source .venv/bin/activate
pip install -r engine/requirements.txt
uvicorn engine.main:app --reload --port 8000
When running locally, ensure ClickHouse is accessible at the configured host/port. The engine’s ensure_schema() will create the tracectrl database and all tables on startup.