Skip to main content

​
Overview

The TraceCtrl Engine is a FastAPI application that:
  1. Runs a background pipeline every PIPELINE_INTERVAL_SECONDS to process spans.
  2. Serves a REST API under /api/v1 for the dashboard and external integrations.
  3. Maintains cumulative state in ClickHouse using ReplacingMergeTree tables.

​
Pipeline

Scheduled via APScheduler at the interval set by PIPELINE_INTERVAL_SECONDS (default: 60). Each tick re-reads all spans — there is no watermark on the main path; ReplacingMergeTree handles deduplication via background merges.
1

Fetch spans

Read every span from otel_traces (via fetch_all_spans()).
2

Update agent inventory

Group spans by tracectrl.agent.id and upsert agent records. Tracks cumulative observation_count, run_count, tools_observed, and computes maturity (LEARNING → MATURE after 10 observations).
3

Build topology

Create agent-to-agent edges (via tracectrl.caller.agent_id or parent span resolution) and agent-to-tool edges. Edges carry cumulative counts and confidence levels (LOW → MEDIUM → HIGH).
4

Ingest guardrail violations

Scan otel_traces for tracectrl.guardrail.evaluation spans with decision in ('fail', 'error') and insert into guardrail_violations. A 5-minute lookback off the latest observed_at keeps it from rescanning the whole table every tick.
5

Ingest guardrail registry

Scan otel_traces for tracectrl.guardrail.registered spans and upsert into guardrail_registry. A health-override pass flips health='error' for any guardrail with a decision=error violation in the last hour.
6

Attack graph + risk

Evaluate TAGAAI rules, build multi-hop attack paths, score risk, and write attack_paths, agent_risk_scores, system_risk.

​
TAGAAI Rules

The engine ships four detection rules. Rule names and base CVSS come from engine/rules/.

​
Guardrails Pipeline

The engine has two ingestion paths feeding the Alerts and Guardrails UI pages, both driven from spans the SDK already emits.

​
Registration ingestion

update_guardrail_registry() reads tracectrl.guardrail.registered spans and writes one row per (agent_id, guardrail_name) to guardrail_registry. The ReplacingMergeTree version column is last_seen_at, so re-emitting the same registration just refreshes the row. Spans carry: severity, mode (monitoring/blocking), timing (pre_input/post_output), judge_model, description, judge_prompt, health (active/error/disabled), and provider (judge_llm or protector_plus).

​
Violation ingestion

update_violations() reads tracectrl.guardrail.evaluation spans with decision in ('fail', 'error') and writes them to guardrail_violations. The violation_id is the eval span id, so the ReplacingMergeTree dedupes re-inserts. Each row carries reason, evidence, severity, judge_model, and provider.

​
ClickHouse tables (guardrails)

​
Configuration

​
Running Locally

When running locally, ensure ClickHouse is accessible at the configured host/port. The engine’s ensure_schema() will create the tracectrl database and all tables on startup.