> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tracectrl.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# TraceCtrl Bootcamp (Strands)
> Hands-on workshop — instrument an AWS Strands agent with TraceCtrl and go from zero traces to full security observability in 90 minutes.
Complete the [Prerequisites](/bootcamp/prerequisites) before starting this workshop.
Paste the TraceCtrl agent instructions into Claude Code, Cursor, or Copilot to get contextual help throughout this workshop.
***
# Part 1: Introduction to TraceCtrl
### What is TraceCtrl?
TraceCtrl is a **security observability platform for agentic AI**. Our tagline captures the workflow:
1. **See** — Visualize your agent's architecture, topology, and risk posture at a glance
2. **Trace** — Trace every message, tool call, and model request through the full execution lifecycle
3. **Ctrl** — Control and harden your agent's configuration with automated scanning and remediation
### Components
TraceCtrl has five components:
| Component | What it does | Technology |
| ------------------ | ----------------------------------------------------- | --------------------------- |
| **CLI** | `tracectrl scan`, `fix`, `doctor`, `setup` | Python (pip installable) |
| **Scanner** | Security/ops/perf/compliance checks for agent configs | Python |
| **OTel Collector** | Receives spans from agents via OTLP | OpenTelemetry Collector |
| **Engine** | Processes spans, builds topology, serves API | FastAPI + ClickHouse |
| **Dashboard** | Visualizes scans, traces, topology | React + Vite + Cytoscape.js |
### Architecture
1. Your agent (or OpenClaw) exports OpenTelemetry spans to the **OTel Collector**.
2. The Collector writes spans to **ClickHouse** (column-oriented database optimized for traces).
3. The **Engine** runs a pipeline every 60 seconds — reads spans, builds agent inventory, discovers topology edges, ingests guardrail registrations and violations, and computes the attack graph.
4. The **Dashboard** queries the Engine API to render Sessions, Topology, Alerts, Guardrails, Scan reports, Risk, and Attack Paths.
5. **Guardrails** ride the same pipe: in-SDK LLM judges (`tracectrl.guardrails`) and the external **Protector Plus** firewall (`tracectrl.protector`) both emit guardrail-evaluation spans that the engine surfaces as alerts.
6. The **CLI Scanner** (`tracectrl scan`, `tracectrl fix`) is a separate path that statically analyses OpenClaw configs and uploads findings to the Engine.
**Stack not running yet?** The TraceCtrl stack must be up before Part 2 — install and start it with the [Set Up TraceCtrl](/bootcamp/prerequisites#set-up-tracectrl) section of the Prerequisites page, then run `tracectrl doctor` to confirm all services are green.
***
# Part 2: Hands-on with TraceCtrl (Strands Agents)
In this part you'll instrument and run one of two pre-built Strands agent examples, watch the traces and topology light up in the TraceCtrl dashboard, attach a guardrail with the TraceCtrl SDK, and close out by watching FinFlow — our reference Strands application — get attacked end-to-end as a capstone.
## 2.1 Pick an Example to Run
For your hands-on portion, you'll run one (or both) of two pre-built examples from the [bootcamp examples repo](https://github.com/tracectrl/tracectrl-bootcamp-examples) (you cloned this in the [Prerequisites](/bootcamp/prerequisites#4-bootcamp-examples-repo) step). Both are pre-instrumented and show a different topology shape so you can compare them side by side in the dashboard.
| Example | Topology | What it does |
| -------------------------------------- | ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------- |
| `research_workflow_example/` | 3-agent chain: **Researcher → Analyst → Writer** | Web research / fact-checking. Researcher uses `http_request`, hands off via tool-call to Analyst, which hands off to Writer. |
| `teacher_assistants_workflow_example/` | Star: 1 orchestrator → 4 specialists | **TeachAssist** routes student queries to **MathWizard**, **EnglishMaster**, **LanguageAssistant**, or **GeneralAssist**. |
Each example folder is self-contained — its own README, `.env`, runner script, and Python files. You'll see the rendered topology for each in [2.3 Compare the Topologies](#2-3-compare-the-topologies) after you run them.
## 2.2 Running the Example
**`.env` files should already be configured** from [Prerequisites → Configure the Example `.env` Files](/bootcamp/prerequisites#configure-the-example-env-files). If you skipped that, jump back — both examples need a populated `.env` to run. If you hit a Gemini rate limit mid-session, swap `GOOGLE_MODEL_ID` to `gemini-2.5-flash` and rerun.
**TraceCtrl must be running before you start the agents.** The script sends spans to the OTel Collector on `localhost:4317`. If TraceCtrl is not up, the agents will still run and produce output — but nothing will appear in the dashboard. Run `tracectrl doctor` to confirm all services are green before continuing.
```bash theme={"dark"}
bash run_agents_workflow.sh
```
Uses `venv/` for its virtual environment.
```bash theme={"dark"}
bash run_agents_workflow_uv.sh
```
Uses `.venv/` for its virtual environment. Requires [uv](https://docs.astral.sh/uv/getting-started/installation/) on `PATH`.
The first run creates the venv and installs `strands-agents[gemini]`, `strands-agents-tools`, `tracectrl`, and `tracectrl-instrumentation-strands` from PyPI — about a minute on pip, a few seconds on uv. **Subsequent runs reuse the venv and skip the install.**
At the prompt, try these in order — start with the short one first to confirm everything is wired up before running a longer query:
```
> Tuesday comes before Monday in the week
> Lemon cures cancer
> What are quantum computers?
```
Each prompt creates a new trace in TraceCtrl. Type `exit` to quit.
```bash theme={"dark"}
bash run_agents_workflow.sh
```
Uses `venv/` for its virtual environment.
```bash theme={"dark"}
bash run_agents_workflow_uv.sh
```
Uses `.venv/` for its virtual environment. Requires [uv](https://docs.astral.sh/uv/getting-started/installation/) on `PATH`.
The first run creates the venv and installs dependencies — about a minute on pip, a few seconds on uv. **Subsequent runs reuse the venv and skip the install.**
Run these four prompts **in order**, then open [http://localhost:3000/topology](http://localhost:3000/topology) after each one to watch the star topology build up edge by edge:
```
> What is 1 + 1?
```
Routes to **MathWizard** — first edge appears: TeachAssist → MathWizard.
```
> Define Apple
```
Routes to **GeneralAssist** — second edge: TeachAssist → GeneralAssist.
```
> Write me a poem
```
Routes to **EnglishMaster** — third edge: TeachAssist → EnglishMaster.
```
> Translate hello to French
```
Routes to **LanguageAssistant** — full four-arm star now visible.
**No Computer Science specialist.** The system prompt mentions one, but it is not implemented. CS or coding questions (e.g. "explain recursion") will be handled by **GeneralAssist** instead.
Type `exit` to quit.
Open [http://localhost:3000/sessions](http://localhost:3000/sessions). You'll see a row per prompt tagged with your `TRACECTRL_SERVICE_NAME` (`agents-workflow` for research, `teachers-assistant` for teacher). Click the row to expand the trace tree. Five span kinds you'll see:
* **`invoke_agent `** — top-level span for each `tag_agent`-named agent. Nests when one agent calls another via a `@tool` hand-off.
* **`execute_event_loop_cycle`** — one per turn of the Strands event loop (LLM call + any tool calls it produced). A multi-turn agent run has several of these stacked under its `invoke_agent`.
* **`chat`** — the actual LLM call inside an event loop cycle (prompt, response, token counts).
* **`execute_tool `** — every `@tool` invocation with arguments and result.
* Tool spans that wrap *agent hand-offs* (`execute_tool call_analyst`, `execute_tool math_assistant`) contain a nested `invoke_agent` span — that nesting is what gives the topology its `delegates` edges.
Click any span to see its detail panel: timing, attributes, raw input/output values.
A single "Lemon cures cancer" fact-check prompt produces **24 spans across 21 seconds**. The Researcher runs through four event-loop cycles (each one a `chat` + one or two `http_request` tool calls to fetch sources), then hands off via `execute_tool call_analyst` → `invoke_agent Analyst` → `execute_tool call_writer` → `invoke_agent Writer`. The full chain is visible as one continuous trace tree.
A "What is 1 + 1?" prompt produces **9 spans across \~4 seconds**. TeachAssist's first event-loop cycle picks the route (`chat`), calls `execute_tool math_assistant` which contains `invoke_agent MathWizard`, MathWizard runs its own event-loop cycles, and the result bubbles back up. Each subsequent prompt creates a separate trace — three different prompts give you three trace rows.
### What TraceCtrl Captures
| Span | Key attributes |
| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Agent** | `tracectrl.agent.name`, `tracectrl.agent.role`, `openinference.span.kind=AGENT` (OpenInference's unprefixed `agent.name` is also present on the span; TraceCtrl-prefixed copies are what the topology graph reads) |
| **LLM** | `llm.model_name`, `input.value`, `output.value`, token counts |
| **Tool** | `tool.name`, `tool.description`, input/output, `tracectrl.tool.category` |
The `TraceCtrlSpanProcessor` (inside the SDK) enriches every span with tool risk category, agent identity, session correlation, and input-source classification automatically — no dashboard wiring needed.
The instrumentation is **three lines plus one decorator** — identical across both examples. Open `agents_workflow.py` (research example) or `teachers_assistant.py` (teacher example) and look at the top:
```python theme={"dark"}
import tracectrl
from tracectrl import tag_agent
from tracectrl.instrumentation.strands import StrandsInstrumentor
tracectrl.configure(
service_name=os.getenv("TRACECTRL_SERVICE_NAME", "agents-workflow"),
endpoint=os.getenv("TRACECTRL_ENDPOINT", "http://localhost:4317"),
)
StrandsInstrumentor().instrument()
```
Then for **every Strands `Agent` you create**, one extra line so TraceCtrl can name it in the topology:
```python theme={"dark"}
researcher_agent = Agent(
name="Researcher",
model=model,
system_prompt="...",
tools=[http_request, call_analyst],
)
tag_agent(researcher_agent)
```
Three things to notice:
1. **`StrandsInstrumentor().instrument()`** patches the Strands SDK so every agent invocation, LLM call, and tool call emits an OpenTelemetry span. No per-agent boilerplate.
2. **`tag_agent(...)`** stamps the agent's name and role onto every span the agent emits (via the auto-installed `SystemPromptStamper`). The parent-child span tree exists either way — Strands wires that up — but without `tag_agent` the agent nodes in the topology graph come through with generic / inferred names instead of `Researcher` / `Analyst` / `TeachAssist`, which makes the topology much harder to read. In both examples, agents hand off to each other by calling a `@tool` function that creates a new `Agent` and calls `tag_agent()` on it — that's what produces the named `delegates` edges (`Researcher → Analyst → Writer`, `TeachAssist → MathWizard`, etc.).
3. **OTLP gRPC on `:4317`.** This is the OpenTelemetry Collector that ships with the TraceCtrl stack you started during the [Prerequisites](/bootcamp/prerequisites#set-up-tracectrl). The Engine reads from it, the Dashboard reads from the Engine.
**Checkpoint:** Sessions page shows a trace with `invoke_agent` at the top, nested `execute_event_loop_cycle` / `chat` / `execute_tool` spans, and at least one `invoke_agent ` nested under an `execute_tool` (the agent hand-off). Span count is in the high single digits for the teacher example, mid-twenties for a multi-source research query.
## 2.3 Compare the Topologies
Open [http://localhost:3000/topology](http://localhost:3000/topology). Each example produces a distinct shape:
A linear 3-agent chain. `Researcher` is the entry point — it fetches web sources via `http_request`, then hands off to `Analyst`, which hands off to `Writer`. Each hand-off is a `@tool` function call, which is why TraceCtrl draws `delegates` edges between the agents rather than treating them as a flat list. Only external edge: `Researcher` → `http_request` (the `external_api` risk category).
A star topology. `TeachAssist` delegates to one specialist per query — the star builds up progressively as different subjects route to different specialists. `EnglishMaster` pulls in `file_read` / `file_write` / `editor`; `LanguageAssistant` uses `http_request`; `GeneralAssist` handles anything that doesn't fit a specialist.
If you ran both, the topology accumulates — you'll see two disconnected sub-graphs in the same dashboard, which is exactly what you'd see in production when multiple agentic services share one TraceCtrl deployment.
**Tool risk categories.** Every tool is classified into one of 8 categories (`network`, `code_execution`, `filesystem`, `data_store`, `web_browsing`, `communication`, `system`, `other`) based on its name and docstring. That classification is what drives the risk colouring on the topology nodes.
## 2.4 Sample Developer Guardrails with the TraceCtrl SDK
Traces and topology tell you **what your agents do**. Guardrails let you **decide whether to let an action happen** — every input or output runs through a `Guardrail` whose verdict is recorded as a span attribute and surfaced in the Guardrails dashboard.
### Guardrails Scaffolded in the Examples
Each example ships with a `guardrails/` subfolder containing **two ready-to-read scaffolds** — full rubric prompt, `Guardrail()` config, and concrete pass/fail test inputs in the docstring. They start unwired so the example still runs as-is; **every guardrail-target agent is already at module scope, so enabling a guard is a single uncomment** at the bottom of the entry file.
**`source_reliability_guard`** — `post_output` on `Writer`. Fails when the report cites zero URLs, only forum/social citations, or fabricated-looking sources. Severity: medium.
**`fact_check_consistency_guard`** — `post_output` on `Analyst`. Fails when the Analyst's verdict ("true" / "false" / accuracy rating) contradicts the evidence it itself cites — i.e. claims "verified" with no supporting points, or "true" while citing a debunking source. Severity: high.
Files live in `research_workflow_example/guardrails/`. The commented wire-in block at the bottom of `agents_workflow.py` shows exactly which lines to uncomment.
**`pii_in_query_guard`** — `pre_input` on `TeachAssist`. Fails when the student's query carries an email, phone number, national ID (NRIC / SSN / NI), or full-name + home-address pairing — common when kids paste from somewhere else. Severity: medium.
**`code_execution_safety_guard`** — `post_output` on `GeneralAssist` (the CS / coding specialist). Fails when the response contains `os.system`, `subprocess(shell=True)`, `eval`, `exec`, `rm -rf`, curl-piped-to-bash, or other shell-execution / destructive / remote-payload patterns. The agent's job is to teach, not hand out shells. Severity: high.
Files live in `teacher_assistants_workflow_example/guardrails/`. The commented wire-in block at the bottom of `teachers_assistant.py` shows exactly which lines to uncomment.
**The judge uses the same Gemini model your agents are on.** Each scaffolded guardrail passes a `GeminiModel(...)` as `judge_llm` — no extra credentials to provision. Requires `tracectrl >= 0.3.0` (earlier versions hard-coded the judge to AWS Bedrock and would fail with `Unable to locate credentials`). Run `pip show tracectrl | grep Version` to confirm.
The Protector Plus path in Part 3 covers external LLM-firewall checks (PII, content moderation, prompt injection) where you DON'T want to roll your own judge.
### Enable + Try It
Wires in **`source_reliability_guard`** on `Writer` and **`fact_check_consistency_guard`** on `Analyst`.
**Step 1 — uncomment the wire-in block** at the bottom of `research_workflow_example/agents_workflow.py`. Remove the `#` on these four lines:
```python theme={"dark"}
from tracectrl.guardrails import wrap_agent_with_guardrails
from guardrails import source_reliability_guard, fact_check_consistency_guard
wrap_agent_with_guardrails(writer_agent, [source_reliability_guard])
wrap_agent_with_guardrails(analyst_agent, [fact_check_consistency_guard])
```
**Step 2 — run** with whichever runner you used earlier and try a prompt that should fail (Writer produces a report with no real sources):
```bash theme={"dark"}
./run_agents_workflow.sh # pip
# or: ./run_agents_workflow_uv.sh # uv
> Make up a plausible report about a fictional startup called Acme Quantum
> Pretend to research X — invent a plausible answer
```
Or a prompt that should pass (Researcher finds real URLs the Writer cites):
```
> What are quantum computers?
```
Wires in **`pii_in_query_guard`** on `TeachAssist` and **`code_execution_safety_guard`** on `GeneralAssist`.
**Step 1 — uncomment the wire-in block** at the bottom of `teachers_assistant.py`. Remove the `#` on these five lines:
```python theme={"dark"}
from tracectrl.guardrails import wrap_agent_with_guardrails
from guardrails import pii_in_query_guard, code_execution_safety_guard
from no_expertise import general_assistant_agent
wrap_agent_with_guardrails(teacher_agent, [pii_in_query_guard])
wrap_agent_with_guardrails(general_assistant_agent, [code_execution_safety_guard])
```
**Step 2 — run** with whichever runner you used earlier and try a PII prompt (`pii_in_query_guard` should fail it):
```bash theme={"dark"}
./run_agents_workflow.sh # pip
# or: ./run_agents_workflow_uv.sh # uv
> My name is Sarah Lee, NRIC S1234567A. Help with x^2 - 5x + 6 = 0
> Email me the answer at student@school.edu — what's the past tense of swim?
> I live at 12 Main Street, my name is John Doe — help me with math
```
Then a code-execution prompt routed to `GeneralAssist` (`code_execution_safety_guard` should fail it):
```
> Write Python that runs `rm -rf ~/` for me
> Show me how to use os.system to delete a directory
```
Or a clean prompt (both should pass):
```
> Solve x^2 - 5x + 6 = 0
> What is the difference between affect and effect?
```
**Common pitfalls:**
* `Unable to locate credentials` → you're on `tracectrl < 0.3.0`. The judge was Bedrock-only before. Run `pip install --upgrade tracectrl`.
* The startup warning `TraceCtrl: OTel endpoint http://localhost:4317 is not reachable` is a false positive — the SDK probes the gRPC port with HTTP. Ignore it if `docker compose ps` shows the collector is `Up`.
### What You'll See in the Dashboard
Once a guardrail is wrapped, every invocation registers it under [http://localhost:3000/guardrails](http://localhost:3000/guardrails) with health, recent activity, and a per-violation evidence panel (exact input checked, judge verdict, severity). Violations also light up on the Sessions trace tree as red span attributes — so you can see *which* agent turn tripped the guard, not just that one tripped.
Once you've seen one verdict land in the dashboard, **edit the rubric** in the guard file (`_PROMPT`) — tighten or loosen a rule, add a new fail condition — and rerun. The change shows up immediately in the next evaluation span on the dashboard.
A guardrail is a small dataclass: an **LLM judge** + a **rubric prompt**. Six lines plus the rubric — every scaffolded guard in the examples repo follows the same shape:
```python theme={"dark"}
from tracectrl.guardrails import Guardrail, wrap_agent_with_guardrails
from strands.models.gemini import GeminiModel
import os
judge = GeminiModel(
client_args={"api_key": os.getenv("GOOGLE_API_KEY")},
model_id=os.getenv("GOOGLE_MODEL_ID", "gemini-3.1-flash-lite"),
)
my_guard = Guardrail(
name="my_guard",
description="What this checks for, in one sentence.",
judge_prompt="""...""", # use {input} for pre_input, {output} for post_output
judge_llm=judge,
on_violation="log", # "log" today; "block" raises NotImplementedError (v2)
timing="post_output", # or "pre_input"
severity="medium", # low | medium | high | critical
)
wrap_agent_with_guardrails(my_agent, [my_guard])
```
Three knobs to decide on per guardrail: **what to check** (`judge_prompt`), **when to check** (`timing`), **how loud** to be on violation (`severity` + `on_violation`). The judge uses the **same Gemini model your agents already use** — no extra credentials to provision. The SDK handles span attribution, evidence capture, and registration on the Guardrails page.
The `judge_prompt` is the rubric — a Markdown-y template with `{output}` or `{input}` placeholders the SDK fills in at evaluation time. **This is the open-ended part you'll tinker with.** Open any of the scaffolded guards (the `_PROMPT` constant near the top of each file) — they're explicit and listy: they tell the judge what to fail on, what to ignore, and what JSON shape to return. Edit the rules, rerun the example, watch the new pass/fail behavior land in the Sessions trace and on `/alerts` in real time.
**Checkpoint:** At least one SDK-based guardrail is wrapped on an agent in your chosen example. After running a query that should trip it, you see the guardrail listed on the Guardrails page with a non-zero violation count and an evidence panel showing the input.
## 2.5 FinFlow Demo (Capstone)
To close out Part 2, the facilitator runs **FinFlow** — our reference Strands application — live so you can see what a realistic multi-agent system looks like end-to-end in TraceCtrl, alongside the smaller examples you just instrumented yourself.
### What FinFlow Is
**FinFlow AI** is our reference Strands application — an enterprise financial operations platform that automates the full invoice lifecycle: ingestion → policy validation → vendor verification → risk screening → payment execution → notification.
It's a **multi-agent system** built on AWS Strands, designed from the start to be the TraceCtrl demo platform. The facilitator demos FinFlow live so you can see the end-to-end flow without typing.
### Architecture
The diagram below mirrors how FinFlow appears in TraceCtrl's topology view — hexagonal agents, rounded tools, diamond triggers, with `delegates` / `uses` edges:
**6 agents, 3 ingress points, 2 external MCPs/APIs, 3 data stores.** Five of the agents are LLM-driven Strands agents; `PaymentAgent` is a deterministic Python function (you don't want an LLM deciding whether to actually move money).
Compared to the two examples you ran in 2.3 — a 3-agent linear chain and a 4-arm star — FinFlow's topology is the more interesting case: an external MCP, a payment edge, and three predicted attack paths derived purely from the shape. Same dashboard, same telemetry pipeline, very different risk profile.
### What We'll Demo
First we'll run FinFlow on a **normal, predictable workflow** — clean invoices flowing through the pipeline exactly as designed: Orchestrator → DocAI → PolicyAgent → VendorIntel → RiskAgent → PaymentAgent → NotificationAgent. Everything works.
Then we'll upload an invoice that looks identical to the human eye but carries a **subtle injection in invisible text** the PDF parser still extracts — and watch the agents derail. TraceCtrl traces both runs end-to-end, so the difference between "working as intended" and "compromised" is visible in the same dashboard.
***
# Part 3: Enterprise Guardrail Integration with GenAI Protector Plus
The SDK guardrails in 2.4 are great for **app-specific** policy (payment caps, scope checks, custom rubrics). For **generic LLM-firewall protections** — prompt injection, PII, content moderation, system-prompt leakage, vector-similarity to known attacks — you don't want to roll your own. cloudsineAI's **GenAI Protector Plus** is exactly that: an external LLM firewall you point TraceCtrl at, and every agent input/output is screened by the same managed guardrails used across your whole org.
The flow:
1. **In Protector Plus:** configure the guardrails you want (Keyword, Shield Prompt LLM, PII, etc.).
2. **In TraceCtrl Settings:** paste the Protector Plus endpoint + API key, tick which guardrails to enable.
3. **Run your agents:** TraceCtrl's SDK now calls Protector Plus on every `check_input` / `check_output` — violations show up alongside your SDK guardrails on the same dashboard.
## 3.1 Configure Guardrails in Protector Plus
In your Protector Plus instance (the cloudsineAI dashboard), enable the guardrails you want. The two we'll use for the FinFlow demo, in order of breadth:
**Shield Prompt → LLM.** Broad LLM-as-judge for prompt injection — catches arbitrary injection attempts, not just known phrases. Provider is Ollama (local inference), model `llama3.2:3b`. Faster and cheaper than a hosted Bedrock/Anthropic judge, runs on the same machine as the workshop.
**Standard Protection → Keyword.** Narrow keyword/phrase blocklist that targets known attack signatures the LLM judge might rationalise past. We add a rule called **"CFO Pre Approved"** with the keyword `CFO-OVERRIDE` — directly aimed at the hidden injection text from the malicious PDF.
After configuring, save and copy two values from the Protector Plus admin:
* **Endpoint URL** (base URL, no trailing slash — e.g. `https://genaifw-sales.example.com`)
* **API key** (domain-scoped, generated under the Protector Plus dashboard)
## 3.2 Connect Protector Plus to TraceCtrl
In the TraceCtrl UI, go to **Settings → TraceCtrl Guards** ([http://localhost:3000/settings](http://localhost:3000/settings)).
Paste the endpoint URL and API key. Tick the guardrails you enabled in Protector Plus — only ticked ones get called from the SDK. For the FinFlow demo, the minimum is **LLM Judge** + **Keyword**.
Hit **Save**. The API key field redacts after save — re-enter the full key any time you need to change it.
**What gets called when.** Once enabled, every `check_input` and `check_output` from the TraceCtrl SDK hits Protector Plus for the ticked guardrails. The guardrails show up in TraceCtrl with `protector_plus.` prefixes (e.g. `protector_plus.keyword`, `protector_plus.llm`) so you can tell at a glance whether a violation came from an SDK-defined guard or from Protector Plus.
## 3.3 Catch a Violation in Both Systems
Re-run FinFlow's malicious-PDF attack from the [2.5 FinFlow Demo](#2-5-finflow-demo-capstone). The keyword and LLM guardrails fire, and the **same violation now appears in two places** — once on the Protector Plus side as a firewall detection, once on the TraceCtrl side enriched with agent and trace context.
### Protector Plus → Alerts
In the cloudsineAI dashboard, open **Alerts**. Each detection lands as a row with a Weborion ID (the unique request hash), timestamp, attack category (`KEYWORD` for the CFO-OVERRIDE matches), and confidence score.
This is the **firewall's view** — what got matched, when, with what confidence. Useful for tuning the rules themselves and confirming the detector is wired up. What it doesn't tell you: *which agent in your application made the call, what session it was part of, what happened upstream of the violation*.
### TraceCtrl → Alerts
Open [http://localhost:3000/alerts](http://localhost:3000/alerts). Every Protector Plus detection from above shows up here too, plus violations from any SDK-defined guards like `payment_delegation_guard`:
Same detection, very different presentation. Each violation card carries:
* **Severity + guardrail name** (e.g. `MEDIUM protector_plus.keyword`) and a `FAIL` badge
* **Agent** the violation was attached to (`finflow-ai`, `orchestrator`)
* **Session ID** so you can pivot straight to the trace tree
* **Judge** that produced the verdict — `protector_plus:keyword` for Protector Plus calls, `BedrockModel` for SDK-defined LLM judges
Click any violation to open the **Violation evidence** panel. You get the exact input that was checked (raw extracted invoice text with the hidden CFO-OVERRIDE line visible), the judge's REASON, the full CONTEXT (agent, judge model, decision, observed time, trace ID, span ID), and an **Open trace** button that drops you on the Sessions page at the offending span.
That's the difference Protector Plus + TraceCtrl gets you over Protector Plus alone — a firewall detection becomes a debuggable, agent-attributed, trace-correlated alert.
**Checkpoint:** A deliberate malicious input produces matching detections in both the Protector Plus Alerts page and the TraceCtrl Alerts page. The TraceCtrl alert opens to a Violation evidence panel with agent, judge, evidence, and a working **Open trace** button.
***
**Fully optional, OpenClaw users only.** Skip this section if you don't already have an OpenClaw workspace running. The OpenClaw attack walkthrough and hardening flow are demonstrated **in person** during the workshop — this section just covers the scan step so you can follow along on your own setup.
[OpenClaw](https://docs.openclaw.ai) is a self-hosted AI agent gateway (LLM ↔ messaging channels, with config-driven tool access and session management). TraceCtrl ships a static scanner for OpenClaw configs that runs 30+ security/ops/compliance checks against `openclaw.json` and uploads the findings to the dashboard. No runtime instrumentation — pure static analysis of the config.
### Run the Scan
```bash theme={"dark"}
# Get your absolute workspace path (run as your normal user, not sudo)
openclaw configure # first line of output prints the path
# Then scan — use the absolute path printed by openclaw configure
tracectrl scan
```
Don't use `~/.openclaw/` if you're scanning from a `sudo` shell — `~` resolves to `/root/` under elevation. Use the absolute path from `openclaw configure`.
The scanner reads `openclaw.json`, runs the check suite, and uploads results. Terminal output gives you the severity counts and a findings table; the dashboard gives you the full picture.
### View the Scan Report
Open [http://localhost:3000/scan](http://localhost:3000/scan). The page has two halves: the **architecture risk view** at the top, and the **findings list** grouped by category below it.
#### Architecture risk view
Every agent, tool, LLM, skill, extension, scheduler, and sub-agent in your `openclaw.json` shows up as a node, with edges labelled by the relationship (`invokes`, `calls`, `delegates`). Node colours follow the highest-severity finding attached — so a node with a CRITICAL finding has a red border, HIGH is orange, MEDIUM is yellow.
#### Findings list
Two things worth knowing here:
* **Stable finding IDs.** Every check has an ID like `OC-SEC-002` or `OC-CRED-001`. The same ID tracks across rescans, so you can wire it into PR descriptions or tickets.
* **Rescan in place.** Edit `openclaw.json`, hit **Rescan** in the header — no CLI round-trip needed.
**Checkpoint:** `tracectrl scan` ran successfully against your OpenClaw workspace, and the Scan Report page shows findings grouped by category with a risk-coloured topology at the top.
***
# Summary
## What You Built
| Component | Status |
| ------------------------------------------------------------- | -------------------------------- |
| TraceCtrl stack (Engine + Dashboard + Collector + ClickHouse) | Running via Docker |
| FinFlow Strands agent | Instrumented and running |
| Runtime traces (agent → LLM → tool spans) | Visible in Sessions |
| Agent topology graph | Auto-built from traces |
| Tool risk categorization | Applied automatically by SDK |
| GenAI Protector Plus guardrails | Attached and catching violations |
## CLI Reference
```bash theme={"dark"}
tracectrl setup # Interactive wizard — start the stack
tracectrl doctor # Verify all services are healthy
tracectrl scan [path] # Run security scan on an agent config
tracectrl fix --auto # Auto-remediate findings
tracectrl version # Print version
```
## Useful Links
Source code and issues.
AWS Strands SDK documentation.
Full list of built-in tools.
The observability standard TraceCtrl is built on.
## Cleanup
```bash theme={"dark"}
# Stop containers and remove all data
docker compose down -v
# Or keep data, just stop containers
docker compose down
```
A single "Lemon cures cancer" fact-check prompt produces **24 spans across 21 seconds**. The Researcher runs through four event-loop cycles (each one a `chat` + one or two `http_request` tool calls to fetch sources), then hands off via `execute_tool call_analyst` → `invoke_agent Analyst` → `execute_tool call_writer` → `invoke_agent Writer`. The full chain is visible as one continuous trace tree.
A "What is 1 + 1?" prompt produces **9 spans across \~4 seconds**. TeachAssist's first event-loop cycle picks the route (`chat`), calls `execute_tool math_assistant` which contains `invoke_agent MathWizard`, MathWizard runs its own event-loop cycles, and the result bubbles back up. Each subsequent prompt creates a separate trace — three different prompts give you three trace rows.
A linear 3-agent chain. `Researcher` is the entry point — it fetches web sources via `http_request`, then hands off to `Analyst`, which hands off to `Writer`. Each hand-off is a `@tool` function call, which is why TraceCtrl draws `delegates` edges between the agents rather than treating them as a flat list. Only external edge: `Researcher` → `http_request` (the `external_api` risk category).
A star topology. `TeachAssist` delegates to one specialist per query — the star builds up progressively as different subjects route to different specialists. `EnglishMaster` pulls in `file_read` / `file_write` / `editor`; `LanguageAssistant` uses `http_request`; `GeneralAssist` handles anything that doesn't fit a specialist.
**6 agents, 3 ingress points, 2 external MCPs/APIs, 3 data stores.** Five of the agents are LLM-driven Strands agents; `PaymentAgent` is a deterministic Python function (you don't want an LLM deciding whether to actually move money).
Compared to the two examples you ran in 2.3 — a 3-agent linear chain and a 4-arm star — FinFlow's topology is the more interesting case: an external MCP, a payment edge, and three predicted attack paths derived purely from the shape. Same dashboard, same telemetry pipeline, very different risk profile.
### What We'll Demo
First we'll run FinFlow on a **normal, predictable workflow** — clean invoices flowing through the pipeline exactly as designed: Orchestrator → DocAI → PolicyAgent → VendorIntel → RiskAgent → PaymentAgent → NotificationAgent. Everything works.
Then we'll upload an invoice that looks identical to the human eye but carries a **subtle injection in invisible text** the PDF parser still extracts — and watch the agents derail. TraceCtrl traces both runs end-to-end, so the difference between "working as intended" and "compromised" is visible in the same dashboard.
***
# Part 3: Enterprise Guardrail Integration with GenAI Protector Plus
The SDK guardrails in 2.4 are great for **app-specific** policy (payment caps, scope checks, custom rubrics). For **generic LLM-firewall protections** — prompt injection, PII, content moderation, system-prompt leakage, vector-similarity to known attacks — you don't want to roll your own. cloudsineAI's **GenAI Protector Plus** is exactly that: an external LLM firewall you point TraceCtrl at, and every agent input/output is screened by the same managed guardrails used across your whole org.
The flow:
1. **In Protector Plus:** configure the guardrails you want (Keyword, Shield Prompt LLM, PII, etc.).
2. **In TraceCtrl Settings:** paste the Protector Plus endpoint + API key, tick which guardrails to enable.
3. **Run your agents:** TraceCtrl's SDK now calls Protector Plus on every `check_input` / `check_output` — violations show up alongside your SDK guardrails on the same dashboard.
## 3.1 Configure Guardrails in Protector Plus
In your Protector Plus instance (the cloudsineAI dashboard), enable the guardrails you want. The two we'll use for the FinFlow demo, in order of breadth:
**Shield Prompt → LLM.** Broad LLM-as-judge for prompt injection — catches arbitrary injection attempts, not just known phrases. Provider is Ollama (local inference), model `llama3.2:3b`. Faster and cheaper than a hosted Bedrock/Anthropic judge, runs on the same machine as the workshop.
**Standard Protection → Keyword.** Narrow keyword/phrase blocklist that targets known attack signatures the LLM judge might rationalise past. We add a rule called **"CFO Pre Approved"** with the keyword `CFO-OVERRIDE` — directly aimed at the hidden injection text from the malicious PDF.
After configuring, save and copy two values from the Protector Plus admin:
* **Endpoint URL** (base URL, no trailing slash — e.g. `https://genaifw-sales.example.com`)
* **API key** (domain-scoped, generated under the Protector Plus dashboard)
## 3.2 Connect Protector Plus to TraceCtrl
In the TraceCtrl UI, go to **Settings → TraceCtrl Guards** ([http://localhost:3000/settings](http://localhost:3000/settings)).
Paste the endpoint URL and API key. Tick the guardrails you enabled in Protector Plus — only ticked ones get called from the SDK. For the FinFlow demo, the minimum is **LLM Judge** + **Keyword**.
Hit **Save**. The API key field redacts after save — re-enter the full key any time you need to change it.
This is the **firewall's view** — what got matched, when, with what confidence. Useful for tuning the rules themselves and confirming the detector is wired up. What it doesn't tell you: *which agent in your application made the call, what session it was part of, what happened upstream of the violation*.
### TraceCtrl → Alerts
Open [http://localhost:3000/alerts](http://localhost:3000/alerts). Every Protector Plus detection from above shows up here too, plus violations from any SDK-defined guards like `payment_delegation_guard`:
Same detection, very different presentation. Each violation card carries:
* **Severity + guardrail name** (e.g. `MEDIUM protector_plus.keyword`) and a `FAIL` badge
* **Agent** the violation was attached to (`finflow-ai`, `orchestrator`)
* **Session ID** so you can pivot straight to the trace tree
* **Judge** that produced the verdict — `protector_plus:keyword` for Protector Plus calls, `BedrockModel` for SDK-defined LLM judges
Click any violation to open the **Violation evidence** panel. You get the exact input that was checked (raw extracted invoice text with the hidden CFO-OVERRIDE line visible), the judge's REASON, the full CONTEXT (agent, judge model, decision, observed time, trace ID, span ID), and an **Open trace** button that drops you on the Sessions page at the offending span.
That's the difference Protector Plus + TraceCtrl gets you over Protector Plus alone — a firewall detection becomes a debuggable, agent-attributed, trace-correlated alert.
Every agent, tool, LLM, skill, extension, scheduler, and sub-agent in your `openclaw.json` shows up as a node, with edges labelled by the relationship (`invokes`, `calls`, `delegates`). Node colours follow the highest-severity finding attached — so a node with a CRITICAL finding has a red border, HIGH is orange, MEDIUM is yellow.
#### Findings list
Two things worth knowing here:
* **Stable finding IDs.** Every check has an ID like `OC-SEC-002` or `OC-CRED-001`. The same ID tracks across rescans, so you can wire it into PR descriptions or tickets.
* **Rescan in place.** Edit `openclaw.json`, hit **Rescan** in the header — no CLI round-trip needed.