← /work
02EVALS · TERRA SECURITY

Intent Analysis System

A production analytics pipeline, built at Terra Security, that evaluates the company's AI pentesting agents from their conversation traces.

TYPEInternal system, Terra Security
STACKSentence-BERT · Claude tool use · LLM-as-judge
ROLEProduct & Data Intern
Intent Analysis dashboard showing intent and satisfaction distributions

Overview

Terra Security is an AI-driven cybersecurity startup. As part of a part-time product and data role, I built an internal system to understand how its AI pentesting agents were being used and where they fell short, using the agents' conversation traces as the source of truth.

What I built

  • Pre-scoring. Sentence-BERT embeddings to pre-score conversations.
  • Structured extraction. Claude, via the tool-use API, extracts structured fields from each trace.
  • Four analytics engines. Intent distribution, satisfaction, a CORE / NOISY / NICHE / REMOVE skill quadrant, and a skill-gap recommender.
  • Calibration. An LLM-as-judge step calibrated against human ratings.

Design choices

Sentence-BERT pre-scoring runs before the Claude extraction step, and the LLM judge is checked against human ratings before its scores are used.