Building Block View

User-facing API: the Experiment CRD

Users do not author Testkube workflows directly. They submit an Experiment custom resource (testbench.agentic-layer.ai/v1alpha1) that declares the agent under test, the LLM judge, the dataset (inline, URL, or S3/MinIO), and optional triggers and schedules.

The Testbench operator (deployed in testbench-operator-system) watches Experiment resources and reconciles each one into:

  • A Testkube TestWorkflow that composes the five TestWorkflowTemplate resources installed in the testkube namespace

  • A ConfigMap holding the rendered Experiment JSON, mounted into the pipeline pods

  • Optionally a TestTrigger (when spec.trigger.enabled is true) that re-runs the workflow whenever the referenced agent is redeployed

Field-level details live in Reference.

Pipeline Overview

The reconciled TestWorkflow runs a sequential 5-phase pipeline. Each phase reads input from the previous phase’s output via a shared data volume.

pipeline-overview

Phase 1: Setup

Purpose: Materialises a test dataset on the shared data volume.

Processing:

  1. Resolves the dataset source from the Experiment: inline (mounted ConfigMap), HTTP URL, or S3/MinIO object

  2. Parses CSV, JSON, or Parquet input into the canonical Experiment JSON model

  3. Writes data/datasets/experiment.json to the shared data volume

Phase 2: Run

Purpose: Executes test queries against an agent via the A2A protocol and records responses.

Processing:

  1. Initializes OpenTelemetry tracing

  2. Loads the test dataset

  3. For each dataset entry:

    • Creates an OpenTelemetry span

    • Sends the query to the agent via A2A

    • Records the response and trace ID

Phase 3: Evaluate

Purpose: Calculates evaluation metrics using the LLM-as-a-judge approach.

Processing:

  1. Loads the experiment file

  2. Connects to the AI Gateway for LLM access

  3. Evaluates each row asynchronously

Phase 4: Publish

Purpose: Publishes per-sample evaluation metrics to an OTLP-compatible backend.

Processing:

  1. Loads the evaluation results

  2. Creates an OTLP metric exporter for HTTP transport

  3. For each sample and metric, creates a gauge observation with attributes:

    • name — metric type (e.g., "faithfulness")

    • workflow_name — test workflow identifier

    • execution_id — Testkube execution ID

    • execution_number — numeric execution counter

    • trace_id — links to the trace from the Run phase

    • sample_hash — unique sample identifier

    • user_input_truncated — first 50 characters of user input

  4. Flushes all metrics to ensure immediate export

Phase 5: Visualize

Purpose: Generates a self-contained HTML dashboard from evaluation results and stores it as a Testkube workflow artifact.

Features:

  • Summary cards — total samples, metrics count, token usage, cost

  • Workflow metadata header — workflow name, execution ID, execution number

  • Overall scores bar chart — horizontal bars showing mean score per metric

  • Metric distribution histograms — per-metric score distributions with min/max/mean/median statistics

  • Detailed results table — all samples with per-metric scores, searchable and color-coded

  • Multi-turn conversation visualization — chat-bubble layout with color-coded message types

  • Self-contained HTML — works offline as a single file