Building Block View
Overview
The Agentic Layer architecture consists of the following building blocks that work together to provide AI orchestration capabilities:
-
Agent Runtime: The execution environment for AI agents, including the Agent Gateway for request routing and the Agent Runtime Operator for lifecycle management
-
AI Gateway: The abstraction layer for LLM provider interactions, providing unified access, security, and intelligent routing
-
Tool Gateway: The proxy layer for tool access, routing agent tool requests to internal and external tool servers via MCP
-
Observability: Real-time visualization and monitoring of agent interactions and system telemetry
-
Compliance / Audit: Audit records and compliance monitoring for governance requirements (planned)
-
Testbench: A testing and debugging environment for validating agent behavior
Overall Request Flow
The following diagram shows how these building blocks interact during typical request processing:
This flow demonstrates the request processing pipeline:
-
External systems (Frontends, Agents, and Apps) send requests via protocol-specific entry points (OpenAI Chat Completion API, A2A, AG-UI)
-
Agent Gateway receives and routes requests to appropriate agents
-
AI Agents process business logic, make LLM requests, and invoke tools
-
AI Gateway handles LLM provider interactions with security and routing
-
Tool Gateway routes agent tool requests to internal tool servers and external tool servers via MCP
-
Observability collects telemetry and provides real-time visualization
-
Testbench provides a testing environment that connects to the Agent Gateway
-
LLM Providers process AI requests and return results
Agent Runtime
The Agent Runtime building block provides the execution environment and management infrastructure for AI agents within the Kubernetes cluster.
Components
-
Agent Runtime Operator: Kubernetes operator that manages agent lifecycles, configurations, and deployments
-
Agent Gateway: API gateway that routes incoming requests to agents and maps external APIs to internal agent interfaces
-
AI Agents: Individual agent instances that execute business logic and orchestrate AI operations
Agent Runtime Responsibilities
Agent Runtime Operator serves as the control plane for agent management:
-
Registers and configures agents with the Agent Gateway
-
Manages agent lifecycles, scaling, and resource allocation
-
Provides Kubernetes-native deployment and operational patterns
Agent Gateway acts as the request entry point:
-
Routes requests to appropriate agents based on capabilities and load
-
Maps external APIs to internal agent interfaces via protocol-specific entry points (OpenAI Chat Completion API, A2A, AG-UI)
-
Provides load balancing and health checking for agent instances
Agentic Workforce groups the individual AI agent instances:
-
Agents process domain-specific workflows and business rules
-
Agents communicate with each other via the A2A (Agent-to-Agent) protocol for collaborative task execution
-
Each agent orchestrates interactions with external systems and services
-
Agents make intelligent decisions about when and how to use LLM capabilities and tools
AI Gateway
The AI Gateway building block abstracts interactions with multiple LLM providers, providing a unified interface with built-in security, monitoring, and intelligent routing capabilities.
AI Gateway Components and Flow
The AI Gateway processes requests through a secure, monitored pipeline:
Access Token Management handles authentication:
-
Manages API keys and authentication tokens for different LLM providers
-
Provides secure credential storage and rotation capabilities
-
Ensures proper authentication for all external AI service calls
AI Guardrails provides security and safety controls:
-
Content filtering and safety checks for both input and output
-
Policy enforcement based on organizational security requirements
-
Prevents malicious or inappropriate content from reaching LLM providers
Metrics component enables comprehensive monitoring:
-
Collects usage statistics, performance metrics, and cost tracking
-
Exports telemetry data to observability infrastructure
-
Provides insights into AI usage patterns and provider performance
Model Router manages intelligent LLM routing:
-
Routes requests to appropriate LLM providers based on capabilities, cost, and availability
-
Provides failover and load balancing across multiple providers
-
Supports cloud-based LLM providers as well as locally deployed language models
Tool Gateway
The Tool Gateway building block acts as a proxy layer that routes agent tool requests to internal and external tool servers via the MCP (Model Context Protocol).
Tool Gateway Responsibilities
Tool Router manages tool request routing:
-
Routes agent tool requests to the appropriate internal or external tool servers
-
Provides a unified interface for agents to access diverse tool capabilities
-
Manages connections to tool servers via the MCP protocol
Tool Servers provide capabilities that agents can invoke:
-
Internal tool servers run within the Kubernetes cluster and provide platform-managed tools
-
External tool servers run outside the platform and provide third-party tool integrations
-
All tool servers communicate via the standardized MCP protocol
Observability
The Observability building block provides real-time visualization and monitoring of agent interactions and system telemetry. It collects traces and metrics from all major components including the Agent Gateway, AI Gateway, and individual agents.