Create an AI Gateway

This guide walks you through deploying an instance of the AiGateway custom resource.

An AiGateway is like a Kubernetes Gateway in that there might be many different implementations (also known as "classes"). Example implementations of the Kubernetes Gateway specification include Envoy Gateway, Cilium, or amazon-vpc-lattice).

One example implementation of the Agentic Layer AiGateway specification is based on LiteLLM. This guide explains how to instantiate one such instance of an AiGateway.

Before following this guide, make sure you have some kind of AiGateway Operator installed. See Install the AI Gateway LiteLLM Operator for installation instructions of one such operator.

Create API Key Secret

First, create a secret containing credentials for the LLM providers you plan to use:

kubectl create secret generic api-key-secrets \
  --namespace=ai-gateway \
  --from-literal=OPENAI_API_KEY=$OPENAI_API_KEY \
  --from-literal=GEMINI_API_KEY=$GEMINI_API_KEY \
  --from-literal=ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY

The secret name must match the value configured in the operator. The default expected name is api-key-secrets. API keys that are not provided will not cause deployment failures - the corresponding models just won’t be available.

Create an AiGateway Resource

Create a AiGateway resource file:

apiVersion: runtime.agentic-layer.ai/v1alpha1
kind: AiGateway
metadata:
  name: ai-gateway
  namespace: ai-gateway
spec:
  AiGatewayClassName: litellm
  aiModels:
    - provider: openai
      name: gpt-3.5-turbo
    - provider: gemini
      name: gemini-1.5-pro

Note: The AiGatewayClassName is optional.

Apply the configuration:
```
kubectl apply -f my-aigateway.yaml
```

Verify the Deployment

Check the AiGateway status:

kubectl get aigateways ai-gateway -o yaml

Verify the created resources:

# Check the deployment created by the operator
kubectl get deployments -l app=ai-gateway

# Check the service
kubectl get services -l app=ai-gateway

# Check the configmap with LiteLLM configuration
kubectl get configmaps ai-gateway-config

Check the pod logs to ensure LiteLLM started successfully:
```
kubectl logs -l app=ai-gateway -c litellm
```

Configure OpenTelemetry (Optional)

To send traces to an OpenTelemetry collector, configure the following environment variables:

OTEL_EXPORTER_OTLP_ENDPOINT: The endpoint URL of your OpenTelemetry collector
OTEL_EXPORTER_OTLP_PROTOCOL: Must be set to http/json (required when using OTLP endpoint)

apiVersion: runtime.agentic-layer.ai/v1alpha1
kind: AiGateway
metadata:
  name: ai-gateway
  namespace: ai-gateway
spec:
  AiGatewayClassName: litellm
  aiModels:
    - provider: openai
      name: gpt-3.5-turbo
    - provider: gemini
      name: gemini-1.5-pro
  env:
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
      value: http://otel-collector:4318
    - name: OTEL_EXPORTER_OTLP_PROTOCOL
      value: http/json

For more information on LiteLLM-specific OpenTelemetry configuration, see the LiteLLM OpenTelemetry documentation.