Create an AI Gateway

This guide walks you through deploying an instance of the AiGateway custom resource.

Before following this guide, make sure you have some kind of AiGateway Operator installed. See Install the LiteLLM Gateway Operator for installation instructions of one such operator.

Create API Key Secret

First, create a secret containing credentials for the LLM providers you plan to use:

kubectl create secret generic api-key-secrets \
  --namespace=ai-gateway \
  --from-literal=OPENAI_API_KEY=$OPENAI_API_KEY \
  --from-literal=GEMINI_API_KEY=$GEMINI_API_KEY \
  --from-literal=ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY
The secret name must match the value configured in the operator. The default expected name is api-key-secrets. API keys that are not provided will not cause deployment failures - the corresponding models just won’t be available.

Create an AiGateway Resource

  1. Create an AiGateway resource file:

    apiVersion: runtime.agentic-layer.ai/v1alpha1
kind: AiGateway
metadata:
  name: ai-gateway
  namespace: ai-gateway
spec:
  aiGatewayClassName: litellm
  aiModels:
    - provider: openai
      name: gpt-3.5-turbo
    - provider: gemini
      name: gemini-1.5-pro
spec.aiGatewayClassName is optional when only one implementation operator is installed.

  1. Apply the configuration:

    kubectl apply -f my-aigateway.yaml

Verify the Deployment

  1. Check the AiGateway status:

    kubectl get aigateways ai-gateway -o yaml

  2. Verify the created resources:

    # Check the deployment created by the operator
kubectl get deployments -l app=ai-gateway

# Check the service
kubectl get services -l app=ai-gateway

# Check the configmap with LiteLLM configuration
kubectl get configmaps ai-gateway-config

  3. Check the pod logs to ensure LiteLLM started successfully:

    kubectl logs -l app=ai-gateway -c litellm

Configure OpenTelemetry (Optional)

To send traces to an OpenTelemetry collector, configure the following environment variables:

  • OTEL_EXPORTER_OTLP_ENDPOINT: The endpoint URL of your OpenTelemetry collector

  • OTEL_EXPORTER_OTLP_PROTOCOL: Must be set to http/json (required when using OTLP endpoint)

apiVersion: runtime.agentic-layer.ai/v1alpha1
kind: AiGateway
metadata:
  name: ai-gateway
  namespace: ai-gateway
spec:
  aiGatewayClassName: litellm
  aiModels:
    - provider: openai
      name: gpt-3.5-turbo
    - provider: gemini
      name: gemini-1.5-pro
  env:
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
      value: http://otel-collector:4318
    - name: OTEL_EXPORTER_OTLP_PROTOCOL
      value: http/json

For more information on LiteLLM-specific OpenTelemetry configuration, see the LiteLLM OpenTelemetry documentation.

Apply Guardrails (Optional)

Attach Guard resources to the gateway to run content inspection (for example PII masking) on every request. The AiGateway CRD exposes spec.guardrails — an ordered list of references to Guard resources:

spec:
  guardrails:
    - name: pii-guard
      namespace: guards

See Create and Use Guardrails in Gateways for a full walkthrough and the Guardrails Reference for the Guard and GuardrailProvider schema.

Related