LibreChat

Deploy LibreChat in Kubernetes and route its LLM traffic through agentgateway.

Deploy LibreChat in Kubernetes and route its LLM traffic through agentgateway to centralize credentials and apply policies across all chats.

Before you begin

Set up an agentgateway proxy.
Set up access to the OpenAI LLM provider.

Get the gateway URL

export INGRESS_GW_ADDRESS=$(kubectl get svc -n agentgateway-system agentgateway-proxy \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

echo "Gateway address: $INGRESS_GW_ADDRESS"

After port-forwarding, the gateway is accessible at http://localhost:8080. Use localhost:8080 wherever the instructions reference $INGRESS_GW_ADDRESS.

kubectl port-forward -n agentgateway-system svc/agentgateway-proxy 8080:80

Set up the OpenAI backend

Export your OpenAI API key.

export OPENAI_API_KEY="sk-your-key-here"

Create a Kubernetes Secret for your API key.

kubectl apply -f- <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: openai-secret
  namespace: agentgateway-system
type: Opaque
stringData:
  Authorization: $OPENAI_API_KEY
EOF

Create an AgentgatewayBackend for OpenAI.

kubectl apply -f- <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: openai
  namespace: agentgateway-system
spec:
  ai:
    provider:
      openai: {}
  policies:
    auth:
      secretRef:
        name: openai-secret
EOF

Create an HTTPRoute to forward traffic to the backend.

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: openai
  namespace: agentgateway-system
spec:
  parentRefs:
    - name: agentgateway-proxy
      namespace: agentgateway-system
  rules:
    - matches:
      - path:
          type: PathPrefix
          value: /
      backendRefs:
      - name: openai
        namespace: agentgateway-system
        group: agentgateway.dev
        kind: AgentgatewayBackend
EOF

Deploy LibreChat

LibreChat requires MongoDB. Deploy both with a ConfigMap that points LibreChat at agentgateway.

Create the LibreChat configuration. Set fetch: false and list models explicitly. When agentgateway is configured with a wildcard (*) model, the /v1/models endpoint returns only the wildcard entry, which LibreChat cannot use to populate its model list.

kubectl apply -f- <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: librechat-config
  namespace: agentgateway-system
data:
  librechat.yaml: |
    version: 1.2.1
    cache: true
    endpoints:
      custom:
        - name: "agentgateway"
          apiKey: "placeholder"
          baseURL: "http://agentgateway-proxy.agentgateway-system.svc.cluster.local/v1"
          models:
            default: ["gpt-4o", "gpt-4o-mini"]
            fetch: false
          titleConvo: true
          titleModel: "gpt-4o-mini"
          modelDisplayLabel: "agentgateway"
EOF

Deploy MongoDB.

kubectl apply -f- <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb
  namespace: agentgateway-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mongodb
  template:
    metadata:
      labels:
        app: mongodb
    spec:
      containers:
      - name: mongodb
        image: mongo:8.0.20
        ports:
        - containerPort: 27017
---
apiVersion: v1
kind: Service
metadata:
  name: mongodb
  namespace: agentgateway-system
spec:
  selector:
    app: mongodb
  ports:
  - port: 27017
    targetPort: 27017
EOF

Deploy LibreChat.

kubectl apply -f- <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: librechat
  namespace: agentgateway-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: librechat
  template:
    metadata:
      labels:
        app: librechat
    spec:
      containers:
      - name: librechat
        image: registry.librechat.ai/danny-avila/librechat-dev:latest
        ports:
        - containerPort: 3080
        env:
        - name: MONGO_URI
          value: "mongodb://mongodb.agentgateway-system.svc.cluster.local:27017/LibreChat"
        - name: DOMAIN_CLIENT
          value: "http://localhost:3080"
        - name: DOMAIN_SERVER
          value: "http://localhost:3080"
        - name: NO_INDEX
          value: "true"
        - name: ALLOW_REGISTRATION
          value: "true"
        - name: OPENAI_API_KEY
          value: "user_provided"
        - name: JWT_SECRET
          value: "change-this-to-a-secure-random-value"
        - name: JWT_REFRESH_SECRET
          value: "change-this-to-a-secure-random-value"
        - name: CREDS_KEY
          value: "<generate-a-64-char-hex-string>"
        - name: CREDS_IV
          value: "<generate-a-32-char-hex-string>"
        volumeMounts:
        - name: librechat-config
          mountPath: /app/librechat.yaml
          subPath: librechat.yaml
      volumes:
      - name: librechat-config
        configMap:
          name: librechat-config
---
apiVersion: v1
kind: Service
metadata:
  name: librechat
  namespace: agentgateway-system
spec:
  selector:
    app: librechat
  ports:
  - port: 3080
    targetPort: 3080
EOF

Replace JWT_SECRET, JWT_REFRESH_SECRET, CREDS_KEY, and CREDS_IV with unique, secure random values before deploying to a shared or production environment. You can generate these with openssl rand -hex 32 (for 64-char keys) or openssl rand -hex 16 (for 32-char IVs). Use a Kubernetes Secret rather than plain env vars for production deployments.

The following table describes the relevant environment variable:

Variable	Description
`OPENAI_API_KEY`	Must be set for LibreChat to start, but it is not used to call OpenAI — agentgateway holds the real key.

Verify the connection

Port-forward to LibreChat.

kubectl port-forward -n agentgateway-system svc/librechat 3080:3080

Open http://localhost:3080, register an account, select the agentgateway endpoint, and send a message.

Confirm the request appears in the agentgateway proxy logs.

kubectl logs deployment/agentgateway-proxy -n agentgateway-system --tail=5

You should see a log entry showing the request was forwarded to the OpenAI endpoint with the configured model:

info  request gateway=agentgateway-system/agentgateway-proxy listener=http route=agentgateway-system/openai endpoint=api.openai.com:443 http.method=POST http.path=/v1/chat/completions http.status=200 protocol=llm gen_ai.operation.name=chat gen_ai.provider.name=openai gen_ai.request.model=gpt-4o gen_ai.usage.input_tokens=4569 gen_ai.usage.output_tokens=10 duration=2242ms

Next steps

Control spendingApply rate limits and token budgets across providers. LLM observabilityMetrics, traces, and access logs for every LLM call.

Goose Open WebUI

Was this page helpful?