Skip to main content
LLM Gateway 2.0 can screen each request and response with guardrail rules, and retry a failed request on another alias with a fallback chain.

​
Guardrails

A guardrail rule screens requests before they leave the gateway, responses before they reach the caller, or both. Each rule has:
  • A direction: input, output, or both.
  • An action: deny, redact, or log.
  • A check: a regex pattern, or an external large language model (LLM) evaluator. An LLM evaluator is any HTTP endpoint that returns a score, which the gateway compares to the rule’s threshold.
Rules apply to every alias unless you scope them to specific ones. For each LLM evaluator, choose whether requests pass or fail when the evaluator times out or errors.

​
What the gateway scans

On input, the gateway scans user turns and tool results, so a personally identifiable information (PII) rule fires on file contents an agent sends back through a tool call. The gateway tags requests with tool results as source="tool_result" in the audit log.

​
Guardrail limits

  • On streaming responses, output rules can only log. The response is already delivered by the time the gateway has the full text, so deny and redact become log.
  • The gateway checks text only. It skips images in multimodal requests.
  • The gateway doesn’t scan system prompts, the Responses instructions field, or earlier assistant turns.
  • The gateway doesn’t scan tool-call arguments the model generates, such as a URL, shell command, or SQL string.
  • Each LLM evaluator rule adds a network round trip to every matching request. Use regex rules for high-volume checks.

​
Fallback routing

A fallback chain is an ordered list of aliases to try when the primary alias fails. Set it on the primary alias under Catalog > edit alias > Fallback chain. Links in the chain inherit the primary alias’s triggers and per-attempt timeout.

​
Triggers

These triggers are all on by default: timeout, 5xx, 429, and budget_exhausted. Other 4xx errors, such as auth failures, guardrail denials, and malformed requests, don’t advance the chain.

​
How the gateway walks the chain

The gateway checks access on each link as if the caller had requested it directly, and skips links the caller can’t use. Each attempt gets its own usage log entry. A reply served by a fallback carries X-LLM-Fallback-Reason and X-LLM-Fallback-Served-By headers. The Logs view can filter to fallback events.

​
Fallback limits

  • Chains apply to /v1/chat/completions only.
  • Fallback happens before the response starts. An error mid-stream goes to the caller.
  • Streaming requests fall back only on a budget block, not on upstream errors.
Last modified on October 15, 2026