| → / Space | Next slide |
| ← | Previous slide |
| Home | First slide |
| End | Last slide |
| Swipe | Touch navigation |
Spend control, audit trails, and risk reduction. How to implement and where not to put it.
AI call volume accumulates faster than governance does. Each surface (coding agents, Genie Agents, Apps agents, MCP servers, external APIs) records calls separately, with no unified audit trail.
Three layers: AI assets (UC securables), AI traffic (control plane), AI service behavior (policies). One gateway governing LLMs, MCPs, and serving endpoints.
Spend control with hard caps. Audit trails. Risk reduction through policy enforcement that fails closed. Velocity through provider switching without code changes.
Four steps: enable, create endpoints, grant permissions, point clients. Runtime controls: rate limits, spend caps, service policies (tool gating + approval workflows).
Never place the gateway in internal OBO paths. Understand fail-closed guardrails. Service policies extend beyond content filtering. Production checklist before launch.
A typical rollout: a production agent on Databricks Apps, a Genie Agent, developers using Cursor or another coding agent, a custom MCP server built last quarter, and an external model API a team wired up directly. Each path was built independently, by a different team, at a different time.
| AI surface | Where its calls are recorded today |
|---|---|
| Coding agent (per developer) | Vendor portal, if the vendor exposes one |
| Production agent on Apps | Agent logs and MLflow traces, if instrumented |
| Genie Agent | Its own dashboard, separate from everything else |
| Custom MCP server | Server stdout or a bucket somewhere |
| External model API (direct) | Whatever the calling team happened to log |
The concrete cost: no single place to see every AI call, guardrails applied on some paths and not others, and no way to answer “what did this cost, and was it safe” without polling every system separately.
AI assets: Unity Catalog securables (models, MCP Services, Agent Services, connections). AI traffic: Central control plane for rate limits, fallbacks, spend caps, usage tracking, and inference logging. AI service behavior: Service Policies govern content and tool calls; guardrails are a built-in subset.
Agents, LLM endpoints, coding agents (Cursor, Codex CLI, Claude Code), Databricks Apps agents. Native multi-provider: OpenAI, Anthropic, Cohere, Bedrock, Vertex AI, Gemini Enterprise, custom OpenAI-compatible.
MCP servers registered as Unity Catalog securables. Tool-level filtering and service policies govern which tools are callable by whom, under what conditions.
AI agents registered as UC securables. Governed alongside tables, models, and functions in the same identity model.
Control plane governs: permissions (UC) · rate limits · service policies · spend caps · usage tracking · inference logging · fallback routing.
Service policies evaluate before invocation (ON CALL) and after response (ON RESULT). Both fail closed — a misconfigured or errored policy blocks the interaction.
One request_id joins all events: system.ai_gateway.usage (cost, principal, blocking decisions) and inference tables (payloads, policy evaluations, latency).
Service policies govern content and tool calls based on caller identity and request attributes. They run at ON CALL (before invocation) and ON RESULT (after response), and they fail closed — a blocked policy returns HTTP 400, a timeout also blocks.
| Category | Scope | Examples |
|---|---|---|
| Content guardrails | LLM input/output | PII redaction/blocking, unsafe content, jailbreak detection, hallucination detection, custom (user-defined) |
| Tool call policies | MCP service invocation | Block a tool, require approval before execution, allow only specific contexts |
| Caller-based policies | Identity + context | Different rules for different users, groups, service principals; context from caller attributes |
| Approval workflows | Human-in-the-loop | Pause and request human approval before proceeding (content, tool invocation, or both) |
Fail-closed by default: A misconfigured service policy, an evaluator timeout, or a policy evaluation error blocks the request. Use mode=LOG to test policies on production traffic without enforcing.
An LLM evaluator (recommended: databricks-gpt-5-nano) scores every request and response. Configure per phase (input/output), per action (block/sanitize), per mode (enforce or log).
| Template | Phase | Action | What it catches |
|---|---|---|---|
| PII Redaction | Input / Output | Sanitize | Replaces PII with placeholders (e.g. [EMAIL]) before the model sees it or the user sees the response. |
| PII Blocking | Input / Output | Block | Refuses the request entirely if the payload contains PII — stricter option for regulated workloads. |
| Unsafe Content | Input / Output | Block | Hate speech, harassment, violence, self-harm, sexual content, weapons, extremism. |
| Jailbreak | Input only | Block | Instruction overrides, obfuscated payloads, role-play exploits, prompt extraction attempts. |
| Hallucination | Output only | Block | Output unsupported by retrieved context — useful where grounding matters (RAG). |
| Custom | Input / Output | You choose | Your own evaluator prompt, up to 5,000 characters — domain-specific policy. |
Fail-closed. Blocked request returns HTTP 400. Evaluator timeout or parse error also blocks (use mode=LOG for dry-run). Service policies extend beyond content filtering to govern tool calls and approvals.
| Traffic | Routing | What you keep | What you lose |
|---|---|---|---|
| Agent ↔ Genie Agents, Agent Bricks, Vector Search, custom MCP on Apps | Direct | Per-user data access control via OBO. Row filters, column masks apply correctly. | N/A — internal Databricks traffic. |
| LLM endpoints, external models (OpenAI, Anthropic, Bedrock, Vertex AI, Gemini) | Managed | Usage tracking, rate limits, service policies, model access control, inference tables, cost attribution via request tags. | None. This is the right path. |
| External LLM API with features we don’t yet wrap (e.g. batch endpoints, files API) | Passthrough | Route the request unmapped. Selective header/query param forwarding. | Usage tracking, rate limits, service policies, model access control, inference tables. Operator beware. |
| External clients (enterprise apps, partners) calling Databricks | External gateway | Boundary crossing from external identity → Databricks token. Then managed paths or direct. | None — external gateway is additive. |
Critical: Never put the gateway in the OBO path. Internal agent-to-Genie Agents or agent-to-MCP calls must stay direct. The gateway re-issues its own token, so current_user() becomes the gateway’s identity, and row filters silently evaluate wrong.
Unity AI Gateway supports spend budgets with shared and per-user thresholds. Thresholds can be configured either to alert or, when block usage is enabled, to stop further requests until the budget resets or an admin raises the threshold. Enforcement is near real time, so a small amount of overspend may still occur.
# budget config shape budgets: - scope: shared | per_user period: monthly limit_usd: 500 action: alert | block
Join system.billing.usage to system.ai_gateway.usage by usage_context tags — the same UC query model finance already uses. No export pipeline.
An agent that calls an LLM, then an MCP tool, then another LLM is three governance domains without a gateway. With one, it is a single trace.
“Show me every AI call this org made last week” means checking a coding-agent vendor portal, agent logs, a Genie Agent dashboard, a custom MCP server’s stdout, and whatever the team using an external API happened to log. Days, not minutes — and still incomplete.
One query against system.ai_gateway.usage plus inference tables. Every principal, every endpoint, every guardrail decision, joinable to billing. Minutes, not days.
Fail-closed guardrails mean a blocked request is a recorded decision, not a silent pass-through — the audit trail includes what was stopped, not just what succeeded.
Primary provider fails → gateway routes to a secondary served entity automatically. A/B test a new model on a percentage of traffic. Stage a rollout from one provider to another with built-in comparison metrics — no client code changes.
Cursor, Codex CLI, Gemini CLI, and Claude Code point at OpenAI-compatible base URLs. A Databricks Apps agent changes one base URL. Both inherit permissions, rate limits, guardrails, usage tracking, and cost attribution — with zero additional wiring.
Native provider APIs, not a translation layer. Each provider’s SDK works unchanged against the gateway’s native surface — streaming, system prompts, structured tool use all behave exactly as the SDK expects.
Account admin enables Unity AI Gateway from the account console (workspace-wide flip).
Pick a target model (FMAPI, external provider, custom). Attach rate limits, guardrails, and service policies.
Add users, groups, and service principals to UC securable EXECUTE grants — same identity model as UC tables.
Update agents, coding tools, and apps to the gateway base URL. Existing SDKs work unchanged.
from openai import OpenAI client = OpenAI( base_url="https://<uag>/v1", api_key=DATABRICKS_TOKEN, ) response = client.chat.completions.create(...)
from anthropic import Anthropic client = Anthropic( base_url="https://<uag>/anthropic/v1", api_key=DATABRICKS_TOKEN, ) response = client.messages.create(...)
Same for Gemini, Cohere, custom OpenAI-compatible. Coding agents use /cursor/v1, /codex/v1.
PUT /api/2.0/serving-endpoints/<name>/ai-gateway { "rate_limits": [{"calls": 100, "renewal_period": "minute"}], "guardrails": { "input": {"safety": true, "pii": {"behavior": "BLOCK"}}, "output": {"safety": true, "pii": {"behavior": "BLOCK"}} }, "usage_tracking_config": {"enabled": true}, "inference_table_config": { "enabled": true, "catalog_name": "main", "schema_name": "ai_governance" } }
| Lever | Governs | Detail |
|---|---|---|
| Rate limits | Call volume | QPM (calls) and TPM (tokens), per endpoint/user/group/SP. Precedence: endpoint ceiling > named SP override > user default > group default. Max 20 limits per endpoint. Updates propagate in ~60s. |
| Spend caps | Dollar budget | Hard block at threshold, per endpoint/user/account. Enforced on near-real-time cost estimate (can overspend slightly). Returns 429 when exhausted. Pair with rate limits for multi-layer cost control. |
| Service policies | Content + tools + approval | ABAC: enforce rules before invocation (ON CALL) and after response (ON RESULT). Allow, deny, or require human approval. Block tool invocation by principal or context. Fail closed. |
Managed vs. passthrough trade-off: Managed paths give you all controls (rate limits, spend caps, service policies, inference tables, model access control). Passthrough bypasses governance for provider features not yet wrapped. Default to managed paths.
Pass custom tags via Databricks-Ai-Gateway-Request-Tags header (JSON key-value pairs). Tags appear in system.ai_gateway.usage for cost attribution to projects, teams, or workflows. Join to system.billing.usage for unified spend reporting.
curl -X POST \\ -H "Databricks-Ai-Gateway-Request-Tags: {...}" \\ https://<workspace>/ai-gateway/openai/v1/... # Tags logged to system.ai_gateway.usage: { "project": "chatbot-v2", "team": "platform", "priority": "high" }
| Issue | Detail |
|---|---|
| Gateway in OBO path | Critical: never put the gateway in the OBO path between an agent and Genie Agents, Agent Bricks, or a custom MCP server. The gateway re-issues its own token, so current_user() becomes the gateway’s identity, and row filters evaluate silently wrong. Keep internal Databricks-to-Databricks traffic direct. |
| Guardrails don’t soft-fail | Guardrails fail closed. An evaluator timeout, parse error, or misconfiguration blocks the request (HTTP 400). Use mode=LOG for dry-run testing without enforcement. |
| Spend cap is a hard block | Not a soft alert. Requests return 429 once the budget is hit. No manual override until the billing period resets or an admin raises the cap. |
| Output guardrails + streaming | Output guardrails evaluate post-response, so streaming responses bypass them. Use input guardrails or non-streaming requests for output policy enforcement. |
| Passthrough loses governance | Passthrough paths (provider features not yet wrapped) bypass rate limits, service policies, and inference tables. Operator beware. |
Grant EXECUTE to specific groups, not workspace-wide. Coding agents accessible only to developer groups.
QPM and TPM caps; tighter limits for service principals than human users. Endpoint ceiling > individual overrides.
Test guardrails on production traffic in mode=LOG dry-run before switching to mode=ENFORCE.
Rate limits control traffic volume. Spend caps control dollars. Use both as multi-layer cost control.
UC catalog with retention policy, audit-grade access controls. Join to system.billing.usage for cost analysis.
Confirm no gateway sits between agent and Genie Agents, Agent Bricks, or custom MCP server. OBO path must be direct.