| → / Space | Next slide |
| ← | Previous slide |
| Home | First slide |
| End | Last slide |
| Swipe | Touch navigation |
Six governance layers: model routing, rate limits, spend caps, guardrails, approval gates, and sandbox isolation
Builds on the Unity AI Gateway deck and the Agent Orchestration & Tool Governance deck. This deck: governing agent cost and risk specifically.
Agent traffic — LLM calls, tool calls, and execution actions — creates cost and risk that's invisible without governance configured at the gateway.
Model routing and allowlisting, rate limits, spend caps, guardrails, approval gates, and sandbox isolation — six independently configurable controls on one gateway.
Cost control, risk reduction, compliance, and engineering velocity — with concrete rate-limit, spend-cap, and cost-reduction figures.
The actual gateway and Omnigent YAML — model allowlists, rate-limit hierarchy, spend caps, service policies, sandbox policy.
system.ai_gateway.usage — the queries that answer a cost question before finance has to ask it.
Unity AI Gateway is the chokepoint all agent traffic already passes through. Without governance configured on top of it, it is a pass-through — the gap is configuration, not a limitation of the gateway itself.
Multi-step agentic loops. Can run for hours without a human checkpoint.
Many engineers hitting completions constantly adds up faster than it is usually modeled.
Token-heavy prompts, long context windows. One template can run 200k tokens per call.
All three route through Unity AI Gateway. The chokepoint exists. Until governance is configured, the controls do not.
Every hop in an agent loop is a separate cost event and a separate risk surface. The governance problem is wider than token pricing.
git push, terraform applysystem.ai_gateway.usage captures every call. Custom request tags (request-tags header) enable cost attribution by project, team, or priority.
Defense in depth. Some controls are account-wide, some per-team, some per-agent type. Each layer can change without touching the others.
| Control layer | Configured by | Scope | What it stops |
|---|---|---|---|
| 1. Model routing | Platform team | Per route / per team | Model-tier creep, unnecessary premium-model usage |
| 2. Rate limits | Platform team | Account → team → user (QPM, TPM) | Runaway loops, burst overconsumption |
| 3. Spend caps | Finance + platform | Account, team, user (daily) — two actions | Billing surprises, budget overruns |
| 4. Guardrails | Security team | Account-level default, all routes | Secret exfiltration, prompt injection, PII leakage |
| 5. Service policies | Engineering leadership | Per tool-call type, contextual | Unauthorized deploys, destructive actions |
| 6. Sandbox isolation | Platform team | Per agent session | Filesystem escape, credential theft, scope creep |
Guardrails run on every request and response, before the model sees the input and before the agent sees the output. First match wins.
Ignore all previous instructions)AKIA*)dapi*)PII in agent context (emails, employee IDs, SSNs, card numbers) is redacted before the LLM processes it.
Redaction is deterministic: the same entity gets the same placeholder across a session.
john.doe@acme.com
→
[EMAIL_1]
pii_detection: true required to enable built-in PII redaction — guardrails are account-level defaults, not per-team opt-in.
| Agent action | Autonomous? | Requires |
|---|---|---|
read_file | ✓ Yes | — |
run_tests | ✓ Yes | — |
create_pr | ✓ Yes | — |
git push main | ✗ No | Human approval |
terraform apply | ✗ No | Human approval |
delete_resource | ✗ No | Human approval |
Service policies live in gateway configuration. An agent cannot bypass them — the gateway intercepts the tool call and issues a hold before forwarding.
| ALLOW | Proceed as requested |
| DENY | Block the action outright |
| ASK | Pause for human approval |
| TRANSFORM | Modify the action, then proceed |
Contextual policies evaluate every tool call and model response during a session and track cumulative state. For cost budgets: a soft threshold triggers ASK for confirmation; a hard cap triggers TRANSFORM, which can swap the agent to a more efficient model rather than blocking outright. The same state tracking backs intent-based policies, which scope an agent’s tools to what it declared it would do for the current session.
Each coding-agent session gets its own isolated Databricks Sandbox — filesystem and network scoped to that session, credentials injected at session start. A session is shareable by URL for full review.
A 60 RPM per-engineer coding-agent limit stops a runaway loop within the first minute, well before an unattended loop could run up hundreds of dollars in overnight spend. Engineer sees a 429, investigates, fixes the loop.
An engineer using a premium-tier model for a task a lower tier could handle can cost roughly 15× more for the same output. Allowlisting removes the choice rather than relying on attention.
Model allowlisting drove a 34% reduction in model spend with no feature changes — the expected range going in was 20–40%.
| Scope | Cap | Period | On breach |
|---|---|---|---|
| Account | $15,000 | monthly | Alert or Block |
| Team | $2,000 | monthly | Alert or Block |
| Engineer | $50 | daily | Alert or Block |
Rate limits stop loops in real time. Caps stop cumulative overruns across the billing period. Both are needed — neither alone is sufficient.
Budget enforcement uses near-real-time cost estimates, so actual spend can overshoot the threshold before blocking engages. For a firm ceiling, pair budgets with rate limits. Use Alert for monitoring; use Block to prevent further usage.
system.ai_gateway.usage. Chargebacks automated.git push main and terraform apply now requires team-lead approval.# Serving endpoint config (simplified) route_config: routes: - name: coding-agent-senior model_allowlist: - claude-sonnet-4-* - claude-opus-4-* tags_filter: {team: engineering, tier: senior} - name: completion-tool-mid model_allowlist: - claude-haiku-3-* tags_filter: {team: engineering, tier: mid} - name: notebook-agent-data model_allowlist: - gemini-2-flash-* tags_filter: {team: data}
Coding agent — senior: Sonnet default, Opus on explicit request. All other model families blocked.
Completion tool — mid-level: Haiku-equivalent tier only. Sonnet, Opus, and Gemini Pro blocked.
Notebook agent — data team: Approved Gemini family, flash by default. All Claude families and GPT-4 blocked.
# What the agent receives on throttle HTTP/1.1 429 Too Many Requests Retry-After: 60 { "error": "rate_limit_exceeded", "limit": "60 RPM per engineer", "reset_at": "2026-07-07T14:32:00Z" }
First-match-wins: the gateway evaluates most-specific to least-specific. A per-engineer burst limit fires before the team or account limit ever needs to.
# Spend cap response — same 429, different cause HTTP/1.1 429 Too Many Requests { "error": "spend_cap_exceeded", "scope": "user", "cap_usd": 50.00, "spent_usd": 50.12, "resets_at": "2026-07-08T00:00:00Z" }
Finance integration: spend data in system.ai_gateway.usage feeds a nightly roll-up — a line item per team per day, no override path once a cap fires.
# Gateway service policy (simplified) service_policies: - name: high-risk-tool-gate match: tool_names: - git_push_main - terraform_apply - delete_resource action: require_approval approval: timeout_seconds: 300 approver_groups: [team-leads] on_timeout: deny
The gateway intercepts the matching tool call and returns a hold. Execution pauses. A notification goes to the on-call team lead. Approve → agent continues. Deny or timeout → agent receives a rejection. The entire flow is logged.
Policies live in gateway configuration, not in a prompt or tool definition — an agent cannot reason its way past one.
# Per-session isolation and cost budget guardrails: policies: cost_budget: type: contextual on: [request] soft_threshold: 500000 # tokens soft_action: ask # pause, ask human hard_limit: 1000000 hard_action: transform # switch to cheaper model isolation: filesystem: workspace_only network: - api.github.com - internal.jira.example.com credentials: env_scoped # rotated per session
A single coding-agent session has its own token budget separate from gateway-level caps. Soft threshold pauses for confirmation; hard cap switches the agent to a more efficient model before terminating.
Each session has a shareable URL. Team leads can inspect what the agent sees: working directory, full tool-call history, LLM conversation, and sandbox decisions.
Filesystem, network, and credentials are scoped at the OS level before the gateway is involved. An agent cannot escape its sandbox or access unlisted services.
system.ai_gateway.usage-- Daily spend by engineer SELECT user_id, date_trunc('day', timestamp) AS day, sum(total_cost_usd) AS spend_usd, sum(total_tokens) AS tokens FROM system.ai_gateway.usage WHERE workspace_id = current_workspace_id() AND timestamp >= current_date - INTERVAL 30 DAYS GROUP BY 1, 2 ORDER BY spend_usd DESC;
-- Anomaly alert: 3x daily average breach WITH daily_avg AS ( SELECT user_id, avg(daily_spend) AS avg_spend FROM ( SELECT user_id, DATE(timestamp) AS d, sum(total_cost_usd) AS daily_spend FROM system.ai_gateway.usage WHERE timestamp >= current_date - 30 GROUP BY 1, 2 ) GROUP BY 1 ) SELECT u.user_id, today.spend, d.avg_spend, today.spend / d.avg_spend AS ratio FROM daily_avg d JOIN (...today_spend...) today USING (user_id) WHERE today.spend > d.avg_spend * 3;
| Issue | Detail |
|---|---|
| Model-tier creep | Without allowlisting, model choice defaults to whoever pays least attention — a premium model can cost roughly 15× a lower tier for the same output. |
| Budgets don't track provisioned throughput | Unity AI Gateway budgets track only pay-per-token and batch inference. Do not rely on budget caps to limit provisioned-throughput spend. |
| Budget scale limits | Max 4 shared thresholds per budget, max 20 per-user overrides per budget, max 1,000 budgets per account. |
| Enforcement is approximate | Budgets use near-real-time cost estimates, so actual spend can overshoot before blocking engages. Pair budgets with rate limits for firm control. |
pii_detection: true is not implicit | Built-in PII redaction requires this flag explicitly set on the route. |
The billable-usage system table rolls up cost at the UC model-service level regardless of which API surface (mlflow-format, native provider API) was used. A pre-built usage dashboard visualizes spend trends by team, agent type, and model.
An open-source meta-harness that wraps a coding-agent CLI (Claude Code, Codex) behind one contract, then runs the agent loop with sessions, a sandbox, and a policy engine. Available self-hosted or Databricks-managed.
It inserts a governed hop in the loop. Every tool call routes back to the server, a stateful policy returns ALLOW, ASK, DENY, or TRANSFORM, then the Omnibox sandbox runs the approved action. The gate runs even under CLI bypass mode, so it is not opt-out.
You get control over what the agent does, not just what the model reads and writes: approval gates, per-session spend ceilings, kernel-level filesystem and network isolation, shared and replayable sessions, and a durable transcript.
Choose Omnigent when the agent needs guardrails on its actions. Managed for zero-setup workspace auth and the strongest sandbox, OSS when you need custom policies, your own host, or your own model keys. If you only need auth and usage, this is more than you need, so use ug.
# install once uv tool install git+https://github.com/databricks/unity-gateway # sign in, discover models, write each agent's config ug configure --workspace https://YOUR-WORKSPACE-HOST --agents claude,codex # launch, pick a governed model service ug claude --model system.ai.claude-sonnet-4-5 ug mcp add # register governed MCP tools ug usage # per-user spend and token totals
ug handles OAuth and writes each agent's config, so there are no provider API keys and no manual endpoint wiring. Requests land on the gateway under workspace identity and record to system.ai_gateway.usage.
A thin launcher (formerly ucode; the old name stays as a backward-compatible alias) that configures a supported agent (Claude Code, Codex, and others) and launches it against the gateway. No harness, no sessions, no sandbox.
The fastest path to per-user auth, model routing, and usage visibility on an agent your engineers already run locally. One CLI configures many agents the same way.
Choose ug when the agent is already installed and you just need per-user auth, model routing, and spend visibility. The moment you need sessions, sandboxing, or policy on actions, graduate to Omnigent. ug governs model traffic, not what the agent does on disk.
Each step is independently valuable. Stop after any step if the org isn't ready. Later steps get easier once earlier ones are in place.
Enable system.ai_gateway.usage and run the spend-by-engineer query. No controls yet — just visibility.
Map teams to model tiers by actual task profile. Expect 20–40% cost reduction within the first billing cycle.
Set per-engineer RPM limits conservatively, then tune up. Runaway loops stop; normal use barely notices.
Account, team, and daily-user caps. Finance gets a committed ceiling; engineering gets autonomy within it.
PII detection and output secret scanning. Security team owns this layer independently.
Service policies and Omnigent sandboxes for agents with infrastructure access. Pilot with one team, tune, then expand.