01 / 15

Applied AI Governance

Identity & Audit
Architecture

16 AI services. 6 identity models. 12 OAuth scopes.
Every audit gap mapped. Every proxy hop documented.

16

Services

6

Identity Models

12

OAuth Scopes

5

Platform Gaps

02 / 15

The Problem

Who did what, and can you prove it?

Identity Fragmentation

Different services use different identity models. Some preserve the human in audit, others show only the service principal UUID. Without a map, you can't predict what shows up in system.access.audit.

Audit Gaps

Under M2M patterns, UC audit records the SP UUID, not the human who initiated the action. Application-plane audit is the only bridge back to the human.

Scope Confusion

OAuth scopes with undocumented requirements, silent 403 failures, and UI-vs-CLI behavior differences. The genie scope isn't even shown in the account UI but is required on Azure.

Proxy Complexity

Databricks Apps proxy strips tokens and injects headers. App-to-app calls add a second proxy. External MCP adds a third. Each hop changes the identity semantics.

03 / 15

Complete Service Identity Map

16 services, one truth table

#	Service	Identity Model	current_user()	UC Audit Shows
1	Genie Agents (Conversation API)	True OBO	`User email`	User
2	AI/BI Dashboard (run-as-viewer)	True OBO	`Viewer email`	Viewer
3	AI/BI Dashboard (run-as-owner)	Delegated	`Owner email`	Owner
4	Agent Bricks / Model Serving	True OBO	`User email`	User
5	SQL Warehouse (user token)	True OBO	`User email`	User
6	SQL Warehouse (SP token)	Pure M2M	`SP UUID`	SP UUID
7	Custom MCP (Databricks Apps)	Proxy + M2M	`SP UUID`	SP UUID
8	UC Functions (via M2M SQL)	Pure M2M	`SP UUID`	SP UUID
9	UC Functions (via Genie Agents OBO)	True OBO	`User email`	User
10	Vector Search	Pure M2M	`N/A`	SP UUID
11	Foundation Model API	M2M / OBO	`N/A`	Caller identity
12	SDP / Lakeflow Pipeline	Delegated	`Owner email`	Owner
13	External MCP (shared bearer)	M2M at Proxy	`N/A`	Caller at proxy
14	External MCP (per-user OAuth)	OBO at Proxy	`N/A`	User at proxy
15	Custom MCP → Custom MCP	Proxy + M2M	`SP UUID`	SP UUID
16	Lakebase	DB → Postgres	`DB email`	Management ops

Key insight: Only 5 of 16 services give you True OBO. The rest require either app-plane audit, delegated trust, or proxy-mediated identity.

04 / 15

Identity Models

6 patterns, one question: who executes?

True OBO

Services: #1, 2, 4, 5, 9

User's token executes the operation. UC sees the human. Automatic audit capture. The gold standard.

Proxy + M2M

Services: #7, 15

App SP executes. Human identity from X-Forwarded-Email (proxy-injected, unforgeable). UC audit shows SP UUID only.

Pure M2M

Services: #6, 8, 10, 11

App SP executes. No user involvement at all. UC audit shows SP UUID. Per-user attribution impossible without app-plane audit.

Delegated

Services: #3, 12

Owner's identity executes. UC sees the owner, not the end user. Fine for dashboards, problematic for multi-user pipelines.

External MCP

Services: #13, 14

Databricks proxy authenticates. External service sees shared bearer (#13) or per-user OAuth (#14). Split audit across two systems.

DB → Postgres

Services: #16

Databricks OAuth identity mapped to Postgres role via databricks_auth. RLS enforced at database level. Data-plane query audit undocumented.

05 / 15

Audit Gap Matrix

Does UC audit capture the human?

Identity Model	UC Captures Human?	App-Plane Audit	Affected Services
True OBO	Yes	Optional	#1, 2, 4, 5, 9
Proxy + M2M	No, SP UUID	Required	#7, 15
Pure M2M	No, SP UUID	Required	#6, 8, 10, 11
Delegated	Owner only	Depends	#3, 12
External MCP	At proxy	Recommended	#13, 14
DB → Postgres	Mgmt ops only	Depends	#16

Bottom line: 8 of 16 services lose the human identity in UC audit. For those services, application-plane audit with the @audited decorator is the only way to close the gap.

06 / 15

OAuth Scopes Reference

12 scopes, 4 gotchas that break things silently

Scope	Grants	Required By	Gotcha
`openid`	OIDC identity token	All OAuth flows	none
`email`	Email claim in token	Proxy X-Forwarded-Email	none
`profile`	Display name claims	Proxy X-Forwarded-Preferred-Username	none
`offline_access`	Refresh token	Long-lived sessions	none
`dashboards.genie`	Genie Agent access	Genie Conversation API	Must pair with `genie` on Azure
`genie`	Genie Conversation API	Genie Agents (Azure)	Not shown in account UI. Required on Azure. Silent 403 without it.
`model-serving`	Model Serving / Agent Bricks	Agent Bricks, FM API OBO	none
`sql`	Statement Execution API	Direct SQL with user token	CLI `custom-app-integration update` does NOT embed scope in JWT. Use UI “User Authorization” instead.
`unity-catalog`	UC REST API + External MCP proxy	External MCP via UC HTTP	Undocumented. Proxy checks this before `USE CONNECTION`. Without it: 403.
`all-apis`	Catch-all for Databricks REST APIs	General API access	Not a superset of granular scopes

07 / 15

Interactive Tool

OAuth Scope Calculator

Select the services your app uses. Get the minimum scopes required.

Select Services

Required Scopes

Select services to calculate minimum scopes.

08 / 15

Proxy Architecture

What happens at each hop

Single proxy path

User (Browser / Claude)

→

Databricks Apps Proxy

→

Your App (:8000)

What the proxy does

1. Authenticates the caller (OAuth cookie or Bearer token)
2. Strips the original Authorization header
3. Injects X-Forwarded headers (unforgeable)

Trusted Headers

Header	Trust	Content
`X-Forwarded-Email`	High	Proxy-verified user identity. Unforgeable. Primary identity source in M2M patterns.
`X-Forwarded-Access-Token`	High	OIDC JWT. SP-scoped but unforgeable. Used for OBO API calls (Genie Agents, Agent Bricks).
`X-Forwarded-User`	High	`{user_id}@{workspace_id}`. Unique cross-workspace identifier.

Two-proxy problem: When App A calls App B, Proxy B strips App A's token and substitutes its own SP token. Solution: authorization: disabled on App B + read X-Forwarded-Email for identity. Trade-off: external clients can't use App B directly.

09 / 15

Service Principal Isolation

One SP per capability boundary

SP	Application	Read Capabilities	Write Capabilities	Why Separate
SP-A	Streamlit app	Genie Agents (OBO), VS indexes, UC Functions, FM API	None	Frontend, no direct write access
SP-B	Custom MCP server	opportunities, quota_viewers, customers	approval_requests (INSERT)	MCP needs write; Streamlit does not
SP-C	Claude Code MCP	opportunities, quota_viewers, customers	approval_requests (INSERT)	Different auth setting, OAuth integration, lifecycle
SP-D	Agent Bricks supervisor	Sub-agents inherit user token (OBO)	Through sub-agent tools only	Platform-managed; sub-agent access = user's access

Principle: Separate SPs per capability boundary, not per app. A read-only tool vulnerability should never become a write exploit. Minimal, non-overlapping grants.

10 / 15

Confused Deputy Prevention

5 anti-patterns to avoid

#	Anti-Pattern	Why Dangerous	Correct Approach
1	Sharing `CLIENT_ID/SECRET` across apps	One compromise = all apps compromised	Each `databricks apps create` auto-generates a unique SP
2	`ALL PRIVILEGES` on a catalog	SP can read, write, drop, grant	Object-level permissions only
3	Same SP for read and write tools	Read vulnerability becomes write exploit	Separate SPs per capability boundary
4	Re-using app OAuth integration for external clients	Scope changes affect production users	Separate OAuth integration per target
5	Adding MCP SP to admin groups	Bypasses all row filters and column masks	SP only in groups matching declared capabilities

11 / 15

Stopgap Audit Solution

Bridge the gap with @audited

The Pattern

Drop-in decorator. Zero changes to tool logic. Silent failure, audit write errors never break the tool.

@mcp.tool()
@audited()
def get_deal_approval_status(opp_id: str) -> dict:
    # ... existing logic unchanged ...

@mcp.tool()
@audited(service_type="uc_function")
def call_uc_function(func_name: str) -> dict:
    # ... existing logic unchanged ...

What It Captures (15 fields)

caller_email, X-Forwarded-Email
caller_user_id, X-Forwarded-User
caller_display, display name
app_sp_id, DATABRICKS_CLIENT_ID
app_name, application name
service_type, custom_mcp | uc_function | ...
tool_name, function name
tool_args_hash, SHA-256 for dedup
tool_args_safe, sanitized args
result_status, success | error | denied
error_message, error detail
trace_id, MLflow trace ID
request_id, UUID per invocation
event_time, UTC timestamp
duration_ms, execution time

12 / 15

End-to-End Traceability

Chain of Custody: Human → Tool → SQL → Data

Alice (human)

→

MCP tool call

→

@audited captures

→

SP executes SQL

→

UC audit records

Join Method

SP UUID + time window (±2s start, +30s end). Falls back to trace_id for precise join when MLflow tracing is available.

Alert Views

critical alert_missing_identity, empty email = broken auth
warning alert_high_frequency, >50 calls/hr per user
info alert_write_ops, all write operations
warning alert_error_rate, >10% error rate

The Query

WITH app_events AS (
  SELECT request_id, caller_email, app_sp_id,
         tool_name, event_time, trace_id,
         event_time - INTERVAL 2 SECONDS  AS w_start,
         event_time + INTERVAL 30 SECONDS AS w_end
  FROM catalog.audit.tool_invocations
  WHERE event_time > current_timestamp() - INTERVAL 24 HOURS
),
uc_events AS (
  SELECT event_time AS uc_time,
         user_identity.email AS uc_executor,
         action_name AS uc_action
  FROM system.access.audit
  WHERE service_name IN ('unityCatalog','databricksSql')
)
SELECT a.caller_email AS human,
       a.tool_name    AS tool,
       u.uc_action    AS uc_operation,
       u.uc_executor  AS uc_identity
FROM app_events a
LEFT JOIN uc_events u
  ON u.uc_executor = a.app_sp_id
  AND u.uc_time BETWEEN a.w_start AND a.w_end
ORDER BY a.event_time DESC;

13 / 15

Governance

USE CONNECTION: the confused deputy defense

The Attack

A malicious user creates a function that calls an external endpoint to exfiltrate data. When a benign user triggers that function (via Genie Agents, OneChat, or a shared space), it executes under the benign user's credentials.

The Defense Stack

Layer 1: Lock down CREATE FUNCTION to curated schemas
Layer 2: Tightly govern UC connections, no broad USE CONNECTION
Layer 3: No EXECUTE grants to account users
Layer 4: Monitor audit logs for function creation + external calls
Layer 5: Advocate for Genie Agent review workflows

Strongest defense: No USE CONNECTION = no exfiltration. Even a perfectly crafted malicious function has nowhere to send the data.

14 / 15

Honest Assessment

5 limitations you should know

Limitation	Impact	Mitigation
Time-window join is approximate	If two users call the same tool within 30s via the same SP, UC audit events may be attributed to the wrong user	Use `trace_id` for precise join when MLflow tracing is available
Audit table requires SP write access	The SP must have MODIFY on the audit table, an additional privilege	Dedicated audit table, not production data. Acceptable trade-off.
No platform enforcement of required tags	MLflow trace tags are convention-based. A developer can ship without `user_email` tag.	`@audited` decorator makes it automatic. Code review catches gaps.
External MCP audit is one-sided	Records that Alice called the GitHub MCP tool but NOT what GitHub returned	Per-user OAuth: GitHub has its own audit. Shared bearer: limited to shared identity.
Lakebase data-plane audit undocumented	Management ops audited. SQL query audit on Lakebase tables: unknown.	Postgres `pg_audit` extension available. Databricks integration TBD.

15 / 15

Identity Explorer

Every service.
Every identity model.
Every gap.

A reference for architecture reviews: which identity each service honors, where audit visibility drops, and how application-plane audit closes the gap.

16

Services

6

Models

12

Scopes

5

Platform Gaps

Applied AI Governance · Identity & Audit Architecture Reference

Identity & AuditArchitecture

Who did what, and can you prove it?

Identity Fragmentation

Audit Gaps

Scope Confusion

Proxy Complexity

16 services, one truth table

6 patterns, one question: who executes?

True OBO

Proxy + M2M

Pure M2M

Delegated

External MCP

DB → Postgres

Does UC audit capture the human?

12 scopes, 4 gotchas that break things silently

OAuth Scope Calculator

Select Services

Required Scopes

What happens at each hop

Single proxy path

What the proxy does

Trusted Headers

One SP per capability boundary

5 anti-patterns to avoid

Bridge the gap with @audited

The Pattern

What It Captures (15 fields)

Chain of Custody: Human → Tool → SQL → Data

Join Method

Alert Views

The Query

USE CONNECTION: the confused deputy defense

The Attack

The Defense Stack

5 limitations you should know

Every service.Every identity model.Every gap.

Identity & Audit
Architecture

Every service.
Every identity model.
Every gap.