| → / Space | Next slide |
| ← | Previous slide |
| Home | First slide |
| End | Last slide |
| Swipe | Touch navigation |
Nightly ETL · Customer API Gateway · Compliance Reporting — each with its own SP, its own group, its own privilege scope
Companion to the Human OBO Identity deck. That deck = human calls Databricks as themselves. This deck = service calls Databricks as a controlled identity.
One shared service-principal credential used across multiple services — and the blast-radius, rotation, and audit problems that design creates.
The SP → Group → Unity Catalog privilege chain, and the client-credentials (M2M) token flow that underlies it.
Blast-radius reduction, clean audit attribution, and independent credential rotation — concrete risk and cost outcomes.
Step-by-step setup, secret management, token scopes, and the SP-vs-OBO decision gate for choosing the right identity model.
Update the secret → redeploy the ETL pipeline → redeploy the API gateway → redeploy the compliance job → coordinate the change window → something breaks. Every. Time.
The compliance reporting job only needs SELECT on system audit tables. But the shared SP has MODIFY on production data. Now a compromised compliance job token can corrupt production data.
system.access.audit shows one SP identity for all operations. Was it the ETL pipeline? The API gateway? The compliance reporting job? Impossible to tell without logs outside Databricks.
One leaked secret grants access to every catalog, every table, every model endpoint the SP was ever granted access to. Revoke it and all three services go down simultaneously.
The group—not the SP directly—holds the UC GRANTs. Membership is the lever for both granting and cutting access.
Revoke group membership → instant access cutoff, no SQL redeployment needed
Rotate SP secret → zero permission changes, no code updates required
Add second SP to group → one operation, automatically inherits all grants
An SP gets a token by posting its identity to the Databricks OIDC token endpoint. No human in the loop.
POST https://<workspace-host>/oidc/v1/token Content-Type: application/x-www-form-urlencoded grant_type=client_credentials &client_id=<sp-application-uuid> &client_secret=<sp-secret> &scope=all-apis
POST https://accounts.cloud.databricks.com/oidc/accounts/<account-id>/v1/token Content-Type: application/x-www-form-urlencoded grant_type=client_credentials &client_id=<sp-application-uuid> &client_secret=<sp-secret> &scope=all-apis
{
"access_token": "eyJhbG...", // JWT
"token_type": "Bearer",
"expires_in": 3600 // seconds
}
The token is a standard Bearer JWT. Pass it in the Authorization header on every Databricks API or SQL call. The SP identity rides inside the JWT—no session state on the server side.
current_user() returns a human email addressall-apis or sql)current_user() returns the SP’s application ID—a UUID, not an emailKey difference: current_user() for an SP returns its UUID-format application ID, not a human email. Row filters and column masks work identically for both—the evaluation mechanism is the same, only the identity token differs.
If two services share a credential, you lose the ability to rotate independently. Scope by function, not by team.
| Service | SP Name | Group | Token Scope | UC Privilege |
|---|---|---|---|---|
| Data pipeline Ingests and transforms raw data through bronze and silver layers |
etl-pipeline-sp |
pipeline-readers |
all-apis | prod.bronze.* SELECT + MODIFYprod.silver.* MODIFY |
| Serving API Reads pre-computed predictions for real-time queries |
serving-api-sp |
api-readers |
sql | prod.silver.predictions SELECT |
| Audit reporting Generates compliance and operational reports from system tables |
reporting-sp |
audit-readers |
sql | system.access.audit SELECTsystem.access.group_membership SELECT |
Independent rotation: ETL secret rotates without touching the API gateway or compliance job
Least privilege by design: compliance job cannot write bronze data even if compromised
Clear audit attribution: every action in system.access.audit maps to exactly one service
A token scope limits what API surface the SP can reach—even if it has UC grants on a broader set of resources.
Full Databricks API surface: Jobs, compute, DBFS, SQL, Secrets. Needed because the pipeline starts scheduled jobs, writes staging data, and runs SQL transforms.
Use only when the service genuinely needs both compute and data APIs.
SQL warehouse access only. The API serves pre-computed results from a silver-layer table via SQL queries. It does not need to start jobs, manage clusters, or read secrets.
Recommended for read-only serving. Cannot call compute or job APIs even if the code is compromised.
SQL warehouse access for SELECT on system audit tables. Reports aggregate access events and group membership changes without modifying any data.
Even if the SP secret leaks, the attacker cannot start a cluster or modify jobs with a sql-scoped token.
Layered defense: UC group membership controls which data the SP can read. Token scope controls which API surface the SP can reach. Compromise one layer and the other still limits the blast radius.
Scoped OAuth secrets: M2M service principals can now issue scoped OAuth secrets restricted to specific API scopes (for example sql, jobs, clusters) instead of blanket all-apis access. This reduces blast radius further: a compromised secret grants only the minimum API surface needed, never the full platform.
The same governance structure from this deck applies to model serving calls. Unity AI Gateway tracks the SP’s application ID as the caller identity.
Unity AI Gateway identifies the caller by the SP’s application ID in the JWT. Rate limits and spend caps apply at that identity level—not at the service-name level, which is mutable.
Set a daily spend cap on serving-api-sp. If the serving API has a runaway loop or bug, the cap triggers before the bill escalates. Other SPs are unaffected.
-- All inference calls from an SP through Gateway SELECT timestamp, client_id, -- SP application UUID model_name, input_tokens, output_tokens, latency_ms FROM system.ai_gateway.usage WHERE client_id = '<sp-application-uuid>' AND timestamp >= now() - interval 7 days ORDER BY timestamp DESC;
Complete audit visibility: every model inference call from the SP appears in system.ai_gateway.usage with token counts, latency, and the SP’s identity. No custom instrumentation needed.
-- Grant privileges to the group (not the SP directly) GRANT USE CATALOG ON CATALOG prod TO GROUP pipeline-readers; GRANT SELECT ON SCHEMA prod.bronze TO GROUP pipeline-readers; -- Add the SP to the group (one line) -- Done via Databricks Groups API or UI -- No GRANT statement needed -- Remove SP from group = instant cutoff -- No REVOKE statement needed -- No redeployment of the service
A new batch job needs the same bronze access. Add batch-loader-sp to pipeline-readers. One API call. No new GRANT statements. Audit shows a separate identity.
-- Who is in each SP group? Track membership changes SELECT group_name, member_name, member_type, change_time FROM system.access.group_membership WHERE group_name IN ( 'pipeline-readers', 'api-readers', 'audit-readers' ) ORDER BY change_time DESC;
Group membership is a system table—queryable, joinable, alertable. You can build a Databricks SQL dashboard that shows every SP group and its current members without leaving the platform.
-- current_user() for a service principal -- returns the SP's application ID (UUID) -- NOT a human email address SELECT current_user(); -- returns: "a1b2c3d4-e5f6-7890-abcd-ef1234567890" -- is_member() evaluates SP's group membership -- works identically for SPs and humans SELECT is_member('compliance-readers'); -- returns: true (if SP is in the group)
Key call-out: current_user() returns the SP’s UUID-format application ID when called by an SP. Audit queries and row filters must handle both email-format (human) and UUID-format (SP) identities as valid principals.
Important: is_member() checks workspace-level groups only, not account-level groups. For M2M, ensure the SP is added to workspace groups explicitly. For OBO with account-level groups, use an allowlist table with current_user() instead.
-- Row filter: audit-readers group sees all audit data -- Other SPs see nothing (access denied) CREATE FUNCTION audit_table_filter( audit_type STRING ) RETURNS BOOLEAN RETURN is_member('audit-readers'); ALTER TABLE system.access.audit SET ROW FILTER audit_table_filter ON (event_type);
Column masks work identically: an SP querying masked columns goes through the same RLS/masking evaluation path as a human. The mask function sees the SP’s UUID via current_user().
Visible in process listings, container environment dumps, and CI logs. One printenv away from a breach.
Even “private” repos are scanned by credential detection tools. One accidental fork or public push and the secret is permanently in git history.
A config file on a shared NFS or S3 bucket grants implicit access to every service and person who can read the bucket. This is a shared credential problem in disguise.
# Create a secret scope (backed by cloud KMS) databricks secrets create-scope \ --scope pipeline-sp-secrets # Store the SP client secret databricks secrets put-secret \ --scope pipeline-sp-secrets \ --key client-secret \ --string-value <secret-value>
{
"env_vars": {
"DATABRICKS_CLIENT_SECRET":
"{{secrets/pipeline-sp-secrets/client-secret}}"
}
}
Secret scope values are never logged, never returned by the API. The {{secrets/...}} reference is resolved at runtime inside the Databricks execution environment. Backed by Azure Key Vault, AWS Secrets Manager, or GCP Secret Manager.
-- All data access by a specific SP SELECT event_time, action_name, request_params.full_name_arg AS resource, source_ip_address FROM system.access.audit WHERE user_identity.email = '<sp-application-uuid>' AND event_time >= now() - interval 7 days AND service_name = 'unityCatalog' ORDER BY event_time DESC;
-- Track group membership changes for SPs SELECT change_time, group_name, member_name, change_type -- ADD or REMOVE FROM system.access.group_membership_changes WHERE group_name IN ( 'pipeline-readers', 'api-readers', 'audit-readers' ) ORDER BY change_time DESC;
-- SP token issuance (auth events) SELECT event_time, user_identity.email AS sp_identity, action_name, source_ip_address, request_params.scope FROM system.access.audit WHERE service_name = 'accounts' AND action_name = 'tokenLogin' AND event_time >= now() - interval 1 day ORDER BY event_time DESC;
Zero custom instrumentation: the audit trail is complete from day one. Group membership, data access, and token issuance are all in system tables, queryable with standard SQL, alertable via Databricks SQL dashboards.
The choice is about identity semantics, not convenience. When downstream RLS must enforce per-human access, the SP cannot substitute.
Runs on schedule, no human intervention, no per-human data scope required. The SP identity is semantically correct: the system speaks for itself, not for any individual person.
The human triggers the query. UC row filters must evaluate that human’s group membership. Using an SP here would bypass per-user RLS: all humans would see the same data as the SP’s group allows.
Generates compliance reports on schedule. No human is requesting this specific execution. Audit trail should show the SP’s identity, making it clear which service performed each operation.
A user clicks “Export my team’s data.” The UC row filter must respect that user’s group membership. The call must propagate the user’s identity via OBO token exchange to enforce per-user access control.
In the Databricks account console, create an SP for each independent service: one for the pipeline, one for the serving API, one for reporting. Name them by function, not by team or environment.
pipeline-readers, api-readers, compliance-readers. The group is the unit of governance—the SP is just a member.
GRANT SELECT ON CATALOG prod TO GROUP pipeline-readers. Never grant directly to the SP’s application ID.
Via the Databricks Groups API or account console. The SP now inherits all group privileges without any additional GRANT statements.
Never in environment variables, config files, or source code. Reference it in job configs as {{secrets/scope-name/key-name}}.
sql for read-only services. all-apis only for services that must start jobs or manage compute.
Use is_member('group-name') in row filter functions. No SP-specific code—the filter works for any member of the group, human or SP.
Run the three queries from Slide 12: SP data access, group membership changes, token issuance events. If all three return expected rows, the identity chain is working end to end.
system.access.audit has one identity for all data access. Impossible to distinguish a compliance read from an ETL write. Forensics are guesswork.system.access.audit has three distinct identities. Every read, write, and token issuance maps to exactly one service. Forensics take minutes, not days.compliance-sp has SELECT only on system.access.audit. It cannot write bronze data even if its token is stolen and the code is modified.