databricks

Least Privilege Workspaces (LPW) VPC-SC Policies

Overview

This folder contains VPC Service Controls policies for Least Privilege Workspaces (LPW) - a special, highly restricted Databricks workspace configuration that requires explicit allowlisting.

What are Least Privilege Workspaces?

Least Privilege Workspaces are Databricks deployments with:

When to Use LPW Policies

Use Least Privilege Workspace policies when:

Key Differences from Standard Policies

Aspect Standard Workspaces Least Privilege Workspaces (LPW)
Security Posture Secure (recommended for most) Maximum security (explicit allowlist)
Setup Complexity Standard High - requires Databricks approval
VPC-SC Rules Comprehensive but flexible Extremely restrictive
Use Cases Most production deployments Highly regulated environments
Support Self-service Requires Databricks engagement

Files in This Folder

create-ws-lpw.yaml

VPC-SC policy for creating Least Privilege Workspaces during workspace creation phase.

Key characteristics:

How to Use LPW Policies

Prerequisites

  1. Databricks Support Engagement:
    • Contact Databricks support to request LPW setup
    • Provide your security requirements and compliance needs
    • Receive approval and configuration guidance from Databricks
  2. Security Requirements Documentation:
    • Document your specific security constraints
    • List compliance frameworks you must meet (SOC 2, HIPAA, FedRAMP, etc.)
    • Identify data classification levels and handling requirements
  3. Technical Prerequisites:
    • VPC Service Controls configured and operational
    • Private Google Access with restricted.googleapis.com
    • VPC firewall rules following deny-by-default approach
    • Access Context Manager policies defined

Setup Process

  1. Coordinate with Databricks: ```
    • Engage Databricks support team
    • Share security requirements
    • Receive LPW-specific configuration guidance
    • Get approval for deployment ```
  2. Review and Customize Policies: ```
    • Review create-ws-lpw.yaml
    • Update with your project numbers and identities
    • Validate against your security requirements
    • Test in dry-run mode ```
  3. Deploy LPW Policies: 
    # Create Access Context Manager Access Level
gcloud access-context-manager levels create databricks_lpw_access \
  --basic-level-spec=lpw-access-level.yaml \
  --policy=$ACCESS_POLICY_ID

# Create VPC-SC Perimeter with LPW policy (dry-run first)
gcloud access-context-manager perimeters dry-run create databricks-lpw-perimeter \
  --title="Databricks LPW Perimeter" \
  --resources=projects/$PROJECT_NUMBER \
  --ingress-policies=create-ws-lpw.yaml \
  --policy=$ACCESS_POLICY_ID

# Test in dry-run mode
# Monitor logs for violations

# Enforce after validation
gcloud access-context-manager perimeters dry-run enforce databricks-lpw-perimeter \
  --policy=$ACCESS_POLICY_ID
  4. Create Workspace: ```
    • Create Databricks workspace using UI or API
    • Monitor workspace creation closely
    • Validate all resources created successfully
    • Test cluster launch and basic operations ```
  5. Post-Creation Configuration: ```
    • After workspace creation, update to operational LPW policies
    • Add egress rules specific to your LPW requirements
    • Continue monitoring VPC-SC logs
    • Coordinate any policy adjustments with Databricks support ```

Important Notes

Not a Self-Service Feature

⚠️ WARNING: Least Privilege Workspaces require Databricks support engagement:

Testing Requirements

Before production deployment:

  1. Dry-Run Testing: Always test policies in dry-run mode first
  2. Non-Production Validation: Create test workspace in dev/test environment
  3. Log Monitoring: Continuously monitor VPC-SC audit logs
  4. Databricks Validation: Have Databricks support review your configuration

Maintenance

LPW deployments require ongoing maintenance:

Troubleshooting

Common Issues

Issue Cause Solution
Workspace creation fails LPW policy too restrictive Review VPC-SC logs, coordinate with Databricks support
Cluster launch fails Missing egress rules for runtime images Add required egress rules with Databricks guidance
Access denied errors Insufficient permissions in LPW policy Review and expand policy with Databricks approval
Policy conflicts LPW policy incompatible with VPC-SC setup Align policies with Databricks LPW requirements

Debug Commands

# Check VPC-SC violations specific to LPW perimeter
gcloud logging read "protoPayload.metadata.@type=type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata AND protoPayload.metadata.securityPolicyInfo.servicePerimeterName:databricks-lpw-perimeter" \
  --limit=50 \
  --format=json

# Describe LPW perimeter
gcloud access-context-manager perimeters describe databricks-lpw-perimeter \
  --policy=$ACCESS_POLICY_ID

# List access levels for LPW
gcloud access-context-manager levels list \
  --policy=$ACCESS_POLICY_ID \
  --filter="name:lpw"

Comparison: Standard vs LPW Policies

Standard Policies (Regular Workspaces)

Located in: /templates/vpcsc-policy/

Files:

Use when: Most production deployments

LPW Policies (This Folder)

Located in: /templates/vpcsc-policy/least-privilege-workspaces/

Files:

Use when: Maximum security, explicit allowlisting required

Migration Path

From Standard to LPW

If you need to migrate from standard to LPW:

  1. Contact Databricks Support: Required for LPW enablement
  2. Review Current Setup: Document existing policies and configurations
  3. Plan Migration: Work with Databricks to plan migration approach
  4. Test in Parallel: Create new LPW workspace, don’t migrate existing
  5. Gradual Transition: Move workloads incrementally after validation

From LPW to Standard

Not recommended - typically LPW is used for compliance reasons that persist.

Support and Resources

Databricks Support

For LPW deployments, always engage Databricks support:

Documentation

Summary

Least Privilege Workspaces provide maximum security for Databricks deployments but require:

For most deployments, use the standard VPC-SC policies in the parent folder instead.

Only use LPW policies when you have specific regulatory requirements that mandate this level of security and have received approval from Databricks support.