databricks

Azure Databricks Security Best Practices

Production-ready, modular Terraform templates for secure Azure Databricks deployments.

📚 Complete Documentation → 🔑 Authentication Setup Guide → - New to Terraform? Start here!

---

🚀 Modular Terraform Structure

This repository provides production-ready, modular Terraform templates for Azure Databricks deployments with comprehensive documentation, UML diagrams, and troubleshooting guides.

📁 Repository Structure

adb4u/
├── docs/                      # 📚 All documentation centralized here
│   ├── README.md              # Documentation index
│   ├── 01-QUICKSTART.md       # Quick start guide
│   ├── 02-DEPLOYMENT-CHECKLIST.md # Pre-flight checklist
│   ├── 03-TRAFFIC-FLOWS.md    # Network traffic patterns
│   ├── 04-TROUBLESHOOTING.md  # ⚠️ Common issues & solutions
│   ├── guides/                # Additional guides
│   │   └── 01-SERVERLESS-SETUP.md
│   ├── modules/               # Module documentation
│   │   ├── 01-NETWORKING.md
│   │   ├── 02-WORKSPACE.md
│   │   ├── 03-UNITY-CATALOG.md
│   │   ├── 04-NCC.md
│   │   ├── 05-CMK.md
│   │   └── 06-SEP.md
│   └── patterns/              # Pattern-specific guides
│       ├── 01-NON-PL.md
│       └── 02-FULL-PRIVATE.md
│
├── deployments/               # Pre-built deployment patterns
│   ├── non-pl/                # ✅ Non-Private Link (Ready)
│   ├── full-private/          # ✅ Full Private (Ready)
│   ├── byor/                  # ✅ Bring Your Own Resources (Ready)
│   └── hub-spoke/             # 🚧 Hub-Spoke (Future)
│
├── modules/                   # Reusable Terraform modules
│   ├── networking/            # VNet, subnets, NSG, NAT
│   ├── workspace/             # Databricks workspace
│   ├── unity-catalog/         # Metastore, storage, credentials
│   ├── ncc/                   # Network Connectivity Config (serverless)
│   ├── key-vault/             # Azure Key Vault integration
│   ├── private-endpoints/     # Private Link endpoints
│   ├── service-endpoint-policy/ # Service Endpoint Policies
│   ├── security/              # Security modules (CMK, IP access lists)
│   └── monitoring/            # Monitoring and observability
│
└── archive/                   # Legacy content and templates
    └── LEGACY-CONTENT.md      # Historical reference

🎯 Deployment Patterns

1. Non-Private Link (Non-PL) ✅ Production Ready

• Control Plane: Public
• Data Plane: Private (NPIP)
• Egress: NAT Gateway
• Storage: Service Endpoints
• Serverless: NCC attached (Service Endpoints or Private Link)

👉 Quick Start Guide → 🚀 Serverless Setup → ⚠️ Troubleshooting Guide → - Review before deploying!

2. Full Private (Air-gapped) ✅ Production Ready

• Workspace Access: Private Link (SCC relay + API)
• Data Plane: Private (NPIP)
• Egress: None (isolated)
• Storage: Private Link
• Serverless: NCC attached (Private Link required)

👉 Pattern Documentation → 🚀 Serverless Setup → ⚠️ Troubleshooting Guide → - Common issues & solutions!

3. BYOR (Bring Your Own Resources) ✅ Production Ready

• Integrate with existing Azure infrastructure
• Bring your own VNet, Storage Account, Key Vault
• Customer-Managed Keys (CMK) for enhanced security
• Flexible configuration for existing environments

👉 BYOR Documentation →

4. Hub-Spoke with Firewall 🚧 Future

• Enterprise-scale multi-workspace deployments

✨ Key Features

• ✅ Secure Cluster Connectivity (NPIP): Always enabled
• ✅ Unity Catalog: Mandatory, regional metastore
• ✅ Network Connectivity Config (NCC): Mandatory for serverless compute
• ✅ Flexible Networking: Create new or BYOV
• ✅ Service Endpoint Policies: Enhanced storage security
• ✅ Customer-Managed Keys (CMK): Optional encryption control
• ✅ Private Link Support: Full private connectivity option
• ✅ BYOR Support: Integrate with existing infrastructure
• ✅ Modular Design: Reusable, composable components
• ✅ Well-Documented: Comprehensive guides in /docs

🚀 Quick Start

# Navigate to deployment
cd deployments/non-pl

# Configure
cp terraform.tfvars.example terraform.tfvars
# Edit terraform.tfvars with your values

# Deploy
export TF_VAR_databricks_account_id="<your-account-id>"
terraform init
terraform plan
terraform apply

Full guide: See docs/01-QUICKSTART.md

🚀 Serverless Compute

All deployments include Network Connectivity Configuration (NCC) for serverless SQL Warehouses and Serverless Notebooks.

Serverless Connectivity Options:

Pattern	Classic Clusters	Serverless Compute
Non-PL	Service Endpoints (VNet)	Service Endpoints or Private Link (via NCC)
Full Private	Private Endpoints (VNet)	Private Link (via NCC)
BYOR	Flexible (based on existing setup)	Configurable via NCC

Post-Deployment Setup:

• 📖 Serverless Setup: See docs/guides/01-SERVERLESS-SETUP.md

Key Points:

• ✅ NCC is mandatory (created automatically like Unity Catalog)
• ✅ Classic clusters work immediately after deployment
• ⏸️ Serverless requires additional configuration (manual approval for Private Link)

📚 Documentation

All documentation is centralized in the docs/ folder:

Getting Started:

• Quick Start Guide - Deploy your first workspace
• Deployment Checklist - Pre-flight validation
• Traffic Flows - Network traffic patterns and sequences
• Troubleshooting Guide - Common issues and solutions

Guides:

• Serverless Setup Guide - Enable SQL Warehouses & Notebooks

Pattern Documentation:

• Non-PL Pattern - Non-Private Link deployment
• Full Private Pattern - Air-gapped deployment

Module Documentation:

• Networking Module - VNet, subnets, NSG, NAT
• Workspace Module - Databricks workspace configuration
• Unity Catalog Module - Metastore and catalogs
• NCC Module - Network Connectivity Configuration
• CMK Module - Customer-Managed Keys
• SEP Module - Service Endpoint Policies

---

📦 Legacy Content

Historical content and diagrams have been archived. See archive/LEGACY-CONTENT.md for reference.

For new deployments, use the modular structure documented above.

---

Repository Version: 2.0