databricks

Azure Databricks Deployment Documentation

Production-ready Azure Databricks workspace deployments with Unity Catalog, networking security, and enterprise features.

🚀 Quick Start

New to this repo? Start here:

Quickstart Guide - Deploy your first workspace in 15 minutes
Deployment Checklist - Prerequisites and preparation
Choose Your Pattern - Select based on your requirements

📋 Deployment Patterns

Choose the pattern that matches your security and connectivity requirements:

Non-Private Link

Best for: Most deployments, cost-effective, simpler setup

✅ Secure Cluster Connectivity (NPIP) - No public IPs on clusters
✅ Unity Catalog with managed identity
✅ Service Endpoint Policy for storage security
✅ NAT Gateway for outbound connectivity
✅ Customer-Managed Keys (optional)
⚡ Quick deployment - Single terraform apply

Use when: Standard security requirements, public control plane acceptable

Full Private

Best for: Highly regulated environments, complete isolation

✅ All features from Non-PL pattern, PLUS:
✅ Private Link for control plane (front-end)
✅ Private Link for data plane (back-end)
✅ Air-gapped deployment option
🔒 Complete isolation - No public endpoints

Use when: Zero-trust network, compliance requirements, complete air-gapping needed

🔧 Module Reference

Deep-dive into each infrastructure component:

Networking - VNet, subnets, NSG, NAT Gateway
Workspace - Databricks workspace configuration
Unity Catalog - Metastore, storage credentials, external locations
Network Connectivity (NCC) - Serverless compute networking
Customer-Managed Keys (CMK) - Encryption with your own keys
Service Endpoint Policy (SEP) - Storage egress control

📚 How-To Guides

Step-by-step instructions for specific tasks:

Serverless Setup

Enable Databricks SQL Warehouses and Serverless Notebooks with:

Service Endpoints (recommended)
Private Link via NCC (for complete isolation)

Applies to: Both Non-PL and Full-Private patterns

🛠️ Additional Documentation

Core Documentation

Traffic Flows - Network architecture and data flow diagrams
Troubleshooting - Common issues and solutions

Configuration

terraform.tfvars.example - Comprehensive configuration examples in each deployment folder
Authentication - Environment variables and service principal setup (see Quickstart)

📖 Documentation Structure

docs/
├── 01-QUICKSTART.md                 # Start here
├── 02-DEPLOYMENT-CHECKLIST.md       # Pre-deployment prep
├── 03-TRAFFIC-FLOWS.md              # Network architecture
├── 04-TROUBLESHOOTING.md            # Problem solving
│
├── patterns/                         # Deployment patterns
│   ├── 01-NON-PL.md                 # Non-Private Link
│   └── 02-FULL-PRIVATE.md           # Full Private
│
├── modules/                          # Component deep-dives
│   ├── 01-NETWORKING.md
│   ├── 02-WORKSPACE.md
│   ├── 03-UNITY-CATALOG.md
│   ├── 04-NCC.md
│   ├── 05-CMK.md
│   └── 06-SEP.md
│
└── guides/                           # How-to guides
    └── 01-SERVERLESS-SETUP.md

🎯 Common Workflows

First-Time Deployment

# 1. Review prerequisites
Read: 02-DEPLOYMENT-CHECKLIST.md

# 2. Choose pattern
Non-PL: docs/patterns/01-NON-PL.md
Full-Private: docs/patterns/02-FULL-PRIVATE.md

# 3. Configure
Edit: deployments/<pattern>/terraform.tfvars

# 4. Deploy
cd deployments/<pattern>
terraform init
terraform apply

# 5. Verify
terraform output deployment_summary

Enable Serverless

# After workspace deployment
Read: guides/01-SERVERLESS-SETUP.md

# Configure storage firewall or Private Link
# Test with SQL Warehouse

Enable CMK Encryption

# Configure in terraform.tfvars
enable_cmk_managed_services = true
enable_cmk_managed_disks    = true
enable_cmk_dbfs_root        = true

# Apply changes
terraform apply

# Verify
terraform output customer_managed_keys

Troubleshoot Issues

# Check troubleshooting guide
Read: 04-TROUBLESHOOTING.md

# Common issues:
# - Metastore deletion errors
# - Network connectivity
# - SEP/NCC destroy issues

🔍 Finding What You Need

“I want to…”

Deploy a workspace → Quickstart
Understand network flows → Traffic Flows
Choose security options → Deployment Patterns
Enable encryption → CMK Module
Control storage access → SEP Module
Enable serverless → Serverless Setup
Fix deployment issues → Troubleshooting

“I need to understand…”

How networking works → Networking Module
Unity Catalog setup → Unity Catalog Module
Workspace configuration → Workspace Module
Serverless connectivity → NCC Module

🏆 Best Practices

✅ DO:

Start with Non-PL pattern for most deployments
Enable Service Endpoint Policy by default
Use Unity Catalog for all workspaces
Tag all resources with owner and keep-until
Test destroy workflow in non-production first
Enable CMK for sensitive workloads

❌ DON’T:

Deploy without Unity Catalog (it’s mandatory)
Skip the deployment checklist
Disable Service Endpoint Policy without good reason
Forget to configure serverless connectivity
Delete resources manually (use terraform destroy)

💡 Tips for Success

For Beginners

Start with Quickstart
Use Non-PL pattern
Keep default settings initially
Follow the deployment checklist
Join the troubleshooting guide if stuck

For Advanced Users

Review pattern comparison for requirements
Customize networking via BYOV
Enable all security features (CMK, SEP, Private Link)
Plan for serverless from the start
Implement hub-spoke for multiple workspaces

For Production

Use service principal authentication
Enable all three CMK scopes
Test destroy workflow first
Document custom configurations
Set up monitoring and alerting
Plan for disaster recovery

📞 Getting Help

Documentation Resources

This docs/ folder - Complete reference
terraform.tfvars.example - Configuration examples
checkpoint/ - Implementation notes and history

External Resources

Support Channels

Check Troubleshooting Guide first
Review checkpoint documents for similar issues
Check provider documentation for recent changes
Contact your platform team or Databricks support

🚀 Ready to Deploy?

Start with the Quickstart Guide and have your first workspace running in 15 minutes!