Databricks on Enterprise Azure: What Nobody Tells You

I’m six months into running Databricks on Azure for an ML-at-scale project at a Big Four consultancy. The platform is brilliant. The Spark runtime is fast, the notebook experience is slick, and the cluster management is genuinely better than running your own Spark on YARN. I’ve drunk the Kool-Aid on the core technology.

But the gap between the Databricks sales demo and running it inside enterprise Azure with real security, networking, and cost constraints is the width of the Clyde. Nobody talks about this gap, because the sales team shows you the public workspace version, your CTO nods enthusiastically, and then your platform engineering team spends three months figuring out why clusters won’t start.

This post is about that gap.

VNet injection: the theory

Azure Databricks runs on a managed infrastructure — Microsoft provisions the compute, manages the control plane, and you get a workspace. By default, this runs in a Microsoft-managed virtual network that you don’t control. For a sandbox or a proof of concept, this is fine. For an enterprise environment where the security team has opinions about every packet leaving the network, it’s a non-starter.

The answer is VNet injection. You deploy the Databricks workspace into your own Azure Virtual Network. Your clusters run in your subnets, subject to your network security groups, routed through your network appliances. The Databricks control plane still lives in Microsoft’s tenancy, but the data plane — the actual compute — sits in your network.

In theory, this gives you the best of both worlds: managed Spark with enterprise network controls. In practice, it gives you a networking puzzle that will consume your platform team for weeks.

VNet injection: the reality

Here’s what the documentation doesn’t adequately prepare you for.

Subnet requirements

Databricks needs two dedicated subnets in your VNet — one for the “public” nodes and one for the “private” nodes (these are Databricks concepts, not Azure public/private). Both subnets must be delegated to Microsoft.Databricks/workspaces. They cannot be shared with any other service. The minimum CIDR for each is /26, but if you’re running any reasonable number of concurrent clusters, you’ll want /24 or larger.

In an enterprise environment, IP address space is political. Every team wants a /16, nobody wants to explain to the networking team why Databricks needs two dedicated subnets that can’t be shared. I’ve spent more time in meetings about CIDR ranges than I have on actual Spark tuning.

NSG rules that break everything

When you inject into a VNet, you need specific Network Security Group rules for the Databricks control plane to communicate with your clusters. The required rules are documented, but they include allowing traffic from the Databricks control plane on specific ports, and the list of required IPs varies by region. Miss one rule, or have an overzealous security team tighten the NSGs beyond what Databricks needs, and cluster creation silently fails.

I say “silently” because the error messages are useless. You’ll see “Cluster terminated. Reason: Cloud Provider Launch Failure” in the workspace, which tells you precisely nothing. The actual cause is buried in the Azure activity log, if you know where to look, or in the cluster’s event log if it managed to partially start. Nine times out of ten, it’s an NSG rule blocking traffic from the control plane.

UDR requirements and the firewall fight

If you have a network virtual appliance — an Azure Firewall, a Palo Alto, a Checkpoint — sitting as the default route for your subnets, congratulations, you’ve just entered UDR hell. Databricks requires specific User Defined Routes to ensure traffic to the control plane doesn’t go through your firewall (or if it does, that your firewall allows it). The required destinations include Azure-specific service tags and Databricks-specific endpoints.

The security team at this consultancy had a default-deny posture on the firewall. Every outbound connection had to be explicitly whitelisted. Reasonable, responsible, and completely incompatible with Databricks out of the box. We spent two weeks working with the networking team to identify every endpoint Databricks clusters need to reach: the control plane, the artifact server, the log delivery endpoint, the DBFS storage account, Maven Central (for library installs), PyPI, and a handful of Azure service endpoints.

Every. Single. Endpoint. Manually whitelisted. And if Databricks changes an endpoint in an update — which they do — your clusters break and you’re back in the firewall rule spreadsheet.

Cluster startup times

On a public Databricks workspace with default networking, a cluster starts in three to four minutes. That’s already slow enough to break your flow if you’re iterating on notebooks, but it’s tolerable.

In our VNet-injected, firewall-routed, NSG-locked environment? Ten to fourteen minutes. Sometimes longer. The cluster has to pull container images through the firewall, contact the control plane through the UDRs, and negotiate with every NSG rule along the way. Each step adds latency, and there’s no parallelism — it’s sequential.

Fourteen minutes to start a cluster. Data scientists sitting with their coffee, watching the spinner, waiting to run a cell that takes thirty seconds. This is the enterprise Databricks experience, and it’s the thing nobody mentions in the architecture review.

We mitigated this somewhat with cluster pools — pre-provisioned idle instances that can be attached to a cluster faster. But cluster pools cost money even when idle, and convincing finance to pay for compute that’s sitting doing nothing because your networking stack is too slow is a conversation I don’t recommend.

Secret management: the early days

In mid-2019, the options for secret management in Databricks on Azure are… limited. Databricks has its own secret scopes, backed either by the Databricks internal store or by Azure Key Vault. The Key Vault integration exists, but it’s clunky.

To create a Key Vault-backed secret scope, you have to hit a specific URL in your workspace (#secrets/createScope) — there’s no UI for it in the workspace settings. You pass in the Key Vault resource ID and DNS name. It works, but it feels bolted on. And the permissions model is Databricks-native (MANAGE, READ, WRITE), not Azure RBAC. So you end up with two permission systems: Key Vault access policies for who can put secrets in, and Databricks secret ACLs for who can read them out. Two systems, two permission models, double the chance of getting it wrong.

In notebooks, you access secrets with dbutils.secrets.get(scope="my-scope", key="my-key"). It works. But there’s no way to reference secrets in cluster environment variables without wrapping them in init scripts, which means your data scientists need to know about dbutils.secrets and use it explicitly. Coming from a world where environment variables and managed identities handle this transparently, it feels primitive.

Workspace isolation: the organisational question

Here’s a question that doesn’t have a good answer right now: how do you isolate teams within Databricks?

The workspace is the unit of isolation. Within a workspace, you have folders with access control, clusters with permissions, and jobs with ownership. But there’s no concept of a tenant or a project boundary within a workspace. If two teams share a workspace, they can see each other’s clusters (even if they can’t use them), they share the same Hive metastore, and there’s limited ability to enforce data access boundaries.

The alternative is one workspace per team. This gives you hard isolation — separate networks, separate storage, separate metastores. It also gives you workspace sprawl, duplicated infrastructure, no shared data catalogue, and a management overhead that scales linearly with the number of teams.

At the consultancy, we went with one workspace per project stream. Three projects, three workspaces. It worked for isolation, but killed any possibility of cross-project data sharing. The data engineering team maintained ETL pipelines in workspace A that produced datasets needed by the ML team in workspace B. The solution? Export to ADLS Gen2 and mount it in both workspaces. A shared storage layer outside of Databricks, because Databricks had no native way to share a governed data catalogue across workspaces.

I’ve heard rumours that Databricks is working on a unified governance layer — something that would manage data access across workspaces. It can’t come soon enough. The current model forces an ugly choice between isolation and collaboration, and neither option is good.

Cost management: DBUs and the art of surprise

Databricks pricing is measured in Databricks Units (DBUs), which are layered on top of the underlying Azure VM costs. So you pay for the VM, and then you pay Databricks a premium on top, and the premium varies by workload type (Data Engineering, Data Analytics, Data Engineering Light) and by VM type.

Understanding what you’re actually going to pay is genuinely difficult. The pricing page is a matrix of SKU types and DBU rates that requires a spreadsheet to decode. I’ve built more Databricks cost models than I care to admit, and every single one was wrong by at least 20% because actual usage never matches the projections.

The real cost killer is autoscaling. Databricks clusters can autoscale from a minimum to a maximum number of workers based on load. Sounds great. But “load” is defined by pending tasks in the Spark scheduler, not by actual resource utilisation. A cluster can scale to its maximum because Spark has queued a bunch of tasks, even if the tasks are I/O-bound and would run fine on half the nodes. I’ve watched a cluster scale from 4 to 20 nodes for a job that was waiting on a slow ADLS read.

The fix is to set aggressive autoscaling timeouts and to right-size your maximums per workload. But this requires understanding your workloads, which requires monitoring, which requires the Spark UI, which brings me to…

Debugging: welcome to the Spark UI

The Spark UI is, in 2019, still the best tool for understanding what your Spark jobs are actually doing. Databricks wraps it in a slightly nicer interface than bare Spark, but the fundamentals are the same: a DAG visualisation, stage-level metrics, executor logs, and SQL query plans.

It’s powerful. It’s also impenetrable to anyone who hasn’t spent time learning how Spark executes queries. Data scientists who come from pandas or R don’t think in terms of stages, partitions, and shuffle operations. They write a DataFrame transformation, it runs slowly, and they have no idea why. The Spark UI can tell them — but only if they understand what they’re looking at.

There’s no Databricks-native query profiler that says “this join is slow because your data is skewed” or “this write is slow because you have too many small files.” You have to infer it from the Spark UI metrics. For experienced Spark engineers, this is fine. For the broader team, it’s a wall.

Notebook version control: the pain

Let me be brief about this because it makes me angry. In mid-2019, notebook version control in Databricks is essentially non-existent in any meaningful sense.

Databricks has a built-in revision history — it auto-saves revisions as you work. This is not version control. This is Google Docs-style revision tracking with no branches, no diffs worth reading, no pull requests, and no code review workflow.

You can export notebooks to files and commit them to Git manually. Some teams set up scripts to sync Databricks notebooks to a Git repo. It works the way any manual process works: inconsistently, with gaps, and only when someone remembers.

There’s a Databricks CLI that can export/import workspaces programmatically, and we built a CI/CD pipeline that exported notebooks to Azure DevOps. It felt like building plumbing that should have come with the house.

For a platform that positions itself as the centre of your data engineering and data science workflow, the lack of native Git integration is baffling. I’ve been told it’s on the roadmap. In the meantime, we’re exporting JSON files and writing shell scripts.

So is it worth it?

Yes. Despite everything I’ve just written, Databricks is the best managed Spark platform available. The runtime is fast — genuinely faster than vanilla Apache Spark, thanks to their Photon engine and other optimisations. The notebook experience, for all its version control failings, is collaborative and responsive. Delta Lake, which Databricks just open-sourced, promises to solve a lot of the data reliability problems that plague data lakes. Auto-scaling, even with its cost quirks, is better than manually managing YARN clusters.

The core technology is excellent. The enterprise integration story is where the pain lives. Networking, security, governance, cost visibility — these are the things that enterprise customers need and that the platform hasn’t fully addressed yet. If you’re evaluating Databricks for an enterprise Azure deployment, budget at least as much time for the platform engineering as for the actual data engineering. The ratio at this project was roughly 40% platform, 60% data work, and that 40% was all networking, security, and integration.

The sales demo shows you the 60%. Nobody shows you the 40%. Now you’ve been warned.