← All handbooks

Databricks Handbook

The Databricks Lakehouse end to end — Apache Spark for distributed compute, Delta Lake for ACID tables on object storage, the medallion pattern for pipelines, plus jobs, streaming, and Unity Catalog governance.

01 Platform architecture

• Control plane (Databricks-managed: UI, jobs, notebooks) vs data plane (compute + storage in your cloud account) — your data stays in your VPC/storage.
• Workspace: notebooks, repos, jobs, SQL editor, dashboards.
• Compute: all-purpose clusters (interactive), job clusters (ephemeral, per-run, cheaper), SQL warehouses (BI), serverless.
• Storage: cloud object store (S3/ADLS/GCS) holding Delta/Parquet; accessed via Unity Catalog volumes & external locations.

02 Spark fundamentals

Spark distributes work across a cluster: a driver plans, executors run tasks on partitions in parallel.

• RDD → DataFrame/Dataset: use DataFrames (optimized via Catalyst + Tungsten); RDDs are low-level legacy.
• Transformations are lazy (select, filter, join, groupBy) — build a DAG; actions (count, collect, write, show) trigger execution.
• Partitions are the unit of parallelism. Narrow transforms (map/filter) stay within a partition; wide transforms (join/groupBy) cause a shuffle (network exchange) — the main cost.
• Catalyst optimizes the logical/physical plan; AQE (Adaptive Query Execution) re-optimizes at runtime (coalesce partitions, skew join handling).

03 DataFrames & Spark SQL

from pyspark.sql import functions as F
df = spark.read.format("delta").table("bronze.orders")
out = (df.filter(F.col("status") == "paid")
         .groupBy("region")
         .agg(F.sum("amount").alias("rev"),
              F.countDistinct("customer_id").alias("cust")))
out.write.mode("overwrite").saveAsTable("gold.region_rev")

-- same thing in Spark SQL
SELECT region, sum(amount) rev, count(distinct customer_id) cust
FROM bronze.orders WHERE status='paid' GROUP BY region;

DataFrame API and SQL compile to the same plan — use whichever reads clearer. Window functions, joins, and UDFs all available; prefer built-in functions over Python UDFs (UDFs break Catalyst & serialize row-by-row).

04 Delta Lake

Delta = Parquet data files + a transaction log (_delta_log) giving warehouse features on the lake.

• ACID transactions — concurrent reads/writes are safe; no half-written tables.
• Time travel — query past versions: SELECT * FROM t VERSION AS OF 12 or TIMESTAMP AS OF.
• Schema enforcement & evolution (mergeSchema).
• MERGE / upsert, UPDATE, DELETE on the lake (impossible in plain Parquet).
• OPTIMIZE (compact small files) + Z-ORDER (co-locate by column) + VACUUM (remove old files).

MERGE INTO silver.customers t
USING updates s ON t.id = s.id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

OPTIMIZE silver.customers ZORDER BY (region);

05 Medallion architecture

Each layer is Delta tables; pipelines promote data bronze→silver→gold. Keep bronze immutable so you can reprocess when logic changes.

06 Structured Streaming

(spark.readStream.format("cloudFiles")            # Auto Loader
      .option("cloudFiles.format", "json")
      .load("s3://lake/raw/")
   .writeStream.format("delta")
      .option("checkpointLocation", "/chk/orders")
      .trigger(availableNow=True)
      .table("bronze.orders"))

• Same DataFrame API for batch and stream (“unbounded table” model). Checkpoints give exactly-once + recovery.
• Auto Loader incrementally ingests new files efficiently. trigger(availableNow=True) = process backlog then stop (cheap micro-batch).
• Stateful ops (windowed aggregations, joins) need watermarks to bound state.

07 Delta Live Tables (DLT)

Declarative pipelines: you define tables and their transforms; DLT manages dependencies, orchestration, retries, and data quality.

import dlt
@dlt.table
def silver_orders():
    return dlt.read_stream("bronze_orders").dropDuplicates(["id"])

@dlt.expect_or_drop("valid_amt", "amount >= 0")   # quality constraint
@dlt.table
def gold_clean(): ...

Expectations enforce data quality (drop/fail/warn on bad rows). DLT auto-builds the dependency DAG and handles incremental processing.

08 Jobs & clusters

• Jobs orchestrate tasks (notebooks, scripts, DLT, dbt) as a DAG with schedules, retries, alerts, parameters.
• Use job clusters (ephemeral) for scheduled work — cheaper than keeping an interactive cluster up.
• Right-size: worker type, autoscaling min/max, spot instances for non-critical, Photon (vectorized C++ engine) for speed.
• Parameterize by run date; make tasks idempotent so retries are safe.

09 Unity Catalog

• Central governance across workspaces: three-level namespace catalog.schema.table.
• Fine-grained access control (GRANT on catalog/schema/table/column), lineage, audit, and a searchable data catalog.
• Manages external locations & storage credentials, volumes for files, and delta sharing for cross-org sharing.
• Replaces the legacy Hive metastore; enables row/column-level security via dynamic views.

10 Optimization

• Minimize shuffles: filter early, select needed columns, broadcast small dimension tables (broadcast(df)) to avoid shuffle joins.
• Data skew: salt keys or rely on AQE skew handling; watch for one straggler task.
• Small-file problem: OPTIMIZE + Auto Optimize; partition by low-cardinality columns only; Z-ORDER high-cardinality filters.
• Cache reused DataFrames; enable Photon; tune partition count (~128–200MB per partition); avoid Python UDFs.
• Read the Spark UI: stages, shuffle read/write, skew, spill to disk.

11 Interview drills

• What is a Lakehouse; Delta vs Parquet; how time travel works.
• Narrow vs wide transforms; what causes a shuffle; lazy vs action.
• Medallion layers and why bronze stays immutable.
• How to fix data skew / small files; when to broadcast a join.
• Structured Streaming exactly-once; role of checkpoints & watermarks.
• What Unity Catalog governs; job cluster vs all-purpose cluster.