LLI Wiki — My Digital Knowledge Base

Microsoft Fabric from a DBA's point of view: the good ideas and the sticking points

My opinion after the Fabric courses: bronze, silver, gold layers and Spark, yes; drag and drop, AI everywhere and workspace-level security, less so.

I took the Microsoft Fabric courses. It didn't click, but not everything should be thrown away. Here's my opinion as a DBA who prefers a script to boxes connected by arrows. It's an opinion, not the truth: a data scientist or a BI team will probably reach a different conclusion.

Fabric in two minutes

Fabric is a SaaS analytics platform: everything in one service, billed by capacity.

Building block Role
OneLake A single data lake for the whole organisation. Data is stored in Delta (Parquet) format
Lakehouse Files and Delta tables, mostly worked on with Spark, and readable in SQL through a read-only endpoint
Warehouse A T-SQL data warehouse: tables, views, procedures, transactions
Data Factory Orchestration pipelines and Dataflows Gen2 (Power Query), drag and drop
Spark notebooks PySpark, Spark SQL, Scala… on the same data
Power BI Reports, which can read OneLake directly (Direct Lake mode)
Copilot AI, more or less everywhere in the interface

What I like

A bronze-to-silver step, in PySpark in a notebook attached to a lakehouse:

from pyspark.sql import functions as F

bronze = spark.read.option("header", "true").csv("Files/raw/orders/")   # raw, as received

silver = (bronze
          .filter(F.col("order_id").isNotNull())
          .dropDuplicates(["order_id"])
          .withColumn("order_date", F.to_date("order_date")))

silver.write.format("delta").mode("overwrite").saveAsTable("silver_orders")

Eight lines, versionable, readable the next day. That's what I call ETL.

What gets stuck (for me)

My verdict

Fabric takes up good ideas: bronze, silver and gold layers, Spark, a single copy of the data. But the packaging (drag and drop, AI everywhere, coarse security) isn't made for me. Probably great for a data scientist or a BI team that wants to move fast; not my style. I've more or less dropped it, and we're still good friends.

The same ideas, with code: Data Vault, Snowflake and bronze, silver, gold layers.

Microsoft sources:

Mastodon LLIMastodon SocialFosstodon