DP-700 Microsoft Fabric Data Engineer Associate Takeaways: #01
Adnane Aboutalib | · 3 min read
Table of contents
I completed the first two courses for the DP-700 Microsoft Certified: Fabric Data Engineer Associate certification. I'll share some key takeaways and discoveries, so if you're interested, bear with me for a few minutes of reading.
As I explored Microsoft Fabric SaaS in Azure, I was amazed by how they gathered different data engineering fundamentals into one place (data loading patterns, data architectures, and orchestration processes) with key features like no data duplication between the data lake and data warehouse, which is the Lakehouse concept.
The Lakehouse
I got curious about the Lakehouse and how it provides the flexibility to use either schema-on-read or schema-on-write approaches in one place without data replication, resolving the modern data architecture debate. For the context, a Lakehouse is a service you can create inside your workspace in Microsoft Fabric. It gives you the flexibility to combine both worlds: storing any kind of data in a data lake (OneLake) while performing analytics through a dimensional data model using T-SQL queries, reports, etc. It turns out that the Lakehouse uses OneLake (which uses ADLS Gen2) to store data in Delta format, which supports ACID transactions, schema enforcement, and time-travel capabilities.
Ingestion and compute
Fabric also offers robust data ingestion features, including Dataflow Gen2, pipelines for orchestration and monitoring, and Notebooks that natively support PySpark. Furthermore, Spark is integrated within Fabric as a fully managed compute engine. When you use Notebooks, Spark Job Definitions, or Lakehouse, Fabric automatically provisions and manages Spark pools (compute clusters) for you. You don't need to manage infrastructure, Fabric handles the Spark runtime, auto-scaling, and integration seamlessly.
Storage: OneLake
Returning to the storage layers of a Lakehouse: as I mentioned earlier, the Lakehouse is stored on OneLake, which is a unified data lake. OneLake uses ADLS Gen2 as its underlying storage, which is blob storage enhanced with Hierarchical Namespace (HNS) capabilities. This empowers it to efficiently retrieve, manipulate, and update data.
Medallion architecture
I explored setting up a medallion architecture in Fabric, which structures data across three layers: Bronze (raw), Silver (validated), and Gold (refined). Data progressively transforms through each layer, with the Gold layer optimized for analytics. Fabric lets you implement this using Lakehouses, adapting the design to fit your specific needs.
Real-time data: Eventhouse
Last but not least, Fabric handles real-time events data through Eventhouse, where you can create multiple KQL (Kusto Query Language) databases for storing and analyzing real-time data. I wrote effective KQL queries, used materialized views (which are pre-computed query results that automatically update with great query speed), and created stored functions in KQL databases that let you save query templates for repeating logic, and these functions run when triggered and can accept parameters.
The {n, n-1} levels
It's quite a stacked set of data engineering concepts. Understanding multiple abstraction layers together, the {n, n-1} levels, gives you more understanding and control over the layer you're operating in. You can actually visualize trade-offs and make better technical decisions.