Microsoft Fabric has shifted how modern data teams approach unified analytics. By bringing storage, compute, and transformation under OneLake, implementing a Medallion Architecture (Bronze $\rightarrow$ Silver $\rightarrow$ Gold) becomes significantly more streamlined—if designed with clear boundaries between layers.

Here is a practical breakdown of how to structure an enterprise-grade Medallion architecture in Fabric using PySpark, dbt, and Delta Lake.

1. The Bronze Layer: Raw Ingestion & Source Fidelity

The goal of the Bronze layer is simple: capture raw source data in its native format with zero business logic applied.

• Pattern: Append-only ingestion using Fabric Pipelines or Eventstreams.

• Storage: Raw Delta tables stored in a Fabric Lakehouse.

• Best Practice: Always retain source metadata. Include lineage columns like _ingested_at_utc, _source_file_name, and _batch_id.

2. The Silver Layer: Cleansing, Conforming, and dbt Modeling

The Silver layer acts as your enterprise “single source of truth.” This is where data is cleaned, schema-enforced, deduplicated, and conformed across disparate source systems.

• Pattern: Incremental transformations using PySpark or dbt on top of Delta tables.

• Key Tasks:

  • Deduplication and late-arriving record handling using MERGE INTO.

  • Schema validation and type casting (e.g., parsing strings into ISO timestamps).

  • Identity resolution (matching matching entities across distinct systems).

• dbt Integration: Running dbt against the Fabric SQL Analytics endpoint or Warehouse allows you to maintain modular SQL models, automated testing (not_null, unique), and documentation.

3. The Gold Layer: Star Schemas & High-Performance Analytics

The Gold layer transforms conformed Silver data into business-ready dimensional models (Fact and Dimension tables) optimized for Power BI Direct Lake mode and ad-hoc SQL querying.

• Pattern: Star schema design using dbt or SparkSQL.

• Storage: Fabric Warehouse or Gold Lakehouse.

• Optimization:

  • Ensure string keys are mapped to integer Surrogate Keys for optimal join performance.

  • Leverage Fabric’s V-Order compression on Delta files to accelerate Direct Lake query performance in Power BI.

Key Takeaways & Lessons Learned

1. Keep Compute Modular: Decouple raw ingestion pipelines from heavy downstream transformations.

2. Standardize Delta Maintenance: Schedule automated OPTIMIZE and VACUUM tasks on your Delta tables to handle small-file problems caused by high-frequency streaming or micro-batching.

3. Governance First: Utilize Fabric item-level permissions and Microsoft Purview cataloging early in the architecture build—not as an afterthought.