Skip to content

Build a Multi-Cloud Medallion Pipeline with DataCoolie Skills

This 35-minute case study follows a Wide World Importers pipeline from source discovery to local execution and three cloud targets. It demonstrates the official DataCoolie Skills workflow with the package and Skills available when the recording was made.

The recorded project workspace is private and is not required for this tutorial. Resource names and credentials are intentionally omitted; use your own accounts, identities, and environment-specific configuration.

Choose a language: English video · Video tiếng Việt

What the demo proves

  • Discover inspects the declared SQL Server source before architecture and metadata are finalized.
  • Design records watermark, load, model, runtime, and recovery decisions and pauses for approval when the material architecture changes.
  • Build produces canonical metadata, environment overlays, target-specific runners, immutable artifacts, and verification evidence.
  • Provision and Release remain separate, explicitly authorized outcomes for each cloud environment.
  • DataCoolie Studio reads the resulting project, metadata, lineage, sources, and operational evidence.

Architecture demonstrated

flowchart TB
  M["Canonical pipeline metadata"] --> O["Environment overlays"]
  O --> L["Local runner<br/>Polars + Delta"]
  O --> A["AWS runners<br/>Polars external + Glue Spark"]
  O --> F["Fabric runners<br/>Polars external/native + Spark"]
  O --> D["Databricks runner<br/>Spark + Unity Catalog"]
  L --> X["Selected stage execution"]
  A --> X
  F --> X
  D --> X

  S[("SQL Server source")] --> B["Bronze<br/>raw business data"]
  B --> V["Silver<br/>validated replicas"]
  V --> G["Gold<br/>selected business models"]

  X -. runs selected stage .-> B
  X -. runs selected stage .-> V
  X -. runs selected stage .-> G

The dotted lines summarize target participation rather than a fixed scheduler. Stages are selected by a runner; DataCoolie does not infer a cross-platform DAG.

Environment matrix

Environment Source → Bronze Bronze → Silver Silver → Gold Storage and catalog intent
Local Local host, Polars Local host, Polars Local host, Polars Local Parquet/Delta paths
AWS On-premises host, Polars with external AWSPlatform AWS Glue, Spark AWS Glue, Spark S3 for data/control paths; Glue databases for Silver and Gold
Fabric On-premises host, Polars with external FabricPlatform Fabric native Python, Polars Fabric notebook, Spark OneLake Bronze files; Lakehouse tables for Silver/Gold; separate ETL control Lakehouse
Databricks Bronze files handed off to a Unity Catalog Volume Databricks notebook/job, Spark Databricks notebook/job, Spark Catalog schemas for layers; dedicated Volume path for DataCoolie control state

This is portability through shared intent, not identical runtime configuration. Each environment supplies its own paths, catalogs, credentials, engine, and runner while preserving the canonical dataflow model.

Chapters

Topic English Tiếng Việt
Architecture and workflow 00:00 00:00
Setup 01:24 01:24
Build the project with Codex 02:06 02:06
Watermarks and Gold scope 03:49 03:49
Metadata and runners 04:47 04:47
DataCoolie Studio 06:42 06:42
Local execution 10:57 10:57
AWS design and deployment 12:59 12:59
Run on AWS 16:40 16:40
AWS in Studio 20:04 20:04
Microsoft Fabric 21:15 21:15
Run on Fabric 25:43 25:43
Databricks 28:28 28:28
Databricks in Studio 32:41 32:41
Conclusion 34:32 34:32

Continue with a target platform