DataCoolie examples

This library is organized by the folder that owns each example. If you are new to DataCoolie, start with Getting started, then return here for a focused project recipe; otherwise jump directly to the feature you need. Recipes link to the maintained project runner and explain the extracted-project working directory, dependencies, expected output and adaptation point.

• Projects: complete runnable project sources and their build layout.
• Configuration: Driver/provider construction and run options.
• Dataflows: focused metadata, SQL and transformation patterns.
• Runners: host-specific entrypoints that call the Driver.
• Operations: replay, recovery and maintenance wrappers.
• Plugins: extension source and packaging examples.

The table actions use one vocabulary. The action describes what the link does; it does not describe the file's business meaning:

Action	Meaning	Scope
source	Opens a generated, readable source projection in the docs. Notebook outputs are omitted.	One file
raw	Opens the exact canonical file content/bytes. It may render inline; it is not a project archive.	One file
download	Downloads the complete project archive, currently a .zip.	One project
project-files	Jumps to the complete project section in this catalog. It does not download anything.	One project
guide	Opens usage, adaptation and verification instructions.	Feature or project

Use raw when a tool needs the original bytes and source when a person or agent needs the readable projection. A browser saving a raw response is a transport detail, not a separate project download. Standalone files do not have a download action; complete projects use download once at the project root.

Project runners are executable recipes. Focused configuration and metadata files are snippets or authoring templates unless their page labels a runnable local contract fixture, such as provider_fixtures.py. Managed-host runners are host contracts. The WWI medallion walkthrough is a case study; it is not a local project fixture.

The Markdown in this page is the authored inventory. During a docs build, folder trees and the first column of each marked table are rendered with visible depth prefixes. The links remain ordinary Markdown links so agents and offline readers can follow them without JavaScript. Section anchors are stable navigation targets; file identity comes from the canonical path, not from a displayed tree prefix.

Projects

Complete projects preserve their authored internal layout. Projects contain metadata and any input or function/query companions needed by the sample. Most include an environment runner; Platform smoke instead uses the separate canonical host runners linked below. Runtime output folders are intentionally not part of the public inventory.

projects/
├── artifact/
│   ├── data/
│   │   └── input/
│   │       └── orders.csv
│   ├── metadata/
│   │   ├── dataflows/
│   │   │   └── orders_query.json
│   │   ├── connections.json
│   │   └── schema_hints.json
│   ├── queries/
│   │   └── orders.sql
│   ├── runners/
│   │   └── dev/
│   │       └── run.py
│   └── datacoolie.yml
├── function/
│   ├── data/
│   │   └── input/
│   ├── functions/
│   │   ├── __init__.py
│   │   ├── range_source.py
│   │   └── sources.py
│   ├── metadata/
│   │   ├── dataflows/
│   │   │   └── orders_function.json
│   │   ├── connections.json
│   │   └── schema_hints.json
│   ├── runners/
│   │   └── dev/
│   │       └── run.py
│   └── datacoolie.yml
├── getting-started/
│   ├── data/
│   │   └── input/
│   │       ├── customers/
│   │       │   └── customers.csv
│   │       └── orders/
│   │           └── orders.csv
│   ├── metadata/
│   │   ├── connections.json
│   │   ├── dataflows.json
│   │   └── schema_hints.json
│   ├── runners/
│   │   └── local/
│   │       ├── checks.py
│   │       ├── run_polars.py
│   │       └── run_spark.py
│   └── datacoolie.yml
├── incremental/
│   ├── data/
│   │   └── input/
│   │       └── orders/
│   │           └── orders.csv
│   ├── metadata/
│   │   ├── dataflows/
│   │   │   └── orders_incremental.json
│   │   ├── connections.json
│   │   └── schema_hints.json
│   ├── runners/
│   │   └── dev/
│   │       └── run.py
│   └── datacoolie.yml
├── platform-smoke/
│   ├── data/
│   │   └── input/
│   │       └── orders/
│   │           └── orders.csv
│   ├── metadata/
│   │   ├── dataflows/
│   │   │   └── orders_platform_smoke.json
│   │   ├── environments/
│   │   │   ├── aws-iceberg.json
│   │   │   ├── aws.json
│   │   │   ├── databricks.json
│   │   │   ├── fabric.json
│   │   │   └── local.json
│   │   ├── connections.json
│   │   └── schema_hints.json
│   └── datacoolie.yml
└── transform/
    ├── metadata/
    │   ├── dataflows/
    │   │   └── orders_clean.json
    │   ├── connections.json
    │   └── schema_hints.json
    ├── runners/
    │   └── dev/
    │       └── run.py
    └── datacoolie.yml

File/Folder	Description	Links
artifact/	Artifact-relative SQL project with a Polars runner.	guide · project-files · download
function/	Python function source project with automatic package handling.	guide · project-files · download
getting-started/	Typed orders onboarding project with Polars/Spark runners, customer refresh and Bronze-to-Silver continuation.	guide · project-files · download
incremental/	File-source project that persists an integer watermark.	guide · project-files · download
platform-smoke/	Three-row CSV-to-Delta fixture with local/cloud overlays and separate canonical runners.	guide · project-files · download
transform/	Small focused transformer project with a Parquet destination.	guide · project-files · download

Getting-started project

This project is the canonical source for the getting-started guides. It contains a typed orders input, a separate customers full-refresh branch and an orders Bronze-to-Silver partitioned-detail branch. Download Getting-started project (download) for a complete checkout.

Use runners/local/run_polars.py for the shortest local path or runners/local/run_spark.py for a local Delta-enabled Spark session. The project-owned guards check input preconditions, intended flow selection, terminal status, persisted state and Delta output. Runtime .runtime/ and data/output/ directories are excluded from the public inventory.

projects/getting-started/
├── data/
│   └── input/
│       ├── customers/
│       │   └── customers.csv
│       └── orders/
│           └── orders.csv
├── metadata/
│   ├── connections.json
│   ├── dataflows.json
│   └── schema_hints.json
├── runners/
│   └── local/
│       ├── checks.py
│       ├── run_polars.py
│       └── run_spark.py
└── datacoolie.yml

File/Folder	Description	Links
datacoolie.yml	Local environment and metadata build contract.	source · raw
data/	Input area; generated runtime output is excluded.	—
── input/	Source fixtures used by the three lessons.	—
──── customers/	Customer full-refresh source folder.	—
────── customers.csv	Two-row customer full-refresh fixture.	source · raw
──── orders/	Orders incremental source folder.	—
────── orders.csv	Four-row orders fixture with one duplicate order ID.	source · raw
metadata/	FileProvider metadata root.	—
── connections.json	Input, Bronze, Silver and customer destination roots.	source · raw
── dataflows.json	Orders, customers and dependent Silver flow definitions.	source · raw
── schema_hints.json	Signed IDs, Decimal amount, timestamp and customer type hints.	source · raw
runners/	Project-owned execution entrypoints.	—
── local/	Local Polars and Spark runner environment.	—
──── checks.py	Tutorial-owned input, status, state and output guards.	source · raw
──── run_polars.py	Polars runner for all three lessons.	source · raw
──── run_spark.py	Spark runner with local Delta session lifecycle.	source · raw

Artifact project

This project demonstrates a FileProvider artifact root, artifact-relative SQL and qualified Polars relations. Open Artifact SQL project (download) when a complete checkout is needed. The tree below is generated from the canonical source.

The project runner is runners/dev/run.py (source · raw); it registers the Polars relations before executing the artifact-relative query.

projects/artifact/
├── data/
│   └── input/
│       └── orders.csv
├── metadata/
│   ├── dataflows/
│   │   └── orders_query.json
│   ├── connections.json
│   └── schema_hints.json
├── queries/
│   └── orders.sql
├── runners/
│   └── dev/
│       └── run.py
└── datacoolie.yml

File/Folder	Description	Links
datacoolie.yml	Project and environment build contract.	source · raw
data/	Reference fixture area kept outside generated runtime output.	—
── input/	Reference CSV retained with the project; the runner registers its relations in memory.	—
──── orders.csv	Reference CSV kept with the project; the runner registers its two SQL relations in memory.	source · raw
metadata/	FileProvider metadata root.	—
── connections.json	Connection definitions for the fixture.	source · raw
── dataflows/	Section-wrapped dataflow metadata.	—
──── orders_query.json	Dataflow using an artifact-relative SQL reference.	source · raw
── schema_hints.json	Explicit input schema hints.	source · raw
queries/	Project SQL root referenced by metadata.	—
── orders.sql	Qualified SQL query joined after runner table registration.	source · raw
runners/	Project-owned execution entrypoints.	—
── dev/	Development environment runner.	—
──── run.py	Local artifact runner invocation.	source · raw

Function project

This project demonstrates a Python function source. Its function root contains __init__.py, so the CLI automatic packaging rule creates a ZIP while the runner keeps the import prefix explicit. Open Function project (download) when a complete checkout is needed.

The package entrypoint is functions/init.py (source · raw).

projects/function/
├── data/
│   └── input/
├── functions/
│   ├── __init__.py
│   ├── range_source.py
│   └── sources.py
├── metadata/
│   ├── dataflows/
│   │   └── orders_function.json
│   ├── connections.json
│   └── schema_hints.json
├── runners/
│   └── dev/
│       └── run.py
└── datacoolie.yml

File/Folder	Description	Links
datacoolie.yml	Project and environment build contract.	source · raw
data/	Optional input area for the function example.	—
── input/	Empty input area created for project layout consistency.	—
functions/	Project-owned Python function package root.	—
── __init__.py	Makes the configured function root importable.	source · raw
── sources.py	Python source callable referenced by metadata.	source · raw
── range_source.py	Separate custom-reader extension fixture; bundled with this package, but not selected by its dataflow.	guide · source · raw
metadata/	FileProvider metadata root.	—
── connections.json	Connection definitions for the fixture.	source · raw
── dataflows/	Dataflow metadata selecting the Python function.	—
──── orders_function.json	Dataflow using a packaged Python source.	source · raw
── schema_hints.json	Explicit input schema hints.	source · raw
runners/	Project-owned execution entrypoints.	—
── dev/	Development environment runner.	—
──── run.py	Local function project runner.	source · raw

Incremental project

This project demonstrates a CSV source, append destination and integer updated_sequence watermark. Run runners/dev/run.py (source · raw) once for two rows, run it again without changing the input to observe a no-change run, append a row with updated_sequence=3, then run it again with the same runtime root. The last run appends one row and advances the watermark. Open Incremental project (download) for a complete checkout.

projects/incremental/
├── data/
│   └── input/
│       └── orders/
│           └── orders.csv
├── metadata/
│   ├── dataflows/
│   │   └── orders_incremental.json
│   ├── connections.json
│   └── schema_hints.json
├── runners/
│   └── dev/
│       └── run.py
└── datacoolie.yml

File/Folder	Description	Links
datacoolie.yml	Project and environment build contract.	source · raw
data/	Input and output fixture area. Generated output is excluded.	—
── input/	Input partition containing ordered source data.	—
──── orders/	Orders source folder used by the file reader.	—
────── orders.csv	Source rows with advancing sequence values.	source · raw
metadata/	FileProvider metadata root.	—
── connections.json	Connection definitions for the fixture.	source · raw
── dataflows/	Dataflow metadata with incremental watermark rules.	—
──── orders_incremental.json	Incremental dataflow and watermark configuration.	source · raw
── schema_hints.json	Explicit input schema hints.	source · raw
runners/	Project-owned execution entrypoints.	—
── dev/	Development environment runner.	—
──── run.py	Runner used for repeated incremental loads.	source · raw

Transform project

This is the smallest built-in transform example: one dataflow, one synthetic input and one Parquet output. Its metadata focuses on column cleanup and projection. The project runner creates the input when the extracted project does not contain it. Open Transform project (download) for a complete checkout.

projects/transform/
├── metadata/
│   ├── dataflows/
│   │   └── orders_clean.json
│   ├── connections.json
│   └── schema_hints.json
├── runners/
│   └── dev/
│       └── run.py
└── datacoolie.yml

File/Folder	Description	Links
datacoolie.yml	Project and environment build contract.	source · raw
metadata/	FileProvider metadata root.	—
── connections.json	Connection definitions for the fixture.	source · raw
── dataflows/	Focused transformer metadata.	—
──── orders_clean.json	Dataflow that cleans and projects order columns.	source · raw
── schema_hints.json	Explicit input schema hints.	source · raw
runners/	Project-owned execution entrypoints.	—
── dev/	Development environment runner.	—
──── run.py	Local transform project runner.	source · raw

Platform smoke project

Use this three-row fixture for the local rehearsal and managed-platform handoff. The download includes input and metadata; obtain the selected runner through its separate raw action. CLI builds do not upload input or execute notebooks. Cloud variants are setup contracts checked locally.

projects/platform-smoke/
├── data/
│   └── input/
│       └── orders/
│           └── orders.csv
├── metadata/
│   ├── dataflows/
│   │   └── orders_platform_smoke.json
│   ├── environments/
│   │   ├── aws-iceberg.json
│   │   ├── aws.json
│   │   ├── databricks.json
│   │   ├── fabric.json
│   │   └── local.json
│   ├── connections.json
│   └── schema_hints.json
└── datacoolie.yml

File/Folder	Description	Links
data/	Fixture source directory.	—
── input/	Fixture source directory.	—
──── orders/	Fixture source directory.	—
────── orders.csv	Three known input rows.	source · raw
datacoolie.yml	Project build/environment configuration.	source · raw
metadata/	Fixture source directory.	—
── connections.json	Shared metadata and type hints.	source · raw
── dataflows/	Fixture source directory.	—
──── orders_platform_smoke.json	Shared metadata and type hints.	source · raw
── environments/	Fixture source directory.	—
──── aws-iceberg.json	Environment-specific connection addressing.	source · raw
──── aws.json	Environment-specific connection addressing.	source · raw
──── databricks.json	Environment-specific connection addressing.	source · raw
──── fabric.json	Environment-specific connection addressing.	source · raw
──── local.json	Environment-specific connection addressing.	source · raw
── schema_hints.json	Shared metadata and type hints.	source · raw

Configuration

These focused files show the public configuration boundaries. A runner owns engine/platform construction and passes metadata, SQL roots, runtime roots and external run attributes to the Driver.

configuration/
├── logging_modes.py
├── provider_construction.py
├── provider_fixtures.py
├── run_attributes.py
├── sql_roots.py
└── standalone_file_provider.py

File/Folder	Description	Links
logging_modes.py	Snapshot and JSON-record batch logging configuration.	source · raw
provider_construction.py	Artifact and explicit provider construction paths.	source · raw
provider_fixtures.py	Standalone database and API provider startup.	source · raw
run_attributes.py	External correlation values passed to a Driver session.	source · raw
sql_roots.py	Resolution through multiple explicit SQL roots.	source · raw
standalone_file_provider.py	FileProvider construction without a Driver.	source · raw

Dataflows

Each dataflow sample focuses on one metadata or execution feature. Inline SQL, SQL files, schema hints, load strategies and transform metadata remain separate authoring references instead of one large pipeline.

dataflows/
├── sql/
│   └── orders.sql
├── format_connections.json
├── inline_sql.json
├── load_strategies.json
├── sql_file.json
└── transform_patterns.json

File/Folder	Description	Links
format_connections.json	Alternative connection metadata representations.	source · raw
inline_sql.json	Inline SQL kept directly in source metadata.	source · raw
load_strategies.json	Append, overwrite and merge load patterns.	source · raw
sql/	SQL files kept next to focused metadata examples.	—
── orders.sql	Qualified SQL used by the SQL-file example.	source · raw
sql_file.json	File-backed SQL query metadata.	source · raw
transform_patterns.json	Column and row transformation patterns.	source · raw

Runners

Runners are project code, not a universal dc run command. They register engine relations, adapt host credentials and choose Driver options for a particular environment.

runners/
├── aws/
│   ├── run_glue_spark.py
│   └── run_polars_s3.py
├── databricks/
│   ├── maintenance_spark.ipynb
│   ├── replay_spark.ipynb
│   ├── run_polars_sdk.py
│   └── run_spark.ipynb
├── fabric/
│   ├── run_polars.ipynb
│   ├── run_polars_azure_sdk.py
│   └── run_spark.ipynb
└── local/
    ├── maintenance.py
    ├── replay.py
    ├── run.py
    ├── run_artifact_minimal.py
    └── run_spark.py

File/Folder	Description	Links
aws/	AWS Glue and S3 runner contracts.	—
── run_glue_spark.py	AWS Glue Spark runner.	source · raw
── run_polars_s3.py	Polars runner with S3 roots.	source · raw
databricks/	Databricks notebook and SDK runner contracts.	—
── maintenance_spark.ipynb	Databricks Spark maintenance notebook.	source · raw
── replay_spark.ipynb	Databricks Spark replay notebook.	source · raw
── run_polars_sdk.py	Databricks Polars SDK runner.	source · raw
── run_spark.ipynb	Databricks Spark runner notebook.	source · raw
fabric/	Fabric Spark and Polars runner contracts.	—
── run_polars.ipynb	Fabric Polars runner notebook.	source · raw
── run_polars_azure_sdk.py	Fabric Polars Azure SDK runner.	source · raw
── run_spark.ipynb	Fabric Spark runner.	source · raw
local/	Local Python, Spark, replay and maintenance runners.	—
── maintenance.py	Local Polars maintenance runner.	source · raw
── replay.py	Local Polars replay runner.	source · raw
── run.py	Configurable local Polars runner.	source · raw
── run_artifact_minimal.py	Minimal artifact-root runner.	source · raw
── run_spark.py	Local Spark runner.	source · raw

Operations

Operational wrappers demonstrate replay, interrupted-session recovery, maintenance confirmation and structured logging without changing the Driver contract.

operations/
└── replay_recovery.py

File/Folder	Description	Links
replay_recovery.py	Repeatable replay wrapper with explicit watermark-save confirmation.	source · raw

Plugins

Plugin samples show the smallest extension boundary and its packaging metadata. Follow the transformer tutorial to install this package and verify its output through a Driver. The transformer guide explains adaptation and pipeline ordering.

plugins/
├── pii_masker.py
└── pyproject.toml

File/Folder	Description	Links
pii_masker.py	Example transformer plugin that masks sensitive values.	source · raw
pyproject.toml	Build metadata, framework dependency and transformer entry point.	source · raw

Using the catalog with the CLI

The examples are documentation sources, not a project template. To inspect or build one of the complete projects locally:

Follow the CLI preparation walkthrough when you want a clean download-and-verify sequence. The commands below are the shortest checkout-based equivalent:

dc --project-dir docs/examples/files/projects/artifact validate --format json
dc --project-dir docs/examples/files/projects/artifact build --dry-run --format json

The equivalent executable names are dc and datacoolie. The project and CLI guide explains adaptation, runtime roots and runner ownership.