Skip to content

Reference

Precise, mechanical contracts. Use this section when you need exact field names, types, defaults, or Python signatures. Prose explanations live under Concepts; task recipes under How-to.

This is not the best place to start if you are still learning the framework. Use Getting started for runnable examples and How-to for guided tasks.

Some reference pages are generated only at docs-build time, so you will not see their source .md files checked into docs/reference/.

Start with the contract you need

Configuration contracts

  • Metadata schema — generated from datacoolie.core.models. Covers the full field-level schema for Connection, DataFlow, Transform, schema hints, load strategies, watermark config, and partition config.
  • Plugin entry points — generated from pyproject.toml. Lists packaged entry-point declarations; in-process-only built-ins are called out separately.
  • Environment variables — runtime overrides that DataCoolie reads from the process environment.
  • CLI — runner scripts available under usecase-sim/runner/.

Python API reference

All packages are rendered directly from docstrings via mkdocstrings.

  • Core — Connection, Source, Destination, Transform, DataFlow, ReplayConfig, DataCoolieRunConfig, registry helpers.
  • Engines — BaseEngine[DF], PolarsEngine, SparkEngine.
  • Platforms — BasePlatform, LocalPlatform, AWSPlatform.
  • Sources — BaseSourceReader, FileReader, APIReader.
  • Destinations — BaseDestinationWriter, FileWriter.
  • Transformers — built-in transformers (SchemaConverter, Deduplicator, ColumnAdder, RowFilter, SCD2ColumnAdder, SystemColumnAdder, PartitionHandler, ColumnNameSanitizer) and TransformerPipeline.
  • Orchestration — DataCoolieDriver, JobDistributor, ParallelExecutor.
  • Metadata — provider classes and BaseMetadataProvider.
  • Watermark — WatermarkManager and the raw-JSON contract.
  • Logging — ETLLogger, LogPurpose, create_etl_logger.