Polars vs Spark — Choose by Platform and ETL Layer¶
Use Polars when a job fits comfortably on one machine and a lightweight Python runtime reduces startup and operating cost. Use Spark when the workload needs distributed compute or the platform's catalog, table format, and optimization features are Spark-native. Data size matters, but it is not the only decision.
On Databricks, Spark is usually the practical default because the managed runtime already configures it. On Microsoft Fabric, the answer can change by medallion layer: Polars can suit small Bronze jobs, while Spark is the safer default for a Gold layer serving Direct Lake because it can write V-Order. On AWS, distinguish Glue or EMR Spark jobs from single-node Python runtimes before choosing an engine.
The 30-second decision¶
| Situation | Start with | Why |
|---|---|---|
| Local development or CI | Polars | Short feedback loop and no Spark session startup |
| Small, single-node file transformation | Polars | Less runtime overhead when the data and joins fit in memory |
| Distributed joins, deduplication, or large merges | Spark | Scale-out execution and mature shuffle controls |
| Databricks production pipeline | Spark | Spark, Delta Lake, and Unity Catalog are integrated into the managed runtime |
| Fabric Gold feeding Direct Lake | Spark | Delta optimization and V-Order are part of the Spark write path |
| AWS Glue Spark or EMR Serverless | Spark | The service supplies and operates the Spark runtime |
| Python 3.11+ container or VM on AWS | Polars, if the job fits | A single-node runtime can avoid provisioning distributed compute |
This table gives a default, not a permanent choice. Validate it against the data shape, table format, concurrency, operational tooling, and downstream consumer.
Choose by medallion layer¶
| Layer | Polars is a good fit when… | Prefer Spark when… |
|---|---|---|
| Bronze | Inputs are small or moderate, transforms are light, and the job mainly appends files or path-based Delta tables | Ingestion volume is high, schema evolution is complex, or the managed platform already provides Spark |
| Silver | Joins and validation fit on one node and engine portability is more valuable than platform-native features | The layer performs large joins, deduplication, MERGE/SCD processing, or needs distributed recovery and observability |
| Gold | Aggregations are modest and the consumer does not require a Spark-specific physical layout | The layer serves a lakehouse catalog, large concurrent reads, or platform optimizations such as Fabric V-Order for Direct Lake |
Layer names alone do not select an engine. A tiny Gold aggregate can still run well on Polars; a high-volume Bronze ingest can require Spark. The downstream contract is often the deciding factor.
Microsoft Fabric: decide by workload and consumer¶
Fabric notebooks support Python and Spark runtimes. Microsoft's current kernel selection guidance positions Python engines such as Polars well for very small workloads, while Spark becomes more competitive as data grows and provides fuller Delta Lake support.
Recommended defaults:
- Bronze: Polars for small file-oriented loads where native Python startup matters; Spark for high volume, streaming, or advanced Delta behavior.
- Silver: choose from measured data size and transformation complexity; prefer Spark for large joins, MERGE/SCD, and scale-out execution.
- Gold → Direct Lake: prefer Spark and explicitly design the Delta output for Direct Lake reads.
Fabric's Direct Lake guidance places Direct Lake over well-tuned Delta tables in the analytics layer. V-Order can improve read-heavy workloads, but Microsoft documents additional write cost and disables it by default in new workspaces. Enable it deliberately for Gold tables that benefit from it; do not pay that cost automatically on every Bronze or staging write.
See Deploy DataCoolie to Microsoft Fabric for notebook and V-Order setup.
Databricks: Spark is the operational default¶
Apache Spark is at the heart of Databricks, and Databricks creates and manages the Spark context and session for cluster workloads. The runtime also integrates Spark with Delta Lake, Unity Catalog, job monitoring, and platform optimizations. Installing another engine must therefore deliver enough benefit to offset dependency management and a second operational path.
Use Spark by default for production Delta tables, Unity Catalog integration, distributed workloads, and shared operational support. Consider Polars for an isolated, measured, small file job when its inputs fit on one node and it does not need Spark-native catalog behavior. DataCoolie's current Polars Databricks sample supports UC Volume file paths; Delta through UC Volume paths still uses the Spark sample as the supported path.
See Deploy DataCoolie to Databricks.
AWS: choose the runtime before the engine¶
“AWS” is not one execution environment:
| AWS runtime | DataCoolie default | Decision note |
|---|---|---|
| AWS Glue Spark ETL | Spark | Glue supplies a managed Spark/PySpark environment |
| Amazon EMR or EMR Serverless Spark | Spark | The application and job contract are Spark-native |
| AWS Glue Python Shell | Not supported by the current package | Glue Python Shell uses Python 3.9, while DataCoolie requires Python 3.11+ |
| Python 3.11+ on a container, VM, or local runner using AWS services | Polars for fitting workloads | AWSPlatform can use S3 and Secrets Manager without requiring Spark |
AWS documents Glue as having separate Spark, Ray, and Python Shell job types. Glue 5.0 Spark jobs provide Python 3.11 and Spark 3.5.4, while the current Python Shell environment uses Python 3.9. AWS also recommends packaging pinned dependencies as wheel artifacts rather than resolving unpinned packages during every job startup; see Using Python libraries with AWS Glue.
For EMR Serverless, submit DataCoolie as a PySpark job to a
Spark application.
For small jobs that do not need Glue or EMR, run Polars in a controlled Python
3.11+ environment and use AWSPlatform for S3 and Secrets Manager access.
See Deploy DataCoolie to AWS Glue.
What the DataCoolie benchmark does—and does not—prove¶
DataCoolie's committed benchmark is a local comparison, not a universal cloud ranking. It ran Polars 1.39.3 and PySpark 3.5.8 on Windows with 12 logical CPU cores and 30.7 GB RAM. Spark used local mode, and both engines were warmed up before timed dataflows.
Representative results from that run show why format and operation matter:
| Workload | Rows | Polars | Local Spark | Faster in this run |
|---|---|---|---|---|
| Parquet → Parquet | 1M | 0.11 s | 1.09 s | Polars |
| Delta → Delta | 50M | 24.55 s | 54.76 s | Polars |
| Parquet → Iceberg | 50M | 75.49 s | 37.12 s | Spark |
These numbers do not include cluster scale-out, managed-runtime convenience, V-Order, Unity Catalog, Glue/EMR startup, network storage, or production concurrency. Use the full benchmark report to reproduce the test, then benchmark the same source format, load strategy, data shape, and target platform as your production job.
A practical selection checklist¶
Choose the engine only after answering these questions:
- Is Spark already provided and supported by the target runtime?
- Does the peak working set fit safely in one machine's memory?
- Do joins, skew, or concurrency require distributed execution?
- Does the destination need Unity Catalog, Glue Catalog, V-Order, or another platform-native write path?
- Is startup and dependency installation a meaningful share of job duration?
- Can the team monitor, patch, and reproduce both runtime paths?
- What does a representative benchmark show on the target platform?
Keep pipeline intent portable with DataCoolie¶
DataCoolie separates pipeline metadata from engine and platform selection:
# Lightweight local or AWS-connected run
engine = PolarsEngine(platform=AWSPlatform(region="ap-southeast-1"))
# Managed Spark runtime: Fabric, Databricks, Glue, or EMR
engine = SparkEngine(spark_session=spark, platform=platform)
The metadata can preserve source, transform, load strategy, schema, watermark, and logging intent while the runner selects the engine appropriate to each environment. Start with the Polars quickstart, compare the Spark quickstart, and use the platform guide for production deployment.
Bottom line¶
- Choose Polars for a small, single-node workload—not merely because a row threshold says it should be faster.
- Choose Spark when scale or a managed platform feature makes Spark the lower-risk operational choice.
- On Fabric, let the layer and Direct Lake contract influence the decision.
- On Databricks, Spark is the default unless a bounded Polars job proves a clear advantage.
- On AWS, select Glue/EMR versus a Python runtime first; then select Spark or Polars.