Polars vs Spark — Choose by Platform and ETL Layer
Use Polars when a job fits comfortably on one machine and a lightweight Python runtime reduces startup and operating cost. Use Spark when the workload needs distributed compute or the platform's catalog, table format, and optimization features are Spark-native. Data size matters, but it is not the only decision.
On Databricks, Spark is usually the practical default because the managed runtime already configures it. On Microsoft Fabric, the answer can change by medallion layer: Polars can suit small Bronze jobs, while Spark is the safer default for a Gold layer serving Direct Lake because it can write V-Order. On AWS, distinguish Glue or EMR Spark jobs from single-node Python runtimes before choosing an engine.