Cut 80% Cloud Costs with Pandas & Polars Memory Optimisation

🚀 Core Idea

The author reduced cloud costs by 80% by optimizing memory usage in dataframes using Pandas and Polars.

🔍 Key Challenges

• Flink jobs were crashing due to out-of-memory errors when processing large CSVs.

• Pandas was consuming 7.6 GB of memory for a 1.38 GB CSV, causing system instability.

🛠️ Solutions

• Pandas Optimization: Specifying column data types (e.g., categorical, float32) reduced memory usage to 285 MB — a 97% drop.

• Polars Optimization: Even without manual tweaks, Polars used less memory due to its Arrow-based architecture.

• Further gains were achieved by explicitly defining schemas in Polars.

💡 Impact

• Jobs ran faster and more reliably.

• Infrastructure costs dropped significantly.

• Memory optimization became a strategic advantage, not just a technical fix.