Teo Van Lier

Spark has become the default choice for data workloads, but distributed compute is not always the right answer. In this session we dig into real workloads and compare Spark to native Python head to head. We will walk through scenarios like cleaning and transforming a few million rows, running aggregations on mid-sized datasets, and performing look-ups and joins on reference data. We'll look honestly at where each approach wins and why that is the case. Along the way we'll cover the trade-offs that don't always make it into the documentation: cold start times, debugging overhead, compute costs... You will leave this session with: - A clear understanding of what Spark is actually optimised for, and what it isn't - Practical examples of workloads where native Python outperforms Spark in speed, cost, and simplicity - A decision framework you can apply immediately to your own pipelines and notebooks The ability to confidently argue on which solution suits which workload Whether you're building new solutions in Microsoft Fabric or inheriting ones that already exist, this session will sharpen how you think about execution models and help you make better architectural decisions from now on. You'll spend less time over-engineering, less money on unnecessary compute, and more time delivering solutions that actually fit the problem.