Paul McComish

Modern data platforms promise scalability and flexibility—but without careful optimisation, costs can spiral and performance can degrade. This session focuses on how to effectively optimise your Azure Databricks Lakehouse to deliver faster data pipelines while significantly reducing platform costs. The intent of this session is to provide a practical, experience-led view of how data engineers can design, build, and operate efficient lakehouse architectures. I will explore common inefficiencies across storage, compute, and pipeline design, and show how small changes in approach can unlock major gains in performance and cost control. Key topics will include optimising Delta Lake storage patterns, improving Spark workload performance, right-sizing clusters, and making effective use of Databricks features such as Photon, caching, and job orchestration. I will also cover how to design pipelines with efficiency in mind from the outset, rather than retrofitting optimisations later. In addition, I will discuss how governance and observability play a critical role in cost management, helping teams understand where spend is occurring and how to control it without slowing down innovation. Attendees will leave with practical techniques, patterns, and best practices they can immediately apply to make their Databricks environments faster, more reliable, and more cost-efficient.

Collaboration is redefining efficiency in insurance. This session explores how technologies such as Databricks Delta Sharing, secure data clean rooms, and data marketplaces are empowering insurers to securely share and analyze data across organizational boundaries—without exposing sensitive information. Discover how these solutions streamline operations, enhance risk modeling with real-time data integration, and enable the creation of tailored products through multi-party collaboration. Learn how insurers are leveraging these collaborative data ecosystems to reduce costs, drive innovation, and deliver better customer outcomes, all while maintaining strong privacy and governance standards. Join us to see how embracing collaborative frameworks is helping insurers operate smarter, faster, and more efficiently.