Luke Moloney

Drawing on deep Microsoft Fabric experience, this session covers the data engineering lifecycle: -ingestion Data Factory and Mirroring, -transforming with Spark and Data Warehouse -orchestrating parallel pipelines -querying data -processing streaming workloads As well as enterprise deployment realities: -OneLake integration -monitoring -capacity visibility -automated deployments Suitable for...

Based on experience working with some of the largest Microsoft Fabric deployments, this session walks through practical tips for optimising data engineering-focused Fabric environments. This is a demo-centric session — real-world examples of tradeoffs that occur at scale and how simple changes can have outsized impact. Through-out this live demo-rich session we will explore the following topics Part 1 — Delta as the Table Format Delta is Fabric's primary table format, but its features come with performance tradeoffs that only surface at scale: - The real impact of Change Data Feed, Delete Vectors, and Deletion Vectors on read/write performance - What's coming in Delta 4.0 with the Fabric 2.0 runtime — and what to prepare for now - How metadata tables and logging tables in Delta quietly degrade performance — and practical solutions to contain the problem Part 2 — Optimising Spark Beyond the table format, Spark execution in Fabric has its own set of levers: - Orchestrating for scale: minimising cold-start time and session overhead - Spark Job Definitions and the Livy API: a walkthrough and demo of developer-centric execution patterns that bypass notebook limitations • Unlocking scenarios with notebookutils: parameterisation, chaining, and cross-item orchestration • Native Execution Engine: how it works under the hood, when it accelerates your workloads, and when it won't help You'll leave with concrete changes you can apply to your Fabric workloads to optimize performance and cost. A strong knowledge of Fabric is necessary, a good knowledge of Spark and Delta is highly recommended, with basic knowledge and experience a necessity.

Based on Luke’s experience in working with large-scale Fabric deployments - especially within the Financial Services Industry, this session goes through some the key design questions and considerations that will be faced during large scale Fabric adoptions in Enterprise environments. Some of the...

Want to know how to use Fabric Real Time Intelligence to make logs data? If so - this is the session for you. In this demo-centric session we will go through the end-to-end set-up process to get more visibility into your processes. We will use Fabric events and Fabric Spark Logs to demonstrate...

In this demo-centric session we will run through the tips, tricks and best practices when using Spark at scale. In this session we will cover: - Different ways to configure and manage your Spark Environments, including cluster sizing, libraries and configuration properties - Tips for performance profiling and…

In this demo-centric session we will run through the tips, tricks and best practices when using Spark at scale. In this session we will cover: - Different ways to configure and manage your Spark Environments, including cluster sizing, libraries and configuration properties - Tips for performance profiling and optimisation including when Delta optimisations in Fabric might be causing performance

In this session demo centric session we will walk through Data Engineering in Microsoft including: Getting started with Notebooks, Learning how Fabric Lakehouse's provides flexibility to use the tools you prefer, The flexibility of data engineering in Spark - including Cluster Configuration,...

In this session you will learn about all the latest data science developments in Microsoft Fabric. We will show you how Fabric supports the end-to-end data science workflow and how we plan to evolve these capabilities. We will also look at exciting new AI capabilities including Copilot...

Are you ready to become analytics superhero? Hesitate no more! Microsoft Fabric is out and building analytics solution cannot get any easier! If you believe that the best way to learn is to learn by example, join us! We will build a new green field analytical solution using new all-in-one Microsoft Fabric that covers everything from data movement to data science, Real-Time Analytics, and business intelligence. You will enjoy a highly integrated, end-to-end, and easy-to-use product that is designed to simplify your analytics needs. We will start off the day with an overview of Microsoft Fabric, set up the environment and explain the scenario for the day. There is no use of analytics solution if there is no data. So, we will show how data is stored in Fabric, how you can integrate existing data, or ingest data from different sources. Here we will create landing zone and integrate existing data without making another copy! Once data is in Microsoft Fabric OneLake, we will process it efficiently while demonstrating best practices. Data engineers are used to Spark so it will be our tool of choice in creating different layers while tackling all the things you need to know. During this part, you will learn how to incrementally load data and make sure your cleansing is done right and in an effective and dynamic way! Rest assured, SQL folks are not forgotten! We will show how you can access the same data using SQL! Of course, there is no using loading data without a serving layer for your business needs. We will show you how you can use Lakehouse & Warehouse in Microsoft Fabric focusing on performance! Finally, we will show you how to build Power BI report using new consumption method reading data directly from the lake, called DirectLake, skipping data engines. If you do not know where to start, don’t worry, during this day we will explain every step and how Fabric helps you get the job done. Also, we will share the best practices seen at most customers! After this day you will have a clear understanding of how Fabric works and what components you must use for you own top notch analytical solution! Fabric is brand new, so instead of throwing terminology here, we are inviting you to join us and become the analytics superhero!

By now you’ve probably heard lots about Microsoft Fabric, the latest and greatest SaaSified analytics solution from Microsoft. In a nutshell, Fabric enables you to work on a single copy of data using your favourite analytical engine. Whether you want to write good old SQL queries, do advanced data engineering using Spark notebooks or you just want to create a report in Power BI, Fabric has you covered. This is all possible through OneLake, the next chapter in the data lake storyline. I hear you thinking “how does that work/perform?”, “does it integrate with other technologies?”, “what are some of the gotcha’s?” ... In this advanced session we’ll answer all these questions and instantly transform you into a OneLake expert! We will dive deep into the technology it’s built on, giving you a technical breakdown of the key features whilst also addressing some of the things you need to be aware of before plunging in! Specifically, we will cover: - Brief overview of the evolution of data lakes - Distinctive features that set OneLake apart, focusing on: - file format - shortcuts - DirectLake - Hands-on demo illustrating its usability and performance - Current limitations and unsolved feature mysteries - How to start working with OneLake The session is designed for data engineers and data analysts who have a fundamental knowledge of Azure data services and want to get a deeper technical understanding of Microsoft Fabric’s OneLake.

Microsoft recently announced Microsoft Fabric - an end-to-end, unified analytics platform that brings together all the data and analytics tools that organizations need. Fabric integrates technologies like Azure Data Factory, Azure Synapse Analytics, and Power BI into a single unified product.

SQLBits' has brought members of the Azure Synapse Program Group to Wales to ask YOUR questions. Bring your questions for: Dedicated SQL Pools, Serverless SQL Pools, Spark Pools, Pipelines, Kusto and everything else Synapse related.

In this session I will briefly provide an overview of the lakehouse approach with Synapse and then dive into the practicalities of building a lakehouse architecture with Synapse with an end-to-end walkthrough / demo of how it the lakehouse can look with Synapse - especially focusing on the low(ish) code approach. Through this session I will cover: * incrementally landing the data into Azure Data Lake * using Synapse Data Flows to enrich and refine the data through the data lake zones * how Synapse Data Flows can tranform, test and validate data. * how Synapse SQL Serverless pools can expose the lakehouse to all users * how Synapse pipelines and Power BI can be combined with the lakehouse approach. This is a follow-on to my sessions I presented to Dataminds in May 2022.