Holly Smith

Every one of us has a list. The tool that should never have shipped. The job title that means nothing. The architectural fashion that cost your last three employers eighteen months each. We all complain about them over coffee and then go back to using them on Monday. Room 404 is where they go to die. Our experts each bring their nominations - a practice, a pattern, a piece of terminology, a technology; they get five minutes to argue why it should never see the light of day again. The rest of the panel either leaps to its defense, or piles right on alongside. Will the nomination go through, will it be consigned forever to the vault - the audience decides, and the verdict is final. Banned or spared, no appeals.

For decades it's been an accepted fact that you need one tech stack for analytical data and one tech stack for transactional data. People tolerated this duplication because it just about did what people needed. Until agents came along, demanding analytics style data with transactional style access patterns. At Databricks, we've seen this problem on the horizon for a few years, and so that's why we've developed Lakebase, a managed Postgres solution that slots neatly into an analytics stack. In this talk, I'll be starting off with the basics of why this is very tricky to overcome, how we did it, how we see people using it, and the Databricks flourishes that make it unique in the market.

Matei is a legend of open source: he started the Apache Spark project in 2009, co-founded Databricks, and worked on other widely used data and AI software, including MLflow and Delta Lake. This will be a conversation covering the latest and greatest of UC and AI governance.

Hear how Databricks is using reinforcement learning to customize open source models to power our internal agents at a fraction of the cost and much higher speed. Hear from Jonathan Frankle, Databricks Chief AI Scientist, on how customers are using the same tools on Databricks AI Runtime (AIR) to do the same thing for their enterprise agents. We will go over the ingredients needed to successfully perform custom RL, the process of developing custom models using RL, and the novel tools and RL methods we have developed at Databricks to make this powerful technology available for enterprises.

Databricks empowers companies to easily build and deploy applications with ease - in just minutes not months! It’s the fastest and most secure way to deliver impactful solutions, with built-in governance of your data based on Unity Catalog. Developers can create applications using popular frameworks, serverless deployment and built-in governance. This allows developers to focus on delivering impactful solutions to users without the complexities of infrastructure management. Hear the latest and greatest from Justin DeBrabant, and how you can get started right away.

You'll hear a lot of success stories this week, but it also pays to learn from mistakes. Hear from three experts on what they've seen over the last few years that guarantee a project will fail. Robin Sutara is an expert in enterprise wide AI adoption via people and process, Maria is a lead advisor in AI project success and Holly spends her days talking to practitioners. Together they've seen a broad spectrum on the best and the worst that AI has to offer.

Arsalan Tavakoli-Shiraji co-founded Databricks, growing it to one of the most influential tech companies in history. Arsalan, who runs Field Engineering, will expand from the mainstage keynotes and discuss how companies are implementing, monetizing, and scaling with enterprise intelligence.

Genie lets you simply talk with your own data in the context of your business, and AI/BI Dashboards that go beyond traditional business intelligence tools - letting business users self-serve their own AI-driven insights without waiting on the BI teams. Genie is fully integrated with your unified data platform, finally delivering AI that truly knows your business and has full enterprise context.

Genie Code accelerates development for more technical users, enabling data scientists, data engineers and others to be so much more effective and efficient in the tasks they need to accomplish. Genie Code delivers AI with enterprise context to the users who need it, in the manner in which they need it. Genie Code builds the analysis for you, not just a query. It uses your enterprise context and business semantics to explain the “why” and points to exactly what happened. Hear from Weston Hutchins, Director of Product Management.

Ever wonder how Databricks operates its own enterprise lakehouse, where all employees and all teams inside use data and AI to solve problems and guide our decisions? Bruce Wong, head of data platforms, will talk about how his team leverages Databricks itself at a massive scale and value.

There's a quiet crisis going on in open source right now — whenever an open table format wants to build a new feature or integration, there's an exponentially more amount of work to do to incorporate it. A project that's increasingly harder to contribute to is the death of innovation. Kernel sets out to solve this problem by rewriting the source of truth. When other projects adopt Kernel, they no longer need to do any additional development for new features — they just appear. Join us for a deep dive into how we turned this project on its head, to solve not just these problems, but future problems we anticipate in open source. What started out as a contribution to Delta, could change the way that open table formats fundamentally integrate everywhere. Join Robert Pack, open-source contributor for Delta, Iceberg, Arrow and all round Rust enthusiast, and Holly Smith, tech communicator, who bullied Robert into this talk.

With the enormous amount of discussion about open storage formats between nerds and even not-nerds, it can be hard to keep track of who’s doing what and how this actually makes any impact on day to day data projects. Let's take a closer look at the three big projects in this space; Delta, Hudi and Iceberg. They’re all trying to solve for similar data problems and have tackled the various challenges in different ways. This talk with start with the very basics of how we got here, what the history is before diving deep into the underlying tech, their roadmaps, and their impacts on the data landscape as a whole.

Claude Code and Codex have taken the world by storm. Once you've seen the magic, it's hard to peel yourself away. As developers and enterprises lean further into these agent harnesses, questions start to arise about what the future direction holds. In this session, Nicolas Pelaez will take us through the expanding landscape of agent harnesses, tools, and connections. Hear his thoughts future direction of open source models, harnesses, and how Databricks fits in as this space begins to mature and patterns start to emerge.

We’re excited to be back at Big Data LDN this year—huge thanks to the organisers for hosting Databricks London once more! Join us for an evening of insights, networking, and community with the Databricks Team and Advancing Analytics! 🎤 Agenda: 6:00 PM – 6:10 PM | Kickoff & Warm Welcome Grab a drink, say hi, and get the lowdown on what’s coming up. We’ll set the scene for an evening of learning and laughs. 6:10 PM – 6:50 PM | The Metadata Marathon: How three projects are racing forward – Holly Smith (Staff Developer Advocate, Databricks) With the enormous amount of discussion about open storage formats between nerds and even not-nerds, it can be hard to keep track of who’s doing what and how this actually makes any impact on day to day data projects. Holly will take a closer look at the three big projects in this space; Delta, Hudi and Iceberg. They’re all trying to solve for similar data problems and have tackled the various challenges in different ways. Her talk will start with the very basics of how we got here, what the history is before diving deep into the underlying tech, their roadmaps, and their impacts on the data landscape as a whole. 6:50 PM – 7:10 PM | What’s New in Databricks & Databricks AI – Simon Whiteley & Gavi Regunath Hot off the press! Simon and Gavi will walk you through the latest and greatest from Databricks, including shiny new AI features and platform updates you’ll want to try ASAP. 7:10 PM onwards | Q&A Panel + Networking Your chance to ask the…

So you’ve heard of Databricks, but still not sure what the fuss is all about. Yes you’ve heard it’s Spark, but then there’s this Delta thing that’s both a data lake and a data warehouse (isn’t that what Iceberg is?) And then there's Unity Catalog, that's not just a catalog, it also does access management but even surprising things like optimise your data and programmatic access to lineage and billing? But then serverless came out and now you don’t even have to learn Spark? And of course there’s a bunch of AI stuff to use or create yourself. So why not spend 30 mins learning the details of what Databricks does, and how it can turn you into a rockstar Data Engineer.

So you’ve heard of Databricks, but after setting up a notebook and writing some code you’re still not sure what the fuss is all about. Yes you’ve heard it’s Spark, but then there’s this Delta thing that’s both a data lake and a data warehouse (isn’t that what Iceberg is?). And then Unity...

Databricks is the bestest platform ever where everything is perfect and nothing else could ever make it any better, right? …right? You and I know, this is not true. Don’t get me wrong, there are features that I absolutely love, but there are also some that require powering through the papercuts. And then there are those that I pretend don’t exist. I’ll be opening up to give my honest take on three of each category, why I do (or don’t) like them, and then telling you which talks to attend to find out more.

Join a live recording of the Over Architected Databricks podcast with Nick and Holly as they take the hottest features for the coming week and try to shoehorn them into one architecture. Audio for this session is delivered in the conference mobile app, you must bring your own headphones to listen.

Each year at Summit, Women in Data and AI have a half day for in-person discussions on empowering Women in Data and AI Breakfast, and networking with like-minded professionals and trailblazers. For this virtual discussion, hear from Kate Ostbye (Pfizer), Lisa Cohen (Anthropic), Pallavi Koppol and Holly Smith (Databricks) about navigating challenges, celebrating successes, and inspire one another as we champion diversity and innovation in data together. And how to get involved year-round.

As event-driven architectures powered by Apache Kafka™ continue to redefine real-time data processing, the demand for flexible, scalable, and efficient stateful streaming solutions has never been higher. Enter transformWithState, Apache Spark™’s groundbreaking new operator for Structured Streaming, designed to tackle the complexities of stateful processing head-on. In this session, we’ll dive into how transformWithState empowers developers to build sophisticated, low-latency streaming applications with Kafka as the backbone. From flexible state management and timer-driven logic to seamless schema evolution and integration with Kafka, we’ll explore real-world use cases—like real-time fraud detection and session-based analytics—that showcase its power. Attendees will leave with a clear understanding of how to leverage transformWithState to supercharge their Kafka-powered Spark pipelines, complete with practical examples, performance insights, and best practices for production deployment. Whether you’re optimizing stateful aggregations or chaining complex event-driven workflows, this talk will equip you to push the boundaries of what’s possible with Kafka and Spark.

I have a hypothesis, that 90% of people doing AI today weren’t doing it two years ago. The landscape is full of people stumbling their way through it, from the AI academics learning that code for papers is not software development ready, all the way to data experts suddenly needing to learn a new skill.

Delta Lake is an open-source storage framework that enables the creation of a Lakehouse architecture using a variety of compute engines such as Spark, PrestoDB, Flink, Trino, and Hive from Python.