Sandip Agarwala

In this session, we’ll explore how to build scalable, permission-aware AI applications by ingesting files and metadata from Google Drive and SharePoint into Databricks.We’ll examine how LakeFlow Connect ingests structured files (Excel, JSON), semi-structured data, and unstructured documents (PDF, DOCX, and more) from enterprise content systems, while capturing rich metadata and preserving file-level permissions for secure downstream use. You’ll learn how to process change feeds, handle deletes, manage schema evolution, and synchronize access controls, ensuring that AI systems respect enterprise security boundaries without sacrificing performance or scalability. We’ll then showcase how this ingestion foundation powers high-impact GenAI applications, including knowledge assistants and enterprise search with retrieval-augmented generation (RAG), transforming distributed enterprise content into governed, AI-ready intelligence that delivers trusted answers and actionable insights.

Auto Loader is the definitive tool for ingesting data from cloud storage into your lakehouse. In this session, we’ll unveil new features and best practices that simplify every aspect of cloud storage ingestion. We’ll demo out-of-the-box observability for pipeline health and data quality, walk through improvements for schema management, introduce a series of new data formats and unveil recent strides in Auto Loader performance. Along the way, we’ll provide examples and best practices for optimizing cost and performance. Finally, we’ll introduce a preview of what’s coming next — including a REST API for pushing files directly to Delta, a UI for creating cloud storage pipelines and more. Join us to help shape the future of file ingestion on Databricks.