Satej Kumar Sahu

Most organizations don't have a data problem, they have a meaning problem. Tables exist, metrics disagree, and tribal knowledge lives in Slack. Engineers waste days answering: "Which table is truth?", "What does 'active user' mean?", "Can I join these safely?" This session shows how to build a semantic metadata layer that makes your data platform searchable and self-service. The architecture combines metadata + lineage, a knowledge graph of entities and metric definitions, and LLM-powered enrichment (auto-generated descriptions, similarity detection, Q&A). Agents ingest catalog metadata, infer relationships, detect duplicates, and propose "source of truth" candidates based on usage signals. The semantic layer enables natural-language search, impact analysis, and guided safe joins. Critically, this isn't "LLM magic." We'll cover keeping it trustworthy: citations to lineage and owners, confidence scoring, and governance workflows. Feedback loops turn corrections into improved mappings. The outcome is data meaning as a product - discoverable, reusable, and consistent across teams.

Agentic AI is rapidly moving from “assistive” to “autonomous”: natural-language requests are now translated into SQL and Spark programs that can execute end-to-end with little or no oversight.

Testing Spark jobs in local environments is often difficult due to the lack of suitable datasets, especially under tight timelines. This creates challenges when jobs work in development clusters but fail in production, or when they run locally but encounter issues in staging clusters due to inadequate documentation or checks. In this session, we’ll discuss how these challenges can be overcome by leveraging Generative AI to create custom synthetic datasets for local testing. By incorporating variations and sampling, a testing framework can be introduced to solve some of these challenges, allowing for the generation of realistic data to aid in performance and load testing. We’ll show how this approach helps identify performance bottlenecks early, optimize job performance and recognize scalability issues while keeping costs low. This methodology fosters better deployment practices and enhances the reliability of Spark jobs across environments.