Esa Denaux

In today’s data-driven world, ensuring data quality is crucial for making informed business decisions. Poor data quality can lead to incorrect analyses, misguided strategies, and ultimately, financial losses. When data is not properly validated, it can contain errors, inconsistencies, and inaccuracies that undermine trust in analytics and reporting. This session will delve into strategies for assessing and maintaining high data quality using Databricks and Pandas, ensuring that your data remains a robust asset for your organization. This session is designed for data engineers responsible for managing and maintaining data pipelines and ensuring data quality within their organizations. Attendees will benefit from practical insights and hands-on techniques for implementing data quality checks and monitoring strategies, all demonstrated through a detailed case study that serves as a red thread throughout the session. The session will cover essential aspects of data quality management, including data profiling upon initial ingestion, regression testing after software updates, and continuous monitoring. By focusing on tools like Databricks and Pandas, attendees will learn how to validate data types, perform statistical analysis, handle missing values and data duplication, enforce business rules, and set up effective monitoring systems. Participants should have a working knowledge of Python and SQL, as the session will involve coding examples and practical demonstrations using these languages. The learning objectives include understanding how to perform data profiling upon initial ingestion, conducting regression testing after software updates to ensure data quality, and setting up continuous monitoring systems. Attendees will also learn to use visualization techniques like histograms, box plots, and scatter plots to detect anomalies, generate summary statistics and data quality reports, implement schema enforcement, adhere to naming conventions and regulatory requirements, and utilize tools like PyTest for automated testing and Grafana for real-time performance monitoring. By the end of the session, attendees will be able to maintain data quality and system performance effectively, ensuring their data is reliable and fit for business use.