Tatiana Al-Chueyr Martins

As teams scale their Airflow workflows, a common question is: "My DAG has 5,000 tasks—how long will it take to run in Airflow?" Beyond execution time, users often face challenges with dynamically generated DAGs, such as: Delayed visualization in the Airflow UI after deployment.

Efficiently handling long-running workflows is crucial for scaling modern data pipelines. Apache Airflow’s deferrable operators help offload tasks during idle periods — freeing worker slots while tracking progress.

As a popular open-source library for analytics engineering, dbt is often combined with Airflow. Orchestrating and executing dbt models as DAGs ensures an additional layer of control over tasks, observability, and provides a reliable, scalable environment to run dbt models.

The integration between dbt and Airflow is a popular topic in the community, both in previous editions of Airflow Summit, in Coalesce and the #airflow-dbt Slack channel.

The main objective of this workshop is to demonstrate how Apache Airflow, together with the Astro Python SDK, can be used to orchestrate data pipelines and perform ETL processes in a scalable and performant way for professionals.

Apache Beam is a critical technology in delivering millions of personalised recommendations to the BBC audience daily. The journey to adopt the technology, however, wasn’t the smoothest. The objective of this talk is to save others time and money. This talk will discuss: Why Beam First pipeline...

Apache Airflow aims to speed the development of workflows, but developers are always ready to add bugs here and there. This talk illustrates a few pitfalls faced while developing workflows at the BBC to build machine learning models.