John Jackson

Organizations looking to orchestrate ETL data pipelines on AWS need tools that meet their requirements. In this chalk talk, learn about the pros and cons of orchestrating ETL pipelines with Apache Airflow running on Amazon MWAA.

On March 13th, 2025, Amazon Web Services announced General Availability of Amazon SageMaker Unified Studio, bringing together AWS machine learning and analytics capabilities.

Analytics professionals are tasked with integrating data from distributed systems (data lakes, data warehouses, search, NoSQL, machine learning, and more) to create better, secure, and cost-optimized experiences for their customers.

10 years after its creation, Airflow is stronger than ever: in last year’s Airflow survey, 81% of users said Airflow is important or very important to their business, 87% said their Airflow usage has grown over time, and 92% said they would recommend Airflow.

Airflow is all about schedules…we use CRON strings and Timetable to define schedules, and there’s an Airflow Scheduler component that manages those timetables, and a lot more, to ensure that DAGs and tasks are addressed based on those schedules.

Analytics professionals are tasked with integrating data from distributed systems (data lakes, data warehouses, search, NoSQL, machine learning, and more) to create better, secure, and cost-optimized experiences for their customers.

Amazon Managed Workflows for Apache Airflow (MWAA) was released in November 2020. Throughout MWAA's design we held the tenets that this service would be open-source first, not forking or deviating from the project, and that the MWAA team would focus on improving Airflow for everyone—whether they run Airflow on MWAA, on AWS, or anywhere else.

Airflow DAGs are Python code (which can pretty much do anything you want) and Airflow has hundreds configuration options (which can dramatically change Airflow behavior).

There are so many ways to run Airflow and as such, lots of folks responsible for running Airflow for downstream users. While many orgs go with a managed service(Astronomer, AWS MWAA, Google Cloud Composer), many also prefer running an Airflow platform themselves.

With data-driven approaches to innovate the business, many organizations find insights by running thousands of data and machine learning pipelines in Amazon Managed Workflows for Apache Airflow (Amazon MWAA). Managing workflows of such magnitude reliably at scale is a priority, but many find it challenging.

In this session we'll be discussing the considerations and challenges when running Apache Airflow at scale. We'll start by defining what it means to run Airflow at scale.

In this session we will discuss the latest features of Amazon Managed Workflows for Apache Airflow (MWAA) as well as some tips and tricks to get the most out of the service.

In this session we will discuss Amazon Managed Workflows for Apache Airflow (MWAA), how Apache Airflow (and specifically version 2.0) is implemented in the service, best practices for deployment and operations, and the Amazon MWAA team’s commitment to open source usage and contributions.

Apache Airflow continues to gain adoption for its ability to configure data pipelines as scripts, monitor them with a great user interface, and extend their functionality through a set of Community Heroes (AWS)-developed and custom plugins. In this session, see how to get started with Airflow using Amazon Managed Workflows for Apache Airflow, a platform that makes it easy for data engineers...

In this session, we explore using Amazon CloudWatch Container Insights to monitor, analyze, and scale your Amazon ECS workloads and microservices. You learn through a guided tutorial how to enable an ECS cluster for Container Insights and set up infrastructure to enrich your DevOps processes on both AWS Fargate and Amazon EC2.