Yunhao Qing

Managing Apache Kafka® infrastructure across multiple regions and environments can quickly become complex, inconsistent, and difficult to scale. In this session, we’ll share how we use Terraform and the Confluent Terraform Provider to define, provision, and operate Kafka infrastructure as code. We’ll cover our approach to creating isolated Kafka environments, standardizing configuration across regions, and managing Confluent Cloud resources—including environments, Kafka clusters, topics, service accounts, role bindings, API keys, and networking—through reusable Terraform modules. We’ll also discuss practical tradeoffs around environment isolation, regional consistency, deployment safety, access control, and operational ownership. Attendees will leave with patterns for building a scalable, repeatable, and auditable Kafka platform that helps teams move faster while maintaining reliability and governance.

As data platforms grow in complexity, so do the orchestration needs behind them. Time-based (cron) scheduling has long been the default in Airflow, but dataset-based scheduling promises a more data-aware, efficient alternative.