Daren Wong

This presentation provides a concise introduction to real-time data streaming with Apache Flink, illustrated through a practical demo/case study involving the processing of IoT sensor data. 🔹 1. Use Case Demonstration: Real-Time IoT Sensor Data Processing Pipeline Begin by presenting a practical and relatable use case: “Imagine a system ingesting millions of IoT sensor events per hour — temperature readings, device telemetry, or industrial metrics — streamed from Apache Kafka into Apache Flink for processing, then routed to a Kafka sink for downstream analytics and alerting.” This sets the stage for showcasing Flink’s capabilities in a real-world, high-throughput scenario. 🔹 2. Benefits & Challenges of Using Apache Flink Use the IoT pipeline to highlight the core trade-offs: ✅ Key Advantages - Low Latency: Near real-time processing with millisecond response. - Fault Tolerance: Robust state recovery through Flink's checkpointing mechanism. - Exactly-Once Semantics: Ensured via two-phase commit sinks. - Event-Time Processing: Native support for handling late or out-of-order events. - Stateful Computation: Built-in support for maintaining application state at scale. - Horizontal Scalability: Flink is designed for distributed, elastic workloads. ⚠️ Operational Considerations - Operational Complexity: Running and tuning a Flink cluster requires expertise, especially at scale. - State Backend Overhead: Efficient state handling demands proper configuration of backends like RocksDB, impacting performance and storage. 🔹 3. Building Flink Jobs: SQL, Table API, and DataStream API Demonstrate how to implement the IoT sensor example using Flink’s three APIs: - SQL: Easiest to use, ideal for quick analysis or for data analysts. - Table API: Balance between SQL simplicity and programmatic flexibility. - DataStream API: Full control for developers implementing complex, event-driven, and stateful logic. Explain how each API fits different user profiles, i.e SQL recommended for Data Analysts as it's easiest to use and ideal for quick data analysis. 🔹 4. Flink's Connector Ecosystem Showcase the rich extensive connector libraries of Apache Flink to connect to different sources and sinks - Show a list of available connectors and where to find them, i.e AWS, Kafka, ElasticSearch, MongoDB, etc. - Update the IoT sensor demo above to show how easy it is to swap between connectors to connect to different sources and sinks. - Show how to configure connectors, i.e timeout, retries etc. This demonstrates Flink’s adaptability in enterprise data platforms. 🔹 5. Monitoring & Observability Wrap up with a focus on operational visibility: - Live walkthrough of the Flink Web Dashboard. - Task and job metrics - Backpressure, Flamegraph and checkpoint monitoring ✅ Optional Additions (if time permits): - A slide on common pitfalls for new Flink users (e.g., not assigning operator UUID, ignoring watermarks). - Example of handling late-arriving events with allowed lateness and side outputs. - Mention deployment options: standalone cluster, YARN, Kubernetes, Flink on AWS, etc. - Mention how user can implement a brand new connector