Nic Townsend

Have you ever wished you could handle problematic events in Flink SQL as easily as with DataStream side outputs? Imagine routing unprocessable records—such as those failing serialization—straight to a dead-letter Kafka queue without stopping your job. The new Apache Flink HTTP connector makes this possible while unlocking even more capabilities. It allows you to treat API endpoints as dynamic Flink tables, enabling seamless integration with any technology that exposes APIs—without writing custom code. For example, you can connect to your favorite AI endpoint simply by declaring a Flink SQL table. In this session, you’ll learn how to leverage the HTTP connector to keep your Flink jobs running even after exceptions, HTTP error codes, or deserialization failures. We’ll explore how its new metadata columns provide powerful tools for error handling and observability. You’ll also discover best practices for tuning the connector for enterprise scenarios, including caching strategies, security configurations, and retry mechanisms. Key Takeaways: • How to integrate APIs into Flink SQL with zero custom code • Techniques for handling errors gracefully and improving resilience • Using…

Waiting for real datasets, or building complex mocks, slows down streaming development. Apache Flink's Datagen connector lets you rapidly prototype pipelines without any external source, generating realistic, controllable data at scale. In this session, we'll show how to use Datagen to shape schema, control distributions, and push throughput so you can test operators, windows, joins, and sinks under production‑like conditions. What You'll Learn We'll cover the full spectrum of Datagen capabilities through hands-on demonstrations: - Schema Design and Data Types: Learn to define complex schemas with nested structures, arrays, and maps. We'll show how to generate realistic data patterns including timestamps, UUIDs, sequences, and random values across Flink SQL data types. - Distribution Control: Master techniques for controlling data distributions to simulate real-world scenarios. - Performance Testing: Discover how to configure throughput rates, control parallelism, and generate high-volume streams to stress-test your pipeline. We'll demonstrate techniques for simulating backpressure, testing watermark strategies, and validating checkpoint performance under load. - Advanced…