Webinar: Distributed Stream Processing in Practice [Scalable, Real-time Data Pipelines]

​About the Event

​This technical session examines real-world challenges and patterns in building distributed stream processing systems. We focus on scalability, fault tolerance, and latency trade-offs through a concrete case study, using specific frameworks like Apache Storm as supporting tools to illustrate production concepts.

​Why Should You Attend

​Learn practical patterns for distributed stream processing at scale:

• ​Master real-world challenges - Understand scalability, fault tolerance, and latency trade-offs in production

• ​See architectural patterns - Stateless vs. stateful processing, event time vs. processing time decisions

• ​Handle scale bottlenecks - Partitioning strategies, backpressure handling, and scheduling challenges

• ​Learn from concrete examples - Real ML feature generation pipeline using Storm and Kafka

​Perfect for: Data engineers building distributed streaming systems who need production-proven patterns.

​------------------------------------------------------------

​Agenda (30 minutes)

​1. Stream Processing: Past and Now (4 minutes)

• ​Rise of real-time data needs in ML, analytics, and user-facing apps

• ​Shift from batch-first to event-first architectures

​2. Distributed Stream Processing Fundamentals (5 minutes)

• ​Definition and fundamentals

• ​Processing types: at-most-once, at-least-once, exactly-once

• ​Batch vs. micro-batch vs. true streaming

​3. Architectural Patterns (6 minutes)

• ​Stateless vs. stateful processing

• ​Event time vs. processing time

• ​Schedulers

​Common architecture: Kafka → Stream Processor → Sink (DB, Lake, Dashboard)

​4. Designing for Scale (6 minutes)

• ​Partitioning strategies and operator parallelism

• ​Handling backpressure and traffic spikes

• ​Scheduling challenges and system bottlenecks

• ​Fault tolerance and availability

​5. Case Study: Real-Time ML Feature Generation (10 minutes)

• ​Event Source (Kafka): Collects user events

• ​Stream Engine (Apache Storm): Processes and transforms streams

• ​Storage (S3): Stores aggregated feature datasets

• ​Setup: 1 Nimbus + 3 Workers distributed topology

• ​Model Training: Python jobs consume features