Scale By the Bay 2017

Streaming is the major trend in the evolution of Big Data technologies and it is becoming a necessary part of systems designed to handle high volume of events. In this talk, we begin with characteristics of Stream Processing, and then discuss the dominant architectures for Streaming systems and continue into examples of technologies and application solutions for streaming applications. We compare the important technologies, such as Akka Streams, Storm, Kafka Streaming, Spark, Flink, Beam. We explore Quality Attributes of such systems as a mechanism for making architectural decisions and then discuss an approach for choosing a streaming systems. Outline: · Stream Processing: What is it? · Architectures for Stream Processing · Akka, Streaming, and Backpressure · Apache Storm: The Dedicated Stream system · Kafka Streaming: Taking Streaming Mainstream · Apache Spark Streaming: A versatile in-memory batch/streaming system · Apache Flink: Novel integration of batch and streaming · Apache Beam: The common API layer · Architectural Scenarios for Streaming · Choosing your Streaming System (Without Remorse!)