Scale By the Bay 2017

Operationalizing data pipelines is difficult at scale. It often requires standing up large compute and memory resources around the clock in order to do small computations on demand. This architecture makes it difficult to isolate pain points in the pipeline, understand resource usage and orchestrate events in the pipeline. Functions As A Service (FaaS) has helped many in the Internet of Things and web applications space to operationalize usage patterns and overcome this problem. In this talk, I discuss how I used OpenWhisk to operationalize data pipelines, increase orchestration and decrease costs. I talk about the benefits of using Scala and it's functional paradigms and type safety for communication across functions, building event driven data pipelines and deploying it with open source technologies. Half of the talk will be background information about OpenWhisk and the other half will be deploying a machine learning model using Spark and OpenWhisk.