Scale By the Bay 2017

The main reason people are productive writing software is composability: engineers can take libraries and functions written by other developers and easily combine them into a program. However, composability has taken a back seat in early parallel processing APIs. For example, composing MapReduce jobs required writing the output of every job to a file, which was both error-prone and slow. Apache Spark helped simplify cluster programming largely because it enabled efficient composition of parallel functions, leading to a large standard library and high-level APIs in various languages. In this talk, I'll explain how composability has evolved in Spark's newer APIs, and I’ll present Weld, a new research project I'm leading at Stanford to enable much richer composition of software on emerging parallel hardware (multicores, GPUs, etc). Systems like Weld and Spark will allow engineers to focus on building their application rather than the intricacies of parallel hardware, and might represent one of the best ways we have to tame the ever-diversifying hardware landscape.