1. Abstract
Despite its limitations, Apache Spark is still the go-to choice for big data workloads across organizations in the industry. However, organizations around the world waste money and productive time by running inefficient Spark jobs. The difference between an efficient Spark pipeline and an inefficient one could be an order of magnitude greater in terms of both compute cost and wall-clock time, and investing in an efficient pipeline could yield more than 75% savings in money and time.

In this workshop, Ammar Chalifah will cover best practices for optimizing a Spark job, from reading the physical plan, minimizing shuffle and skew, avoiding UDFs, choosing the right storage format and storage layout, and right-sizing the cluster.
2. Agenda
aim of the workshop is to ensure participants understand how Spark works under the hood, and are able to apply the knowledge in the workplace.

The agenda for the workshop covers:

1. Brief introduction to Spark’s inner working: a distributed data processing system. We will cover the concept of jobs, stages, and tasks.
2.  The divergence between code and physical plan: a conceptual understanding of how Spark converts the query (or code) into logical plan, and translates this plan into physical plan.
3. Practical guide on how to read Spark UI: understanding the interface, getting the necessary information, and finding bottlenecks
4. Optimization strategy: how to reduce shuffle
5. Optimization strategy: how to reduce spill
6. Optimization strategy: how to avoid wasteful/repeated computations by understanding lazy vs eager execution
7. Optimization strategy: how to avoid serialization cost; Python UDF vs Pandas UDF vs Native Spark
8. Optimization strategy: hardware selection & right-sizing, and configuration tuning
9. Optimization strategy: storage format & layout

Aside from the topics above, we’ll also discuss recent developments in the space and how these latest features could help your organizations be more efficient.

It is advised for participants to bring their own laptop with at least 8GB of RAM to be able to participate in the exercises.

3. Objectives
Attendees understand the biggest bottlenecks in Spark pipelines, know how to identify them, are able to implement an optimization technique, and are aware of production best practices.

4. Target audience and Prerequisites

• Data Engineer in early-to-mid career
• Engineers looking to deepen their expertise in Spark

5. Technical requirements

• SQL, Python, and have used Spark in production environments
• Comfortable with git
• Able to participate in the workshop with their own computer