Outline updated November 2026
Practical Apache Spark for Data Pipelines
Class Duration
21 hours of live training delivered over 3-5 days.
Student Prerequisites
- Familiarity with Python
- Working knowledge of SQL
- Basic data processing knowledge
Target Audience
Data engineers and Python developers who build batch and streaming pipelines with Apache Spark. Core material runs on open source Spark, with notes for Databricks, Amazon EMR, Microsoft Fabric, and Google Cloud Managed Service for Apache Spark, formerly Dataproc. Teams who also need to schedule these pipelines can continue to Apache Airflow 3 for Developers.
Description
This course teaches Apache Spark pipeline development in Python. Students learn Spark's execution model and Spark Connect, then work with DataFrames, Spark SQL, and the VARIANT type. They build batch and Structured Streaming jobs, including stateful processing with transformWithState. Spark Declarative Pipelines cover declared datasets, incremental processing, and Auto CDC flows. Later modules cover Real-Time Mode for low-latency streaming, performance tuning, and running Spark on managed platforms.
Learning Outcomes
- Choose between batch processing and Structured Streaming for a workload.
- Explain Spark's execution model and the Spark Connect architecture.
- Work with DataFrames, Spark SQL in ANSI mode, and VARIANT data.
- Write reusable PySpark code for batch and streaming jobs.
- Build stateful streaming jobs with transformWithState.
- Build pipelines with Spark Declarative Pipelines and Auto CDC flows.
- Use Real-Time Mode for low-latency streaming.
- Tune jobs with Adaptive Query Execution, partitioning, and caching.
- Run Spark on managed platforms with Delta Lake or Apache Iceberg tables.
Training Materials
Comprehensive courseware is distributed online at the start of class.
Software Requirements
Students will need permission to install Python, Java, and Visual Studio Code on their computers, along with Python packages. Core topics run on Apache Spark 4.3 with PySpark locally; managed platforms may run an earlier Spark 4.x runtime. Access to the team's platform of choice (Databricks, Amazon EMR, Microsoft Fabric, or Google Cloud Managed Service for Apache Spark) can be used for platform-specific topics. If students are unable to configure access, a cloud environment can be provided.
Training Topics
Spark Overview
- Introduction to Apache Spark and its ecosystem
- ANSI mode, VARIANT, and Spark Connect
- Spark fundamentals overview
- Pipeline development overview
- Advanced Spark and optimization overview
Spark Architecture and Use Cases
- Driver and cluster manager
- Worker nodes and executors
- Spark Connect client-server architecture
- Use cases for batch and Structured Streaming
- Spark in the lakehouse
Core Data Structures
- DataFrames and Spark SQL basics
- ANSI mode by default
- VARIANT type for semi-structured data
- SQL pipe syntax and SQL scripting
- Datasets and RDDs as legacy context
- Filtering and aggregations
- Joins and as-of joins
Spark Execution Model
- Partitioning
- Lazy execution
- Fault tolerance
- Checkpointing
- Serialization
- Arrow-optimized Python UDFs
Batch and Streaming Pipelines
- Designing batch pipelines
- Structured Streaming fundamentals
- Stateful processing with
transformWithState - Python Data Source API for custom connectors
- Reusable code components
Spark Declarative Pipelines
- Declarative vs. imperative pipeline development
- Datasets and flows
- Materialized views and streaming tables
- Temporary views
- Defining pipelines in Python and SQL
- Dependency resolution and incremental processing
- Auto CDC flows
- SCD Type 1 and Type 2
- Running pipelines with the
spark-pipelinesCLI
Real-Time Mode
- Low-latency streaming with Real-Time Mode
- Stateless and stateful queries in Real-Time Mode
- Real-Time Mode vs. micro-batch triggers
Advanced Features
- Broadcast variables
- Accumulators
- Serialization challenges
Performance Tuning
- Memory and CPU settings
- Partition sizing
- Adaptive Query Execution
- Caching
- Shuffle reduction and skew handling
- Reading query plans and the Spark UI
Spark on Managed Platforms
- Delta Lake and Apache Iceberg
- Comparing managed Spark platforms
- Serverless vs. cluster-based Spark
- Spark on Kubernetes
- Runtime version differences
- Portability practices
Case Studies
- Spark application case studies