Outline updated November 2026

Foundation21 hours delivered over 3-5 daysLive, online

Request this course

Practical Apache Spark for Data Pipelines

Class Duration

21 hours of live training delivered over 3-5 days.

Student Prerequisites

  • Familiarity with Python
  • Working knowledge of SQL
  • Basic data processing knowledge

Target Audience

Data engineers and Python developers who build batch and streaming pipelines with Apache Spark. Core material runs on open source Spark, with notes for Databricks, Amazon EMR, Microsoft Fabric, and Google Cloud Managed Service for Apache Spark, formerly Dataproc. Teams who also need to schedule these pipelines can continue to Apache Airflow 3 for Developers.

Description

This course teaches Apache Spark pipeline development in Python. Students learn Spark's execution model and Spark Connect, then work with DataFrames, Spark SQL, and the VARIANT type. They build batch and Structured Streaming jobs, including stateful processing with transformWithState. Spark Declarative Pipelines cover declared datasets, incremental processing, and Auto CDC flows. Later modules cover Real-Time Mode for low-latency streaming, performance tuning, and running Spark on managed platforms.

Learning Outcomes

  • Choose between batch processing and Structured Streaming for a workload.
  • Explain Spark's execution model and the Spark Connect architecture.
  • Work with DataFrames, Spark SQL in ANSI mode, and VARIANT data.
  • Write reusable PySpark code for batch and streaming jobs.
  • Build stateful streaming jobs with transformWithState.
  • Build pipelines with Spark Declarative Pipelines and Auto CDC flows.
  • Use Real-Time Mode for low-latency streaming.
  • Tune jobs with Adaptive Query Execution, partitioning, and caching.
  • Run Spark on managed platforms with Delta Lake or Apache Iceberg tables.

Training Materials

Comprehensive courseware is distributed online at the start of class.

Software Requirements

Students will need permission to install Python, Java, and Visual Studio Code on their computers, along with Python packages. Core topics run on Apache Spark 4.3 with PySpark locally; managed platforms may run an earlier Spark 4.x runtime. Access to the team's platform of choice (Databricks, Amazon EMR, Microsoft Fabric, or Google Cloud Managed Service for Apache Spark) can be used for platform-specific topics. If students are unable to configure access, a cloud environment can be provided.

Training Topics

Spark Overview

  • Introduction to Apache Spark and its ecosystem
  • ANSI mode, VARIANT, and Spark Connect
  • Spark fundamentals overview
  • Pipeline development overview
  • Advanced Spark and optimization overview

Spark Architecture and Use Cases

  • Driver and cluster manager
  • Worker nodes and executors
  • Spark Connect client-server architecture
  • Use cases for batch and Structured Streaming
  • Spark in the lakehouse

Core Data Structures

  • DataFrames and Spark SQL basics
  • ANSI mode by default
  • VARIANT type for semi-structured data
  • SQL pipe syntax and SQL scripting
  • Datasets and RDDs as legacy context
  • Filtering and aggregations
  • Joins and as-of joins

Spark Execution Model

  • Partitioning
  • Lazy execution
  • Fault tolerance
  • Checkpointing
  • Serialization
  • Arrow-optimized Python UDFs

Batch and Streaming Pipelines

  • Designing batch pipelines
  • Structured Streaming fundamentals
  • Stateful processing with transformWithState
  • Python Data Source API for custom connectors
  • Reusable code components

Spark Declarative Pipelines

  • Declarative vs. imperative pipeline development
  • Datasets and flows
  • Materialized views and streaming tables
  • Temporary views
  • Defining pipelines in Python and SQL
  • Dependency resolution and incremental processing
  • Auto CDC flows
  • SCD Type 1 and Type 2
  • Running pipelines with the spark-pipelines CLI

Real-Time Mode

  • Low-latency streaming with Real-Time Mode
  • Stateless and stateful queries in Real-Time Mode
  • Real-Time Mode vs. micro-batch triggers

Advanced Features

  • Broadcast variables
  • Accumulators
  • Serialization challenges

Performance Tuning

  • Memory and CPU settings
  • Partition sizing
  • Adaptive Query Execution
  • Caching
  • Shuffle reduction and skew handling
  • Reading query plans and the Spark UI

Spark on Managed Platforms

  • Delta Lake and Apache Iceberg
  • Comparing managed Spark platforms
  • Serverless vs. cluster-based Spark
  • Spark on Kubernetes
  • Runtime version differences
  • Portability practices

Case Studies

  • Spark application case studies

Private team training

Bring this course to your team

Tell us about your team and we will tailor this outline to your tools, stack, and schedule. Courses can also be combined into a longer program.

  1. Tell us what you needYour tools, stack, team size, and timeline.
  2. We send a proposalA course or program, an outline adjusted to your team, and possible dates.
  3. Your team trains liveOnline or on-site, with courseware for every student.

Prefer to talk? Call +1 434-509-6890 or email help@cloudcontraptions.com.

We reply within 1 business day.

Training partner?