Outline updated November 2026

Advanced35 hours delivered over 5 daysLive, online or on-site

Request this course

Advanced Machine Learning and Data Engineering

Class Duration

35 hours of live training delivered over 5 days.

Student Prerequisites

  • Completion of AI and Modern Machine Learning for Software Developers or equivalent experience with ML fundamentals (preparing data, then training, evaluating, and tuning models in Python)
  • Solid Python programming skills, including pandas and scikit-learn
  • Solid SQL skills: joins, aggregations, and analytic queries

Target Audience

Machine learning engineers, data engineers, and software developers taking machine learning from prototypes to production scale. Teams that only need data pipelines can start with Practical Apache Spark for Data Pipelines.

Description

This course covers the data platform and ML operations work behind production machine learning. Students build pipelines with Apache Spark, dbt, Apache Airflow, and Apache Kafka on Apache Iceberg and Delta Lake tables. They then build feature pipelines, scale training, track models with MLflow, and automate retraining and monitoring. The problems use messy multi-source data, changing schemas, and models that degrade after deployment. AI coding agents help write and debug the code, and students learn to review that work.

Learning Outcomes

  • Design a lakehouse on Apache Iceberg or Delta Lake for analytics and ML.
  • Build performance-tuned data pipelines with Apache Spark.
  • Build layered SQL transformations with dbt.
  • Orchestrate data and ML workflows with Apache Airflow, including backfills.
  • Process streaming data with Apache Kafka and Spark Structured Streaming.
  • Enforce data quality with schema management, validation checks, and data contracts.
  • Build feature pipelines with point-in-time correctness.
  • Scale model training with distributed training and efficient data loading.
  • Track experiments and manage model versions with MLflow.
  • Automate retraining and promotion with validation gates and rollback.
  • Detect drift and model degradation, and respond to incidents.
  • Use AI coding agents to generate, debug, and review platform code.

Training Materials

Comprehensive courseware is distributed online at the start of class.

Software Requirements

Students will need a computer capable of running Docker containers (16 GB of RAM recommended) to host the class data and ML stack. If students are unable to configure a local environment, a cloud-based environment can be provided.

Training Topics

Production Data Architecture and the Lakehouse

  • From warehouses and data lakes to the lakehouse
  • Apache Iceberg and Delta Lake
  • ACID transactions on object storage
  • Time travel and schema evolution
  • Bronze, silver, and gold layers
  • Catalogs, governance, and table maintenance
  • Where ML workloads fit

Large-Scale Processing with Apache Spark

  • Driver, executors, and Spark Connect
  • DataFrames and Spark SQL
  • The VARIANT type for semi-structured data
  • Joins, aggregations, and window functions
  • Partitioning and shuffles
  • Adaptive Query Execution
  • Reading and writing Iceberg and Delta tables
  • Performance tuning and cost control

SQL Transformation with dbt

  • Models, sources, and seeds
  • Views, tables, and incremental models
  • Jinja templating and macros
  • Tests, documentation, and lineage
  • Staging-to-marts layer design

Orchestration with Apache Airflow

  • Airflow architecture
  • The airflow.sdk authoring interface
  • DAGs with @dag and @task
  • Asset-based and event-driven scheduling
  • Dynamic task mapping and deferrable operators
  • Backfills, retries, and DAG versioning
  • Orchestrating Spark and dbt jobs

Streaming Ingestion with Apache Kafka

  • Topics and partitions
  • KRaft mode
  • Producers, consumer groups, and delivery semantics
  • Schema registries and serialization
  • Kafka Connect sources and sinks
  • Spark Structured Streaming
  • Streaming into lakehouse tables

Data Quality, Schemas, and Contracts

  • Where pipelines break
  • Expectation-based validation
  • Schema evolution across batch and streaming
  • Data contracts
  • Freshness, completeness, and anomaly checks
  • Data observability and lineage

Feature Engineering at Scale and Feature Stores

  • Feature pipelines on Spark and dbt
  • Point-in-time correctness and leakage
  • Offline and online feature stores
  • Feature freshness and backfills
  • Sharing features across teams
  • Embeddings and streaming features

Distributed and Large-Scale Training

  • When one machine is not enough
  • Data-parallel training
  • Gradient synchronization
  • CPU and GPU clusters
  • Loading training data from lakehouse tables
  • Hyperparameter search at scale
  • Checkpointing and fault tolerance

Experiment Tracking and Model Registry with MLflow

  • Runs, parameters, metrics, and artifacts
  • Logged models and signatures
  • Model versions, aliases, and approvals
  • Lineage from training data to deployment
  • Comparing candidate models

Automated Retraining and Continuous Delivery

  • CI/CD for machine learning
  • Retraining triggers
  • Retraining pipelines in Airflow
  • Validation gates and automated promotion
  • Shadow, canary, and blue-green deployment
  • Rollback and version pinning

Production Monitoring, Drift, and Incident Response

  • Monitoring predictions, latency, and business metrics
  • Data drift and concept drift
  • Delayed ground truth and proxy metrics
  • Alert thresholds and escalation
  • Root cause analysis across data and models
  • Monitoring signals that drive retraining

Agentic AI Assistants Across the ML Platform

  • Where agents help
  • Agent-written Spark, dbt, and DAG code
  • Debugging data and training failures with agents
  • Reviewing agent-written code
  • Human approval for production changes

Private team training

Bring this course to your team

Tell us about your team and we will tailor this outline to your tools, stack, and schedule. Courses can also be combined into a longer program.

  1. Tell us what you needYour tools, stack, team size, and timeline.
  2. We send a proposalA course or program, an outline adjusted to your team, and possible dates.
  3. Your team trains liveOnline or on-site, with courseware for every student.

Prefer to talk? Call +1 434-509-6890 or email help@cloudcontraptions.com.

We reply within 1 business day.

Training partner?