Outline updated November 2026
Advanced Machine Learning and Data Engineering
Class Duration
35 hours of live training delivered over 5 days.
Student Prerequisites
- Completion of AI and Modern Machine Learning for Software Developers or equivalent experience with ML fundamentals (preparing data, then training, evaluating, and tuning models in Python)
- Solid Python programming skills, including pandas and scikit-learn
- Solid SQL skills: joins, aggregations, and analytic queries
Target Audience
Machine learning engineers, data engineers, and software developers taking machine learning from prototypes to production scale. Teams that only need data pipelines can start with Practical Apache Spark for Data Pipelines.
Description
This course covers the data platform and ML operations work behind production machine learning. Students build pipelines with Apache Spark, dbt, Apache Airflow, and Apache Kafka on Apache Iceberg and Delta Lake tables. They then build feature pipelines, scale training, track models with MLflow, and automate retraining and monitoring. The problems use messy multi-source data, changing schemas, and models that degrade after deployment. AI coding agents help write and debug the code, and students learn to review that work.
Learning Outcomes
- Design a lakehouse on Apache Iceberg or Delta Lake for analytics and ML.
- Build performance-tuned data pipelines with Apache Spark.
- Build layered SQL transformations with dbt.
- Orchestrate data and ML workflows with Apache Airflow, including backfills.
- Process streaming data with Apache Kafka and Spark Structured Streaming.
- Enforce data quality with schema management, validation checks, and data contracts.
- Build feature pipelines with point-in-time correctness.
- Scale model training with distributed training and efficient data loading.
- Track experiments and manage model versions with MLflow.
- Automate retraining and promotion with validation gates and rollback.
- Detect drift and model degradation, and respond to incidents.
- Use AI coding agents to generate, debug, and review platform code.
Training Materials
Comprehensive courseware is distributed online at the start of class.
Software Requirements
Students will need a computer capable of running Docker containers (16 GB of RAM recommended) to host the class data and ML stack. If students are unable to configure a local environment, a cloud-based environment can be provided.
Training Topics
Production Data Architecture and the Lakehouse
- From warehouses and data lakes to the lakehouse
- Apache Iceberg and Delta Lake
- ACID transactions on object storage
- Time travel and schema evolution
- Bronze, silver, and gold layers
- Catalogs, governance, and table maintenance
- Where ML workloads fit
Large-Scale Processing with Apache Spark
- Driver, executors, and Spark Connect
- DataFrames and Spark SQL
- The VARIANT type for semi-structured data
- Joins, aggregations, and window functions
- Partitioning and shuffles
- Adaptive Query Execution
- Reading and writing Iceberg and Delta tables
- Performance tuning and cost control
SQL Transformation with dbt
- Models, sources, and seeds
- Views, tables, and incremental models
- Jinja templating and macros
- Tests, documentation, and lineage
- Staging-to-marts layer design
Orchestration with Apache Airflow
- Airflow architecture
- The
airflow.sdkauthoring interface - DAGs with
@dagand@task - Asset-based and event-driven scheduling
- Dynamic task mapping and deferrable operators
- Backfills, retries, and DAG versioning
- Orchestrating Spark and dbt jobs
Streaming Ingestion with Apache Kafka
- Topics and partitions
- KRaft mode
- Producers, consumer groups, and delivery semantics
- Schema registries and serialization
- Kafka Connect sources and sinks
- Spark Structured Streaming
- Streaming into lakehouse tables
Data Quality, Schemas, and Contracts
- Where pipelines break
- Expectation-based validation
- Schema evolution across batch and streaming
- Data contracts
- Freshness, completeness, and anomaly checks
- Data observability and lineage
Feature Engineering at Scale and Feature Stores
- Feature pipelines on Spark and dbt
- Point-in-time correctness and leakage
- Offline and online feature stores
- Feature freshness and backfills
- Sharing features across teams
- Embeddings and streaming features
Distributed and Large-Scale Training
- When one machine is not enough
- Data-parallel training
- Gradient synchronization
- CPU and GPU clusters
- Loading training data from lakehouse tables
- Hyperparameter search at scale
- Checkpointing and fault tolerance
Experiment Tracking and Model Registry with MLflow
- Runs, parameters, metrics, and artifacts
- Logged models and signatures
- Model versions, aliases, and approvals
- Lineage from training data to deployment
- Comparing candidate models
Automated Retraining and Continuous Delivery
- CI/CD for machine learning
- Retraining triggers
- Retraining pipelines in Airflow
- Validation gates and automated promotion
- Shadow, canary, and blue-green deployment
- Rollback and version pinning
Production Monitoring, Drift, and Incident Response
- Monitoring predictions, latency, and business metrics
- Data drift and concept drift
- Delayed ground truth and proxy metrics
- Alert thresholds and escalation
- Root cause analysis across data and models
- Monitoring signals that drive retraining
Agentic AI Assistants Across the ML Platform
- Where agents help
- Agent-written Spark, dbt, and DAG code
- Debugging data and training failures with agents
- Reviewing agent-written code
- Human approval for production changes