Updated June 2026

Apache Airflow Administration: Scalable Workflow Automation and Orchestration

Class Duration

14 hours of live training delivered over 2-4 days.

Student Prerequisites

Practical experience writing Python programs
Basic Linux command line skills
Familiarity with containerization and Kubernetes concepts is helpful

Target Audience

Platform engineers, DevOps professionals, system administrators, and data engineers responsible for deploying, configuring, securing, and scaling Apache Airflow 3 in production environments, including Kubernetes-based deployments. Teams who write the workflows themselves can pair this with Apache Airflow Programming: Developing, Configuring, and Automating Workflows.

Description

This intensive course delivers a deep dive into Apache Airflow 3's architecture and core services - the api-server, scheduler, dag processor, triggerer, and workers - while contrasting it with Cron Jobs and Celery. Participants explore Directed Acyclic Graphs (DAGs), operators, and executors, along with installation, configuration, and auth managers. The course covers upgrading from Airflow 2 to Airflow 3 and integration with Kubernetes, AWS EKS, and Helm. Attendees gain hands-on experience deploying Airflow with Python/PostgreSQL and on Kubernetes, optimizing workflows, and building custom container images with additional dependencies. They also learn to monitor performance using logs, OpenTelemetry metrics, and Grafana. By the end, participants can configure, secure, scale, and optimize production-grade workflow automation solutions that are reliable and efficient.

Learning Outcomes

Understand Apache Airflow 3's architecture - api-server, scheduler, dag processor, triggerer, and workers - and how it compares to Cron Jobs and Celery.
Learn the fundamentals of DAGs, operators, tasks, variables, assets, and schedulers.
Install and configure Apache Airflow using Python environments, PostgreSQL, and Kubernetes, and plan upgrades from Airflow 2 to Airflow 3.
Gain hands-on experience deploying Airflow on Kubernetes, including EKS and Helm.
Configure Airflow's executors - Local, Celery, Kubernetes, and Edge - plus auth managers, logs, and advanced settings for scalability and security.
Build and use custom Airflow container images with additional dependencies.
Implement monitoring solutions using logs, OpenTelemetry metrics, Grafana, and external storage.
Apply best practices for workflow reliability, scaling, security, and automation in production environments.

Training Materials

Comprehensive courseware is distributed online at the start of class. All students receive a downloadable MP4 recording of the training.

Software Requirements

Students will need a free, personal GitHub account to access the courseware. Students will need permission to install Python and Visual Studio Code on their computers. Also, students will need permission to install Python Packages and Visual Studio Code extensions. If students are unable to configure a local environment, a cloud-based environment can be provided.

Training Topics

What is Apache Airflow?

Distributed Task Automation
Compared to Cron Jobs
Compared to Celery
Scalability and Reliability
Directed Acyclic Graphs (DAGs)
Workflows as Code
What's New in Airflow 3
Airflow 3 Services: API Server, Scheduler, DAG Processor, Triggerer, and Workers

Workflows as Code (no programming)

Anatomy of a DAG
Directed Acyclic Graphs
Operators
Tasks
Variables
XComs
Providers
Connections
Assets (formerly Datasets)
DAG Versioning
Explore how DAG parts connect to the new UI
DAG Serialization
Schedulers
Pools

Installation and Configuration

Python Virtual Environment
Install Airflow
Airflow Constraints File
Standalone Mode
Run the API Server, Scheduler, DAG Processor, and Triggerer Independently
SQLite vs PostgreSQL
Configure with PostgreSQL
Airflow and Kubernetes (with Minikube)
Airflow and AWS Elastic Kubernetes Service (EKS)
Airflow Helm Chart

Hands-On Kubernetes (K8s)

Containerization and Orchestration
Kubectl
Helm
Nodes
Namespaces
Pods, Containers, and Services
Connect to the Internet (EKS)
Keda Autoscaler
Pod Logs
SSH into Pods/Containers
Live Upgrading Airflow
Upgrading from Airflow 2 to Airflow 3

Airflow Configuration

Airflow Configuration File Location
Airflow Executor Configuration
Airflow Log Levels
Helm Chart Configuration
Learn How to Configure Airflow and K8s Pods
Local Executor
Celery Executor
Kubernetes Executor
Edge Executor
Auth Managers (SimpleAuthManager and FAB)

Airflow Custom Image

Airflow Container Image
Why Create a Custom Image?
Create a Custom Image
Install Software with Apt
Install Software with PyPi
Install Providers and Custom Software
Use the Custom Container Image

Monitoring

Logging
Log File Structure
Log Levels
Review Task Logs in the UI
External Log Storage
OpenTelemetry Metrics and Traces
Monitor with Grafana
Notifications and Deadline Alerts