Outline updated November 2026

Intermediate14 hours delivered over 2-4 daysLive, online

Request this course

Apache Airflow 3 Operations and Administration

Class Duration

14 hours of live training delivered over 2-4 days.

Student Prerequisites

  • Basic Linux command line skills
  • Familiarity with containers and Kubernetes concepts
  • Working knowledge of what DAGs, tasks, and operators are; DAG authoring itself is not covered
  • Reading knowledge of Python is helpful

Target Audience

Platform engineers, DevOps engineers, SREs, and data platform owners who deploy and run Apache Airflow 3, often on Kubernetes. Teams who write the workflows can pair this with Apache Airflow 3 for Developers.

Description

This course teaches how to run Apache Airflow 3 as a platform. It is for the engineers who deploy, secure, and upgrade Airflow, not the people who write DAGs. Students learn the Airflow services and the Execution API, then install Airflow and choose executors. They deploy with the official Helm chart on Kubernetes and Amazon EKS, with KEDA autoscaling. Later modules cover auth managers, secrets backends, upgrades from Airflow 2, metadata database maintenance, and OpenTelemetry monitoring.

Learning Outcomes

  • Explain the Airflow services and how workers use the Execution API.
  • Install and configure Airflow with a PostgreSQL metadata database.
  • Configure the Local, Celery, Kubernetes, and Edge executors.
  • Deploy Airflow on Kubernetes with the official Helm chart.
  • Secure Airflow with auth managers, API tokens, and secrets backends.
  • Upgrade from Airflow 2 to Airflow 3 and across Airflow 3 releases.
  • Maintain the metadata database and DAG bundle configuration.
  • Monitor Airflow with remote logging, OpenTelemetry, and Grafana.

Training Materials

Comprehensive courseware is distributed online at the start of class.

Software Requirements

Students will need a free, personal GitHub account to access the courseware. Students will need permission to install Python, Visual Studio Code, Docker Desktop or Podman, Minikube, kubectl, and Helm on their computers, along with Python packages and Visual Studio Code extensions. Courseware targets Apache Airflow 3.4 and the current official Helm chart. If students are unable to configure a local environment, a cloud-based environment can be provided.

Training Topics

Platform Architecture

  • Airflow services and their roles
  • The Execution API and Task SDK
  • Workers without metadata database access
  • Delivering DAG code with DAG bundles
  • The metadata database
  • Service sizing
  • Network boundaries

Installation and Configuration

  • Installing Airflow with the constraints file
  • Standalone mode vs. separate services
  • PostgreSQL as the metadata database
  • Configuration files and environment variables
  • Configuration precedence
  • Log levels and log file structure
  • Official container images
  • Building custom images
  • Adding providers to images

Executors

  • Choosing an executor
  • Running multiple executors concurrently
  • Celery Executor brokers and queues
  • Monitoring Celery workers with Flower
  • Kubernetes Executor pod templates
  • Pod resources and cleanup
  • Edge Executor for remote workers
  • Pools, concurrency, and parallelism settings

Airflow on Kubernetes

  • The official Helm chart
  • Helm values
  • Deploying to Minikube for development
  • Deploying to Amazon EKS
  • Exposing the API Server
  • KEDA autoscaling for workers
  • Troubleshooting with logs and exec
  • Live configuration changes and rolling restarts

Security and Access Control

  • Auth managers
  • SimpleAuthManager and the FAB auth manager
  • Provider-based auth managers
  • Roles and permissions
  • Audit log access
  • API token authentication
  • Secrets backends
  • HashiCorp Vault and cloud secret managers
  • Multi-team deployments
  • Team isolation limits
  • Network policies and TLS between components

Upgrades and Maintenance

  • Upgrading from Airflow 2 to Airflow 3
  • DAG linting with Ruff
  • Upgrading between minor releases
  • airflow db migrate
  • Rollback planning
  • Provider version compatibility
  • Metadata cleanup with airflow db clean
  • Draining and pausing DAGs
  • Scheduler tuning
  • Metadata database tuning

Monitoring and Observability

  • Structured logging and remote log storage
  • Reviewing task logs in the UI
  • OpenTelemetry metrics and traces
  • Metric dashboards and alerting with Grafana
  • Health checks and heartbeats
  • Notifications and Deadline Alerts

Production Operations

  • Capacity planning and cost optimization
  • High availability for schedulers and API servers
  • Metadata database backup and recovery
  • Common platform problems

Private team training

Bring this course to your team

Tell us about your team and we will tailor this outline to your tools, stack, and schedule. Courses can also be combined into a longer program.

  1. Tell us what you needYour tools, stack, team size, and timeline.
  2. We send a proposalA course or program, an outline adjusted to your team, and possible dates.
  3. Your team trains liveOnline or on-site, with courseware for every student.

Prefer to talk? Call +1 434-509-6890 or email help@cloudcontraptions.com.

We reply within 1 business day.

Training partner?