Outline updated November 2026
Data Engineering: Ansible, Airflow, and Snowflake
Class Duration
21-28 hours of live training delivered over 3-4 days. The longer format adds depth on Snowpark, dbt, and production practices.
Student Prerequisites
- Practical experience with Python
- Basic Linux command line skills
- Familiarity with SQL fundamentals
- Basic understanding of cloud computing concepts
Target Audience
Data engineers, DevOps engineers, and Python developers who build automated data pipelines. Teams who want more depth on orchestration can continue to Apache Airflow 3 for Developers.
Description
This course combines Ansible, Apache Airflow, and Snowflake for data engineering. Ansible provisions and configures the environment, and Airflow schedules and monitors the pipelines. In Snowflake, students load data with stages, COPY INTO, Snowpipe, and Openflow. They transform data with SQL, Snowpark Python, Dynamic Tables, and dbt Projects on Snowflake, and call Cortex AI Functions from SQL. Final modules connect the three tools in ELT pipelines and cover testing, CI/CD, and cost control.
Learning Outcomes
- Provision infrastructure with Ansible playbooks, roles, and collections.
- Manage dynamic inventories and secrets with Ansible Vault.
- Write Airflow DAGs with the TaskFlow API, assets, and dynamic task mapping.
- Size Snowflake virtual warehouses and control cost.
- Load data with stages, COPY INTO, Snowpipe, and Openflow connectors.
- Transform data with Snowflake SQL, Snowpark Python, and Dynamic Tables.
- Capture changes with streams and tasks.
- Build and run dbt projects with dbt Projects on Snowflake.
- Call Cortex AI Functions from SQL while managing cost and governance.
- Orchestrate Snowflake ELT pipelines with Airflow and the Snowflake provider.
Training Materials
Comprehensive courseware is distributed online at the start of class.
Software Requirements
Students will need a free, personal GitHub account to access the courseware. Students will need permission to install Python and Visual Studio Code on their computers, along with Python packages and VS Code extensions. Courseware targets ansible-core 2.22, Apache Airflow 3.4, and current Snowflake features. A Snowflake trial or training account will be used. Openflow is not enabled on trial accounts by default, and some Cortex AI Functions depend on region, so both are confirmed during class planning. If students are unable to configure a local environment, a cloud-based environment can be provided.
Training Topics
Introduction to Ansible
- Overview and architecture
- Declarative, push-based configuration
ansible-coreand collections- Fully qualified collection names
- YAML syntax review
- Command-line tools
Ansible Setup and Inventory
- Installing and configuring
ansible-core - Configuration files
- Defining hosts and groups
- Dynamic inventory for cloud environments
Ansible Playbooks, Variables, and Templates
- Playbook writing basics
- Tasks and handlers
- Commonly used modules
- Variables and facts
- Jinja2 templates and template modules
- Linting playbooks with ansible-lint
Ansible Roles and Reuse
- Creating and using roles
- Role directory structure
- Ansible Galaxy and collections
- Writing reusable code
Apache Airflow Essentials
- Workflows as code
- Directed acyclic graphs (DAGs)
- Airflow services
- Installing with a constraints file
- Metadata database options
- Executor types and configuration
Developing DAGs
- The
airflow.sdkauthoring interface - TaskFlow API and Operator API
- Connections, variables, and XComs
- Assets and asset-based scheduling
- Dynamic task mapping
- Monitoring runs and task logs
Introduction to Snowflake
- Cloud data platform architecture
- Separation of storage and compute
- Virtual warehouses
- Snowflake editions and pricing
- Snowsight and Workspaces
Snowflake Objects and Data Modeling
- Databases, schemas, and tables
- Views and materialized views
- Data types and constraints
- Clustering keys
- Time Travel and Fail-safe
- Apache Iceberg tables
Loading Data into Snowflake
- Internal and external stages
- The COPY INTO command
- CSV, JSON, and Parquet file formats
- Snowpipe for continuous loading
- Snowpipe Streaming for low-latency ingestion
- Openflow managed connectors
- Openflow Snowflake deployments and BYOC
Transforming Data in Snowflake
- Snowflake SQL extensions and window functions
- Semi-structured data with VARIANT
- Snowpark Python DataFrames
- Snowpark UDFs and stored procedures
- Dynamic Tables for declarative pipelines
- Streams and tasks for change data capture
- Query optimization
- Query History and Query Profile
dbt Projects on Snowflake
- dbt models, sources, and tests
- dbt documentation
- dbt projects in Snowflake Workspaces
- Deploying dbt projects with Snowflake CLI
- Scheduling dbt runs with tasks or Airflow
Cortex AI Functions in Pipelines
- AI_COMPLETE and AI_CLASSIFY
- AI_EXTRACT and AI_AGG
- Summarization, sentiment, and document parsing
- Cost controls for AI Functions
- Choosing a model family
- Regional availability
- AI governance in data pipelines
Security and Access Control
- Role-based access control
- Users, roles, and privileges
- Key-pair and OAuth authentication
- Service users
- Row access and masking policies
- Network policies
- Secure Data Sharing
Orchestrating Snowflake with Airflow
- Snowflake provider for Airflow
SQLExecuteQueryOperatorandSnowflakeSqlApiOperator- Connection configuration
- Running Snowpark and dbt workloads from DAGs
- Error handling and retries
- Idempotent pipeline design
Using Ansible for Data Infrastructure
- Provisioning Airflow environments
- Managing Snowflake users, roles, and warehouses
- Secrets management with Ansible Vault
Production Considerations
- Data quality checks
- Testing data pipelines
- CI/CD for data pipelines
- Monitoring, alerting, and cost management
- Warehouse sizing and performance tuning