Upgrading Apache Airflow

This document guides you through the process of upgrading Apache Airflow to a newer version. Upgrading involves several steps to ensure a smooth transition and minimize downtime.

Important: Always back up your metadata database and Airflow configuration files before starting an upgrade. Consult the production-ready environment guide for recommendations.

Before You Begin

Review Release Notes: Carefully read the Changelog and Release Notes for the version you are upgrading to. Pay close attention to any breaking changes, deprecations, or new features that might affect your DAGs or infrastructure.
Check Dependencies: Ensure that your Python environment and any external dependencies (like database drivers or provider packages) are compatible with the new Airflow version.
Test in a Staging Environment: It is highly recommended to perform the upgrade in a non-production (staging) environment first. This allows you to identify and resolve any issues before affecting your production workflows.
Understand Breaking Changes: Major versions (e.g., from 2.x to 3.x) typically introduce breaking changes. Minor versions (e.g., from 2.7.0 to 2.7.1) are usually backward-compatible but may introduce new features or bug fixes.

Upgrade Steps

1. Upgrade Airflow Packages

First, upgrade the Airflow Python package(s). If you are using specific providers, upgrade them as well. It's generally recommended to upgrade all relevant packages at once.


# Using pip
pip install apache-airflow --upgrade

# If using specific providers
pip install apache-airflow-providers-cncf-kubernetes apache-airflow-providers-postgres --upgrade

2. Upgrade the Metadata Database

After upgrading the Airflow packages, you need to upgrade your metadata database schema to match the new version. This is typically done by running the airflow db upgrade command.


airflow db upgrade

This command will apply any necessary schema migrations. Monitor the output for any errors. If you encounter issues, consult the Troubleshooting section.

3. Update Configuration Files

Review your airflow.cfg (or environment variables) for any new configuration options or changes. Refer to the Configuration Management documentation for details on available settings.

4. Restart Airflow Components

Once the database is upgraded and configurations are reviewed, restart all Airflow components:

Webserver
Scheduler
Workers (if applicable)

Ensure that each component starts successfully without errors.

5. Test Your DAGs

After restarting, thoroughly test your DAGs to ensure they run as expected. Pay attention to:

DAG parsing
Task execution
Task dependencies
Integrations with external systems
Custom operators and hooks

Downgrading

Downgrading Airflow is generally not recommended and can be complex, especially if the metadata database has been upgraded. If you must downgrade, it usually involves restoring your metadata database from a backup and reverting the Airflow packages. Always test downgrading thoroughly in a staging environment if it's a critical requirement.

Troubleshooting Common Issues

Database Migration Errors

If airflow db upgrade fails, the most common cause is an incomplete or interrupted previous migration. You might need to manually address the SQL statements or consult the specific error messages in the Airflow logs.

Warning: Manual database modifications should be done with extreme caution. It is often safer to restore from a backup if possible.

DAG Parsing Errors

Newer Airflow versions may have stricter parsing rules or deprecate certain DAG-writing patterns. Check the webserver and scheduler logs for detailed error messages. Common issues include:

Using deprecated imports or functions.
Syntax errors introduced by changes in Airflow's internal APIs.
Incorrectly formatted configurations within DAG files.

Provider Package Compatibility

Ensure that the versions of your provider packages are compatible with the Airflow version you are upgrading to. Check the provider documentation for compatibility matrices.

Component Startup Failures

If the webserver, scheduler, or workers fail to start, examine their respective logs for specific error messages. This could be due to configuration issues, permission problems, or unmet dependencies.

Tip: When encountering issues, always refer to the detailed logs generated by Airflow components. These logs are your primary source for diagnosing problems.

Best Practices for Upgrades

Automate Upgrades: Whenever possible, automate your upgrade process using infrastructure-as-code tools and CI/CD pipelines.
Version Control Everything: Keep your Airflow code, configurations, and DAGs under version control.
Monitor Closely: After an upgrade, monitor your Airflow instance and workflows closely for any anomalies.
Communicate: Inform your team and stakeholders about upcoming upgrades and potential impacts.