Initial thoughts on improving our Airflow setup

Hey everyone,

I've been looking into ways to make our Airflow environment at Airbnb more efficient. We're seeing increased load and occasional performance bottlenecks, especially during peak hours. I've identified a few areas that might benefit from optimization:

• Resource Allocation: How can we better tune Celery worker resources and executor configurations?
• DAG Performance: Identifying and optimizing slow-running tasks and improving DAG parsing times.
• Database Performance: Monitoring and optimizing the Airflow metadata database.
• Monitoring & Alerting: Enhancing our current monitoring setup for proactive issue detection.

I've been experimenting with dynamic task mapping and some advanced configurations for the Celery executor. Has anyone else had success with specific tuning parameters or architectural changes?

Looking forward to discussing ideas!

Re: Initial thoughts on improving our Airflow setup

Great topic, data_engineer_pro!

We've seen some improvements by implementing horizontal scaling for our Celery workers and tuning the `max_active_runs_per_dag` parameter on the scheduler. Also, aggressively caching XComs for DAGs that don't require frequent updates has helped reduce database load.

For DAG performance, I found that using `task_group` can significantly improve readability and allow for better parallelization within a logical group of tasks. We also adopted a policy of minimizing external API calls within task logic, preferring to batch them or use dedicated services.

# Example of dynamic task mapping
from airflow.decorators import task, dag

@dag(start_date=datetime(2023, 1, 1), schedule=None, catchup=False)
def dynamic_mapping_dag():
    @task
    def process_item(item):
        print(f"Processing item: {item}")
        return item * 2

    items = [1, 2, 3, 4, 5]
    results = process_item.expand(item=items)

dynamic_mapping_dag()
                

Re: Initial thoughts on improving our Airflow setup

Excellent points from both of you. I'd like to add emphasis on database health. Migrating to a more robust database solution (like Amazon RDS or Google Cloud SQL with proper indexing) and regularly running `VACUUM FULL` on key tables can make a substantial difference.

Also, consider the impact of the task logs. Storing them locally on worker nodes can fill up disk space quickly. Centralizing log storage (e.g., S3, GCS) and setting up log rotation policies are crucial for long-term stability.

Re: Initial thoughts on improving our Airflow setup

Building on the monitoring aspect, using tools like Prometheus and Grafana in conjunction with Airflow's built-in metrics has been a game-changer for us. We've set up alerts for scheduler heartbeat failures, high task execution times, and resource utilization spikes.

One trick we found useful is to add custom metadata to tasks that include business context, which helps in tracking down issues that span across different teams and systems.

Re: Initial thoughts on improving our Airflow setup

Fantastic insights, everyone! Journey_dev, the `task_group` suggestion is spot on for readability. Airflow_master, the database and log storage points are critical – we’ve definitely felt the pain there. Analyst_piper, integrating with Prometheus/Grafana is on our roadmap.

I’ve been thinking about the scheduler itself. Are there any specific configurations for high-concurrency scenarios that have proven effective? Perhaps tweaking `parallelism` and `dag_concurrency` in `airflow.cfg` beyond default recommendations?