Apache Airflow is an open-source platform used to programmatically author, schedule, and monitor workflows, particularly for data pipelines. It defines workflows as Directed Acyclic Graphs (DAGs) in Python, enabling dynamic, scalable, and extensible orchestration. The current stable version is 3.1.8, with releases occurring regularly to introduce new features, improvements, and bug fixes.
pip install "apache-airflow[celery,cncf.kubernetes,http,postgres,amazon]"==3.1.8 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.1.8/constraints-3.10.txt"Verified import paths — ran on the pinned version, not inferred.
This quickstart defines a simple DAG with Bash and Python operators. To run this locally after installing Airflow, save the code as a `.py` file (e.g., `dags/quickstart_dag.py`) in your `AIRFLOW_HOME/dags` directory. Then, initialize the database and start the Airflow standalone environment. **To set up and run Airflow locally (assuming `apache-airflow` is installed with `sqlite` support via `pip install "apache-airflow[sqlite]"`):** ```bash # (Optional) Set AIRFLOW_HOME, e.g., to a temporary directory export AIRFLOW_HOME=$(pwd)/airflow_home # Initialize the database and create an admin user (first time only) airflow standalone # Follow prompts to set admin password. This command also starts webserver, scheduler, and triggerer. # You can also start components separately: # airflow db migrate # airflow users create --username admin --firstname Airflow --lastname Admin --role Admin --email admin@example.com -p mypassword # airflow webserver --port 8080 # airflow scheduler # airflow triggerer # After starting, visit http://localhost:8080 to enable the DAG. ``` Remember to define `AIRFLOW_HOME` before running `airflow standalone` or `airflow db init`.
Rewrite task code to use the Task Execution API or the Airflow Python Client for database interactions. Avoid direct SQLAlchemy imports or session usage within task logic. Consider requesting new API endpoints or Task SDK features if required functionality is missing.
Migrate existing SubDAGs to use TaskGroups for grouping related tasks, or explore using Assets and Data Aware Scheduling for more advanced scenarios.
Update Airflow configuration to use `LocalExecutor` instead of `SequentialExecutor`.
Remove SLA definitions from DAGs. Monitor for the introduction of 'Deadline Alerts' as a replacement.
Always use full Python package paths for imports within DAGs. Ensure shared code is either installed as a Python package or added to `PYTHONPATH` with a unique top-level name to prevent clashes.
Access Airflow Variables and Connections inside operator `execute()` methods, or pass them to operators using Jinja templating, which defers evaluation until task execution. For sensitive data, use Secrets Backend.
Upgrade your Python environment to version 3.10 or a newer compatible version (e.g., Python 3.10, 3.11, 3.12).
Add C/C++ build tools to the Dockerfile before installing Python packages (e.g., `apk add --no-cache build-base g++` for Alpine), or use a non-Alpine Python base image (e.g., `python:3.13-slim`).