dag-factory is an open-source Python library that dynamically generates Apache Airflow DAGs from YAML configuration files. It enables users to define complex data pipelines using a declarative syntax, reducing the need for extensive Python knowledge and promoting consistency across many DAGs. The library is actively maintained by Astronomer, with the current stable version being 1.0.1, and receives regular updates and feature enhancements.
pip install dag-factoryVerified import paths — ran on the pinned version, not inferred.
This quickstart demonstrates how to define an Airflow DAG using a YAML configuration and then use `dag-factory`'s `load_yaml_dags` function to generate it. The `load_yaml_dags` function is designed to be called within an Airflow DAG file, automatically populating the `globals()` dictionary with the generated DAGs, making them discoverable by the Airflow scheduler.
If your DAGs rely on these providers, you must explicitly install them, e.g., `pip install dag-factory[all]` or `pip install dag-factory[kubernetes]`.
Use the recommended function `from dagfactory import load_yaml_dags` to generate DAGs from your YAML configurations.
Use the `schedule` parameter instead to define DAG schedules.
Remove any calls to `example_dag_factory.clean_dags(globals())` from your DAG files. Rely on Airflow's native mechanisms for DAG lifecycle management.
Ensure your environment meets the minimum requirements: Python 3.9+ and Apache Airflow 2.4+.
Switch to Airflow's direct equivalents: `dagrun_timeout`, `retry_delay`, `sla`, `execution_delta`, `execution_timeout`. Ensure these are specified using `__type__: datetime.timedelta` for time-related values where applicable.
For Airflow 3.1.0 and above, `sla_miss_callback` is no longer supported directly within `dag-factory`. Airflow 3 deprecates SLA features in favor of 'deadline alerts'. Consider migrating to deadline alerts or Airflow's standard callback mechanisms.
Ensure 'dag-factory' is included in your Airflow environment's `requirements.txt` file and that these requirements are installed. For Docker-based Airflow, this means rebuilding the image or ensuring the `requirements.txt` is processed during environment setup.
Ensure your YAML configuration correctly references the `python_callable` function name without parentheses and passes any necessary arguments using the `op_kwargs` dictionary. Example:
```yaml
tasks:
my_task:
operator: airflow.operators.python.PythonOperator
python_callable: my_module.my_function # Reference the function
op_kwargs:
arg1: "value1"
arg2: "value2"
```Separate your `defaults.yml` file into a dedicated directory and provide its path using the `defaults_config_path` parameter to `load_yaml_dags`, distinct from your `dag_folder` where other DAG YAMLs are stored.
Example:
```python
from dagfactory import load_yaml_dags
load_yaml_dags(
globals_dict=globals(),
dag_folder="/opt/airflow/dags/configs", # Path to your DAG YAMLs
defaults_config_path="/opt/airflow/dags/defaults" # Path to defaults.yml
)
```Update your YAML configuration files to use `schedule` instead of `schedule_interval`. Example: ```yaml my_dag: schedule: "0 0 * * *" ```