Registry / workflow / airflow-provider-duckdb

airflow-provider-duckdb

library0.2.0pypypi✓ verified 80d ago

An Apache Airflow provider that integrates DuckDB, enabling SQL execution, data loading, and transformations within Airflow DAGs. Current version 0.2.0 targets DuckDB 0.3.x+ and Airflow 2.x. Release cadence is irregular.

workflow database data devops

pip install airflow-provider-duckdb

package ↗homepage ↗

INSTALL

IMPORT

SIG · AIRFLOW-PROVIDER-D

airflow-provider-duckdb

workflowpythonv0.2.0

harness data pending

Install & Compatibility

Where this runs

No compatibility data collected yet for this library.

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

get_provider_info

✓ from duckdb_provider import get_provider_info

✗ from airflow.providers.duckdb.operators.duckdb import DuckDBOperator

Minimal DAG creating a table, inserting rows, and selecting data with DuckDBOperator.

from datetime import datetime from airflow import DAG from airflow.providers.duckdb.operators.duckdb import DuckDBOperator default_args = {'owner': 'airflow', 'start_date': datetime(2023, 1, 1)} with DAG('duckdb_example', schedule_interval=None, default_args=default_args, catchup=False) as dag: create_table = DuckDBOperator( task_id='create_table', sql="CREATE TABLE IF NOT EXISTS test (id INTEGER, name VARCHAR)", duckdb_conn_id='duckdb_default' ) insert_data = DuckDBOperator( task_id='insert_data', sql="INSERT INTO test VALUES (1, 'Alice'), (2, 'Bob')", duckdb_conn_id='duckdb_default' ) select_data = DuckDBOperator( task_id='select_data', sql="SELECT * FROM test", duckdb_conn_id='duckdb_default', do_xcom_push=True ) create_table >> insert_data >> select_data

Debug

Known issues

breakingOperator `sql` parameter expects a single SQL string; multiple statements must be separated by semicolons (DuckDB multi-statement support is limited)

fix

Combine statements in one string with semicolons or use multiple tasks.

affects: all

gotchaConnection ID 'duckdb_default' is not automatically created; you must configure it in Airflow UI or environment. Missing connection leads to cryptic Airflow errors.

fix

Set environment variable AIRFLOW_CONN_DUCKDB_DEFAULT='duckdb://localhost:5432/mydb' or configure via UI.

affects: all

deprecatedOld import path `from airflow_provider_duckdb import ...` is deprecated; always use `from airflow.providers.duckdb import ...`

fix

Use dotted provider import path as shown in quickstart above.

affects: >=0.2.0

Upgrade

Version history

0.2.0latest on PyPI · released Jun 26, 2023

Audit

Dependencies

apache-airflowrequiredRequired runtime dependency

duckdbrequiredCore DuckDB library

Agent activity

56 hits · last 30 days

node

OpenAI (training)

Resources

packagehttps://pypi.org/project/airflow-provider-duckdb/ ↗

homepagehttp://astronomer.io/ ↗