Registry / data / apache-airflow-providers-airbyte

apache-airflow-providers-airbyte

library6.0.1pypypi✓ verified 24d ago

The `apache-airflow-providers-airbyte` package provides Apache Airflow operators and sensors to interact with Airbyte, an open-source data integration platform. It enables users to trigger and monitor Airbyte synchronization jobs directly from Airflow DAGs. The current version is 5.4.0, supporting Airflow >=2.11.0 and Python >=3.10, and it maintains a regular release cadence with ongoing development.

data workflow

pip install apache-airflow-providers-airbyte

github ↗package ↗homepage ↗docs ↗changelog ↗

INSTALL

IMPORT

SIG · APACHE-AIRFLOW-PRO

apache-airflow-providers-airbyte

datapythonv6.0.1

Install

25.1s avg

Import

6240ms

Disk

268MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v6.0.1 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.910 runs

installs and imports cleanly · install 0.0s · import 6.420s · 268.8MB

glibc

py 3.10–3.910 runs

installs and imports cleanly · install 25.1s · import 6.060s · 267MB

268MB installed

● package 268MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

AirbyteTriggerSyncOperator

✓ from airflow.providers.airbyte.operators.airbyte import AirbyteTriggerSyncOperator

AirbyteJobSensor

✓ from airflow.providers.airbyte.sensors.airbyte import AirbyteJobSensor

AirbyteHook

✓ from airflow.providers.airbyte.hooks.airbyte import AirbyteHook

Typically used internally by operators/sensors or for custom interactions, less common directly in DAGs.

This quickstart DAG demonstrates how to trigger and monitor an Airbyte synchronization job using the `AirbyteTriggerSyncOperator` and `AirbyteJobSensor`. Before running, you need to: 1. Install the provider: `pip install apache-airflow-providers-airbyte[http]`. 2. Configure an Airflow 'Airbyte' connection (e.g., `AIRBYTE_CONNECTION_ID`) pointing to your Airbyte instance's API (e.g., `http://localhost:8001`). 3. Obtain the UUID of the specific Airbyte connection you wish to sync from the Airbyte UI (this is `AIRBYTE_SYNC_CONNECTION_ID`). 4. Set `AIRBYTE_CONN_ID` and `AIRBYTE_SYNC_CONN_ID` as environment variables or replace them directly in the DAG code.

import os from datetime import datetime, timedelta from airflow import DAG from airflow.providers.airbyte.operators.airbyte import AirbyteTriggerSyncOperator from airflow.providers.airbyte.sensors.airbyte import AirbyteJobSensor from airflow.utils.dates import days_ago AIRBYTE_CONNECTION_ID = os.environ.get('AIRBYTE_CONN_ID', 'your_airflow_airbyte_connection_id') AIRBYTE_SYNC_CONNECTION_ID = os.environ.get('AIRBYTE_SYNC_CONN_ID', 'your_airbyte_workspace_connection_id') with DAG( dag_id='example_airbyte_sync_dag', start_date=days_ago(1), schedule_interval=None, catchup=False, tags=['airbyte', 'example'], dagrun_timeout=timedelta(minutes=60), default_args={ 'owner': 'airflow', } ) as dag: trigger_airbyte_sync = AirbyteTriggerSyncOperator( task_id='trigger_airbyte_connection_sync', airbyte_conn_id=AIRBYTE_CONNECTION_ID, connection_id=AIRBYTE_SYNC_CONNECTION_ID, # This is the UUID of the Airbyte connection to trigger asynchronous=True, # Recommended to use with a sensor for long-running jobs ) monitor_airbyte_sync = AirbyteJobSensor( task_id='monitor_airbyte_connection_sync', airbyte_conn_id=AIRBYTE_CONNECTION_ID, airbyte_job_id=trigger_airbyte_sync.output, poke_interval=5, # Check every 5 seconds timeout=3600, # Timeout after 1 hour ) trigger_airbyte_sync >> monitor_airbyte_sync

Debug

Known issues

breakingAuthentication mechanism for Airbyte connections changed in provider version 4.0.0. It now uses `client_id` and `client_secret` instead of `login` and `password`. The `host` parameter for the Airflow Airbyte connection must be a Fully Qualified Domain Name (FQDN) including schema (e.g., `https://my.company:8000/airbyte/v1/`). The `api_type` parameter was also removed.

fix

Update your Airflow Airbyte connection configuration to use `client_id` and `client_secret` for authentication and ensure the host is a complete FQDN. Remove `api_type` parameter if present.

affects: >=4.0.0

breakingThe minimum required Apache Airflow version has increased over time. For provider version 5.4.0, Airflow 2.11.0+ is required. Additionally, provider version 5.4.0 removed the `polling_interval` parameter from `AirbyteJobSensor`, favoring `poke_interval`.

fix

Ensure your Airflow environment is at least 2.11.0. Replace `polling_interval` with `poke_interval` in `AirbyteJobSensor`.

affects: >=5.4.0

breakingProvider version 2.0.0 (and subsequently 2.1.0+) required Airflow 2.1.0+ due to the removal of the `apply_default` decorator. If upgrading the provider on an older Airflow, this could lead to automatic Airflow package upgrades and require a `airflow upgrade db` command.

fix

Upgrade Airflow to at least 2.1.0 (preferably the latest compatible version for your provider) before upgrading the Airbyte provider. Run `airflow upgrade db` if automatic upgrade occurs.

affects: 2.0.0 - <2.2.0

gotchaThe `AirbyteTriggerSyncOperator` is not idempotent by design. Re-triggering the operator may initiate a new sync job in Airbyte, depending on Airbyte's configuration for the connection. Users should be aware of the Airbyte source/destination sync mode.

fix

Understand Airbyte's sync configuration (e.g., incremental vs. full refresh) for the connection being triggered and design your DAGs accordingly to handle potential non-idempotent behavior.

affects: All

gotchaThe Airbyte operator in this provider is primarily designed to work with Airbyte self-managed instances (using its internal Config API). For orchestrating Airbyte Cloud, it's generally recommended to use Airflow's generic HTTP operators to interact with the newer Airbyte API directly.

fix

If using Airbyte Cloud, consider implementing `airflow.providers.http.operators.http.SimpleHttpOperator` for more direct API calls to the Airbyte Cloud API instead of the dedicated Airbyte provider operators.

affects: All

gotchaTo prevent conflicts and ensure Airflow maintains control, it's highly recommended to set the replication frequency for Airbyte connections triggered by Airflow to 'Manual' within the Airbyte UI.

fix

In the Airbyte UI, for any connection orchestrated by Airflow, set its replication frequency to 'Manual'.

affects: All

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'airflow.providers.airbyte'

The 'apache-airflow-providers-airbyte' package is not installed or not properly configured in the Airflow environment.

fix

Ensure that 'apache-airflow-providers-airbyte' is installed by adding it to your requirements.txt and verifying its presence in the Airflow UI under Admin > Providers.

HTTPConnectionPool(host='localhost', port=8001): Max retries exceeded with url: /api/v1/health (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f30e9e4fb10>: Failed to establish a new connection: [Errno 111] Connection refused'))

Airflow is unable to connect to the Airbyte API, possibly due to Airbyte not running, incorrect host/port configuration, or network issues.

fix

Verify that Airbyte is running and accessible at the specified host and port, and ensure that the Airflow connection settings for Airbyte are correctly configured.

RuntimeError: The package `apache-airflow-providers-airbyte:5.2.3` needs Apache Airflow 2.10.0+

The installed version of 'apache-airflow-providers-airbyte' requires a newer version of Apache Airflow than is currently installed.

fix

Upgrade Apache Airflow to version 2.10.0 or higher to be compatible with the installed 'apache-airflow-providers-airbyte' package.

requests.exceptions.MissingSchema: Invalid URL 'localhost/jobs': No scheme supplied.

The Airbyte connection configured in Airflow (via the UI or environment variables) is missing the 'http://' or 'https://' schema in the host field.

fix

Edit your Airflow HTTP connection for Airbyte and ensure the 'Host' field contains a complete URL, such as `http://localhost:8000` or `https://your-airbyte-instance.com`.

401 Client Error: Unauthorized for url:

The Airflow Airbyte connection is failing to authenticate with the Airbyte API, often due to incorrect `client_id`, `client_secret`, or `token_url` configuration, or an attempt to use authentication with Airbyte Open Source (OSS) where it might be disabled by default.

fix

Verify the `Client ID`, `Client Secret`, and `Token URL` in your Airflow Airbyte connection settings. For Airbyte OSS, the default `token_url` is typically `/api/public/v1/applications/token`. If using Airbyte OSS without authentication, consider if the standard operator is appropriate or if a custom `PythonOperator` is needed to interact with an unauthenticated Airbyte API.

Upgrade

Version history

6.0.1latest on PyPI · released Aug 8, 2026

Audit

Dependencies

apache-airflowrequiredCore Airflow functionality; provider version 5.4.0 requires Airflow >=2.11.0.

apache-airflow-providers-httpoptionalRequired for establishing HTTP connections, which Airbyte APIs utilize.

Agent activity

52 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/apache/airflow ↗

packagehttps://pypi.org/project/apache-airflow-providers-airbyte/ ↗

homepagehttps://airflow.apache.org ↗

docshttps://airflow.apache.org/docs/apache-airflow-providers-airbyte/5.4.2 ↗

changeloghttps://airflow.apache.org/docs/apache-airflow-providers-airbyte/5.4.2/changelog.html ↗