Registry /
workflow / apache-airflow-providers-sqlite
The `apache-airflow-providers-sqlite` package provides the necessary components to interact with SQLite databases within Apache Airflow DAGs. It includes the `SqliteHook` for programmatic access and `SqliteOperator` for defining tasks. This provider is currently at version 4.3.1 and typically releases alongside major Apache Airflow provider updates.
Install & Compatibility
Where this runs
No compatibility data collected yet for this library.
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
SqliteHook
✓ from airflow.providers.sqlite.hooks.sqlite import SqliteHook
SqliteOperator
✓ from airflow.providers.sqlite.operators.sqlite import SqliteOperator
This quickstart demonstrates a basic Airflow DAG using the `SqliteOperator` to create a table, insert data, and query it. Ensure an Airflow connection named `sqlite_default` is configured, or it will default to an in-memory database. For a persistent database file, specify the path in the 'Host' field of the connection or in the 'Extra' field (e.g., `{"database":"/path/to/my.db"}`).
from __future__ import annotations
import pendulum
from airflow.models.dag import DAG
from airflow.providers.sqlite.operators.sqlite import SqliteOperator
with DAG(
dag_id="sqlite_example_dag",
start_date=pendulum.datetime(2023, 1, 1, tz="UTC"),
schedule=None,
catchup=False,
tags=["sqlite", "example"],
) as dag:
create_table = SqliteOperator(
task_id="create_table",
sqlite_conn_id="sqlite_default",
sql="""
CREATE TABLE IF NOT EXISTS test_table (
id INTEGER PRIMARY KEY,
name TEXT
);
""",
)
insert_data = SqliteOperator(
task_id="insert_data",
sqlite_conn_id="sqlite_default",
sql="INSERT INTO test_table (name) VALUES ('Airflow'), ('SQLite');",
)
query_data = SqliteOperator(
task_id="query_data",
sqlite_conn_id="sqlite_default",
sql="SELECT * FROM test_table;",
handler=lambda x: [print(row) for row in x] # Example handler to print results to logs
)
create_table >> insert_data >> query_data
Debug
Known issues
gotchaSQLite database persistence and location: By default, if the `sqlite_default` connection's 'Host' or 'Extra' field is not configured with a database path, the tasks will use an in-memory SQLite database (`:memory:`). This means all data will be lost after the task completes. For persistent storage, you *must* specify a file path (e.g., `/opt/airflow/dags/data/my_db.db`) in the Airflow connection.fixConfigure the `sqlite_default` Airflow connection to specify a file path for the 'Host' field or in the 'Extra' field as `{"database":"/path/to/my.db"}`. affects: All versions
gotchaNot suitable for distributed Airflow: SQLite is a file-based database. In distributed Airflow setups (e.g., using CeleryExecutor or KubernetesExecutor with multiple workers), the SQLite database file must reside on a shared, mounted filesystem accessible by all workers. Otherwise, each worker will operate on its own independent copy of the database, leading to data inconsistency. It's generally recommended only for single-node Airflow deployments or for ephemeral, task-local data.fixUse SQLite only for single-node Airflow environments or for temporary, task-local data. For shared, persistent state in distributed setups, consider a client-server database like PostgreSQL or MySQL.
affects: All versions
breakingAirflow 1.x incompatibility: This provider package (like all `apache-airflow-providers-*` packages) is designed exclusively for Apache Airflow 2.0 and later. Attempting to install or use it in an Airflow 1.x environment will result in import errors and general incompatibility.fixEnsure your Airflow environment is version 2.0 or newer. Upgrade Airflow if necessary.
affects: < 2.0.0 (Airflow)
gotchaNo SQLite Sensor: Unlike some other database providers (e.g., Postgres, MySQL), the `apache-airflow-providers-sqlite` package currently only offers `SqliteHook` and `SqliteOperator`. There is no dedicated `SqliteSensor` available for polling database states directly.fixImplement custom sensing logic using a `PythonOperator` calling the `SqliteHook` or combine with other Airflow features like `ExternalTaskSensor` if monitoring external processes that modify the database.
affects: All versions
breakingIncompatible Python version: Apache Airflow 2.x and its provider packages have specific Python version requirements. Python 3.13 is currently not supported by Apache Airflow 2.x or its providers. Attempting to install or use the provider with an unsupported Python version will result in installation failures or `ModuleNotFoundError`.fixUse a supported Python version for Airflow 2.x (e.g., Python 3.8, 3.9, 3.10, 3.11). Refer to the official Apache Airflow documentation for the latest supported Python versions.
affects: > 3.12 (Python)
Errors
Common errors & fixes
ModuleNotFoundError: No module named 'airflow.providers.sqlite'
The 'apache-airflow-providers-sqlite' package is not installed.
fixInstall the package using 'pip install apache-airflow-providers-sqlite'.
ImportError: cannot import name 'SqliteOperator' from 'airflow.providers.sqlite.operators.sqlite'
The 'SqliteOperator' class is not available in the specified module.
fixEnsure you are importing 'SqliteOperator' from 'airflow.providers.sqlite.operators.sqlite'.
AttributeError: module 'airflow.providers.sqlite.hooks.sqlite' has no attribute 'SqliteHook'
The 'SqliteHook' class is not found in the 'airflow.providers.sqlite.hooks.sqlite' module.
fixVerify that 'SqliteHook' is correctly defined in 'airflow.providers.sqlite.hooks.sqlite'.
sqlite3.OperationalError: unable to open database file
Airflow's SQLite provider cannot access the specified database file, often due to incorrect file path, insufficient file permissions, or the database file not existing at the given location. This can also happen if the database path is relative when Airflow expects an absolute path.
fixEnsure the database file path specified in your Airflow connection (`sqlite_conn_id`) is absolute, the file exists, and the Airflow user has read/write permissions to both the file and its parent directory. For example, in Airflow Connections, set the Host field to an absolute path like `/path/to/your_database.db`.
airflow.exceptions.AirflowConfigException: Cannot use relative path: `sqlite:///C:\airflow/airflow.db` to connect to sqlite. Please use absolute path such as `sqlite:////tmp/airflow.db`
Airflow's SQLite configuration, particularly for the metadata database or a connection, was set with a relative path, but Airflow requires an absolute path for SQLite database files.
fixUpdate your `AIRFLOW__DATABASE__SQL_ALCHEMY_CONN` configuration or your custom SQLite connection (e.g., `sqlite_default`) to use an absolute path for the SQLite database file. For example, change `sqlite:///airflow.db` to `sqlite:////home/user/airflow/airflow.db`.
Audit
Dependencies
apache-airflowrequiredRequired for provider functionality; this package extends Apache Airflow core.
Agent activity
0 hits · last 30 days
No traffic data recorded yet.