Registry /
workflow / apache-airflow-providers-elasticsearch
Install & Compatibility
Where this runs
tested against v6.6.0 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.920 runs
installs and imports cleanly · install 0.0s · import 0.000s · 261.7MB
glibcpy 3.10–3.920 runs
installs and imports cleanly · install 24.0s · import 0.000s · 260MB
263MB installed
● package 263MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
ElasticsearchHook
✓ from airflow.providers.elasticsearch.hooks.elasticsearch import ElasticsearchHook
✗ from airflow.providers.elasticsearch.hooks.elasticsearch import ElasticsearchHook
This quickstart demonstrates how to use the `ElasticsearchPythonHook` within an Airflow DAG. It shows how to connect to an Elasticsearch instance (expecting a connection ID `elasticsearch_default` or `ELASTICSEARCH_HOST` env var), perform a simple search query, and add a document to an index. Remember to configure your Elasticsearch connection in Airflow UI or via environment variables.
import os
from datetime import datetime
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.providers.elasticsearch.hooks.elasticsearch import ElasticsearchPythonHook
def _query_elasticsearch(**kwargs):
# Ensure your Airflow Connection 'elasticsearch_default' is configured
# or set ELASTICSEARCH_HOST environment variable.
es_host = os.environ.get('ELASTICSEARCH_HOST', 'http://localhost:9200')
es_hook = ElasticsearchPythonHook(hosts=[es_host])
# Simple match_all query
query = {"query": {"match_all": {}}}
try:
result = es_hook.search(index='your_index_name', query=query)
print(f"Successfully queried Elasticsearch. Hits: {result['hits']['total']['value']}")
# Example: Ingesting data
doc = {"timestamp": datetime.now().isoformat(), "message": "Hello from Airflow!"}
es_hook.add_doc(index='airflow_logs', doc=doc)
print("Successfully added a document to 'airflow_logs'.")
except Exception as e:
print(f"Error interacting with Elasticsearch: {e}")
raise
with DAG(
dag_id='elasticsearch_quickstart_dag',
start_date=datetime(2023, 1, 1),
schedule_interval=None,
catchup=False,
tags=['elasticsearch', 'example'],
) as dag:
query_es_task = PythonOperator(
task_id='query_and_add_doc_to_elasticsearch',
python_callable=_query_elasticsearch,
)
Debug
Known issues
breakingMinimum Airflow version requirement for providers has increased. For `apache-airflow-providers-elasticsearch` versions 6.4.0 and above, `Apache Airflow >=2.11.0` is required. Older provider versions have different minimum Airflow requirements.fixEnsure your Airflow environment meets the minimum version requirement. Upgrade Airflow to 2.11.0 or newer if using provider >=6.4.0.
affects: >=6.4.0
breakingElasticsearch client compatibility changes. Provider version 5.0.0 upgraded to use Elasticsearch 8 Python client, recommending an upgrade of your Elasticsearch database to 8. Provider 6.5.1 adds support for Elasticsearch 9.fixUpgrade your Elasticsearch cluster to version 8 or 9 for full compatibility, especially if experiencing issues with client-server communication.
affects: >=5.0.0
breakingRemoval of Jinja templating support in `log_id` for Elasticsearch task logging. Previously undocumented Jinja templating was removed.fixIf you used Jinja templates for `log_id`, migrate to a template string format (e.g., `{dag_id}-{task_id}-{execution_date}-{try_number}`). Update existing Elasticsearch documents if `execution_date` was templated. affects: >=2.0.1
breakingThe `apply_default` decorator was removed in Airflow 2.1.0. Provider versions `2.0.1` and newer requiring `Airflow 2.1.0+` might cause automatic Airflow package upgrades if your Airflow version is older, necessitating a manual `airflow upgrade db`.fixUpgrade Airflow to at least version 2.1.0 before installing or upgrading this provider to avoid automatic Airflow upgrades and subsequent database migration requirements.
affects: >=2.0.1 (with Airflow <2.1.0)
gotchaThe `write_to_es` Elasticsearch task logging feature was incompatible with Airflow 3 in provider versions prior to 6.5.0.fixUpgrade to `apache-airflow-providers-elasticsearch>=6.5.0` to ensure proper functionality of `write_to_es` with Airflow 3.
affects: <6.5.0 (with Airflow 3)
gotchaIncorrect log ordering in Elasticsearch can occur if the 'offset' field is not properly configured. This was a known issue that required specific Fluent Bit configuration and Airflow settings.fixEnsure your Fluent Bit configuration appends an 'Offset_Key' to logs and set `AIRFLOW__ELASTICSEARCH__OFFSET_FIELD` to match this key (e.g., `AIRFLOW__ELASTICSEARCH__OFFSET_FIELD="custom_offset"`).
affects: <2.0.3 (and specific configurations)
Upgrade
Version history
6.6.0latest on PyPI · released Jun 7, 2026
Audit
Dependencies
apache-airflowrequiredCore Airflow dependency. Provider version 6.4.0+ requires Airflow >=2.11.0.
elasticsearchrequiredPython client for Elasticsearch. Provider 6.5.1 supports Elasticsearch 9.
elasticsearch-dbapirequiredUsed for Elasticsearch SQL API interaction.
elasticsearch-dslrequiredDSL for building and running complex queries.
apache-airflow-providers-common-sqloptionalCross-provider dependency for SQL-related features, installed via `[common.sql]` extra.