Registry /
workflow / apache-airflow-providers-apache-livy
Install & Compatibility
Where this runs
tested against v4.6.1 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.95 runs
installs and imports cleanly · install 0.0s · import 6.116s · 265.7MB
glibcpy 3.10–3.95 runs
installs and imports cleanly · install 26.6s · import 5.710s · 266MB
267MB installed
● package 267MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
LivyOperator
✓ from airflow.providers.apache.livy.operators.livy import LivyOperator
LivyHook
✓ from airflow.providers.apache.livy.hooks.livy import LivyHook
This quickstart demonstrates a basic Airflow DAG using the `LivyOperator` to submit a Spark Pi calculation job to a Livy server. Ensure you have a 'livy_default' connection configured in your Airflow UI pointing to your Livy instance. The `file` parameter should point to the Spark example JAR on your cluster's accessible path.
from __future__ import annotations
import os
from datetime import datetime
from airflow.models.dag import DAG
from airflow.providers.apache.livy.operators.livy import LivyOperator
with DAG(
dag_id='livy_spark_pi_example',
schedule=None,
start_date=datetime(2023, 1, 1),
catchup=False,
tags=['livy', 'spark', 'example'],
) as dag:
# Ensure 'livy_default' connection is configured in Airflow UI (Admin -> Connections)
# with appropriate host and port for your Livy server.
# Example: livy_default, Host: localhost, Port: 8998
submit_spark_pi_job = LivyOperator(
task_id='submit_spark_pi_job',
file=os.getenv('LIVY_SPARK_PI_JAR', '/opt/spark/examples/jars/spark-examples_2.12-3.2.1.jar'),
class_name='org.apache.spark.examples.SparkPi',
args=['10'], # Example: Calculate Pi with 10 iterations
livy_conn_id='livy_default',
driver_memory='1g',
executor_memory='1g',
num_executors=1
)
Debug
Known issues
breakingProvider versions have specific minimum Airflow core versions. For `apache-airflow-providers-apache-livy` version 4.5.x, Airflow 2.11.0 or higher is required. Using older Airflow versions with newer providers can lead to incompatibility errors or unexpected behavior.fixAlways check the provider's documentation for the minimum supported Airflow version. Upgrade your Airflow installation if it doesn't meet the provider's requirements. For current version 4.5.5, ensure Airflow >= 2.11.0.
affects: <4.5.x (for older Airflow versions)
breakingOlder provider versions (e.g., 3.0.0 and earlier) introduced breaking changes due to the removal of the `apply_default` decorator. If you upgrade the Livy provider on an Airflow instance older than 2.1.0, Airflow might automatically upgrade, necessitating a manual `airflow upgrade db` command.fixWhen upgrading providers, especially on older Airflow installations, be aware of potential core Airflow upgrades. Always run `airflow upgrade db` after an Airflow upgrade to ensure database schema compatibility. It's recommended to upgrade Airflow to a modern version (>=2.11.0) before upgrading to recent Livy provider versions.
affects: <3.0.0 (provider versions)
gotchaThe `LivyOperator` requires a Livy connection to be configured in the Airflow UI (Admin -> Connections). The default `livy_conn_id` is 'livy_default'. Misconfiguration or absence of this connection will result in task failures.fixBefore running a DAG with `LivyOperator`, navigate to Airflow UI -> Admin -> Connections. Create or verify a connection named 'livy_default' (or your specified `livy_conn_id`) with the correct host and port for your Apache Livy server.
affects: All
Errors
Common errors & fixes
ModuleNotFoundError: No module named 'airflow_livy'
The 'airflow_livy' module is not installed or not available in the Python environment.
fixEnsure that the 'apache-airflow-providers-apache-livy' package is installed using pip: 'pip install apache-airflow-providers-apache-livy'.
AirflowException: Batch {batch_id} did not succeed
The Livy batch job did not complete successfully, possibly due to issues in the Spark application or cluster.
fixCheck the Livy and Spark logs to identify the cause of failure and address any issues in the Spark application or cluster configuration.
AttributeError: module 'airflow.providers.apache.livy.operators.livy' has no attribute 'LivyBatchOperator'
The 'LivyBatchOperator' has been renamed or is not available in the current version of the 'apache-airflow-providers-apache-livy' package.
fixUse 'LivyOperator' instead of 'LivyBatchOperator' as per the latest documentation.
Upgrade
Version history
4.6.1latest on PyPI · released Aug 23, 2026
Audit
Dependencies
apache-airflowrequiredCore Airflow installation is required to use providers.
apache-airflow-providers-httpoptionalRequired for HTTP connection management.
apache-airflow-providers-common-compatoptionalRequired for compatibility features within the Airflow ecosystem.
aiohttpoptionalAsynchronous HTTP client library, used by underlying HTTP operations.