Install & Compatibility
Where this runs
tested against v0.8.0 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.95 runs
installs and imports cleanly · install 0.0s · import 1.266s · 168MB
glibcpy 3.10–3.95 runs
installs and imports cleanly · install 8.4s · import 1.186s · 161MB
168MB installed
● package 168MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
LivySession
✓ from livy import LivySession
✗ from pylivy import LivySession
The PyPI package is named 'livy', not 'pylivy', despite the GitHub repository name.
LivyBatch
✓ from livy import LivyBatch
This quickstart demonstrates how to create an interactive Livy session, run PySpark code remotely, and download results. It uses environment variables for the Livy server URL and authentication credentials for security.
import os
from livy import LivySession
from requests.auth import HTTPBasicAuth
# Configure Livy server URL and optional authentication
LIVY_URL = os.environ.get('LIVY_SERVER_URL', 'http://localhost:8998')
LIVY_USERNAME = os.environ.get('LIVY_USERNAME', 'livy_user')
LIVY_PASSWORD = os.environ.get('LIVY_PASSWORD', 'livy_password')
auth = HTTPBasicAuth(LIVY_USERNAME, LIVY_PASSWORD) if LIVY_USERNAME else None
try:
with LivySession.create(LIVY_URL, auth=auth) as session:
print(f"Livy session {session.id} created successfully.")
# Run some Spark code on the remote cluster
session.run("df = spark.createDataFrame([(1, 'Alice'), (2, 'Bob')], ['id', 'name'])")
session.run("filtered_df = df.filter(df.name == 'Bob')")
# Retrieve the result (e.g., as a pandas DataFrame)
local_df = session.download("filtered_df")
print("Downloaded DataFrame:")
print(local_df)
except Exception as e:
print(f"An error occurred: {e}")
print("Ensure a Livy server is running and accessible at the specified URL.")
Errors
Common errors & fixes
SyntaxError: invalid syntax
The 'pylivy' library requires Python 3.6 or later due to its use of modern Python syntax features like type hints, but it is being run with an older Python 2.x interpreter.
fixEnsure you are using a Python 3.6+ environment. This can be achieved by activating a virtual environment with the correct Python version or explicitly invoking the script with 'python3' instead of 'python'.
ConnectionRefusedError: [Errno 111] Connection refused
The Python client is unable to establish a network connection with the Livy server. This typically indicates that the Livy server is not running, is configured on a different host or port than specified, or a firewall is blocking the connection.
fixVerify that the Livy server is running and accessible at the specified URL and port (e.g., 'http://localhost:8998'). Check Livy server logs for startup issues, confirm network connectivity, and review any active firewall rules.
{"msg":"requirement failed: Session isn't active."}
The Livy session failed to initialize properly, died prematurely due to resource constraints, encountered incompatible Spark configurations (e.g., incorrect Python version for PySpark sessions on the cluster), or was created with an unsupported session 'kind'.
fixExamine the Livy server logs for detailed error messages. Ensure that Spark is correctly configured on the cluster, the 'kind' parameter (e.g., 'pyspark') is appropriate for your Livy/Spark setup, and sufficient resources (CPU, memory) are available. Specifically, check the 'spark.pyspark.python' setting on the Spark cluster.
{"msg":"requirement failed: Local path <path/to/job/file> cannot be added to user sessions."}
Livy's default security configuration often restricts the direct submission of local files (like .py or .jar files) from the client machine to the Spark cluster. It typically expects files to reside on a distributed file system (e.g., HDFS, S3) or requires explicit whitelisting of local directories on the Livy server.
fixUpload your Python files or JARs to a distributed file system (e.g., HDFS, S3) and provide their URI to Livy during session or batch creation. Alternatively, if submitting from a local path on the Livy server machine, configure the 'livy.file.local-dir-whitelist' property in 'livy.conf' to include the allowed directories.
ModuleNotFoundError: No module named 'livy'
The 'pylivy' package has not been installed in the Python environment currently in use, or the Python interpreter cannot locate the installed package.
fixInstall the 'pylivy' package using pip: 'pip install livy'. If working with virtual environments, ensure the correct environment is activated. If multiple Python versions are present, use 'pip3 install livy' or 'python3 -m pip install livy' to target the appropriate Python installation.
Upgrade
Version history
0.8.0latest on PyPI · released Jan 27, 2021
Audit
Dependencies
requestsrequiredUsed internally for making HTTP requests to the Livy server.