sparkmagic

library0.23.0pypypi✓ verified 83d ago

SparkMagic is a Jupyter/IPython magic command extension for interactively working with Spark clusters via Livy (REST API). It supports multiple sessions, language interpreters (PySpark, Spark, Scala, SQL), and automatic visualization of DataFrames. Current version is 0.23.0 (released 2025-02-14) with active but low-frequency releases; the project is in maintenance mode.

data devops http-networking

pip install sparkmagic

github ↗package ↗

INSTALL

IMPORT

SIG · SPARKMAGIC

sparkmagic

datapythonv0.23.0

Install

—

Import

—

Disk

—

Pass rate

0/ 10

Env Coverage0 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v? · pip install

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.95 runs

build_error

glibc

py 3.10–3.95 runs

build_error

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

sparkmagic

✓ import sparkmagic

Direct import is rarely needed; load via %load_ext sparkmagic

Load the extension, manage Spark sessions, and execute code against a remote Spark cluster via Livy.

# In a Jupyter notebook cell: %load_ext sparkmagic %manage_spark # Then start a session: %%spark -s mysession # Create a simple DataFrame: df = spark.range(10) print(df.count())

Debug

Known issues

breakingIn version 0.20.0, the extension changed to support async kernel execution in ipykernel>=6. Custom magic commands or code that assumes synchronous execution may break.

fix

Update any custom code to handle async execution; test with simple cells first.

affects: >=0.20.0

deprecatedPython 3.6 support was dropped in v0.20.0. Users on Python 3.6 must stay on v0.19.2 or lower.

fix

Upgrade Python to 3.7+.

affects: >=0.20.0

gotchaYou must have a running Livy server reachable from the notebook server. The extension does not start Livy.

fix

Ensure Livy is installed and running, and configure the endpoint via %manage_spark or config file.

affects: all

gotchaThe %manage_spark widget may not display correctly in JupyterLab without the appropriate JupyterLab extensions (jupyterlab-sparkmagic).

fix

Install the JupyterLab extension: `jupyter labextension install jupyterlab-sparkmagic`

affects: all

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'sparkmagic'

sparkmagic is not installed in the current Python environment.

fix

Run `pip install sparkmagic` in the terminal and restart the kernel.

ImportError: cannot import name 'SparkMagic' from 'sparkmagic'

Incorrect import path; sparkmagic provides a magic extension, not a direct importable class.

fix

Use `%load_ext sparkmagic` in a notebook cell instead of a Python import.

TypeError: 'NoneType' object is not subscriptable

Livy endpoint not configured or not responding. The extension returns None for session info.

fix

Check your Livy URL and ensure the Livy server is running. Use `%manage_spark` to set the endpoint.

Upgrade

Version history

0.23.0latest on PyPI · released Jul 7, 2025

Audit

Dependencies

ipythonrequiredCore dependency for magic command support; version >=7 and <9 required.

livyrequiredSpark sessions are created via Livy REST API.

pandasoptionalUsed for DataFrame conversion and display; pinned <2.0.0 in some versions.

tornadorequiredUsed for asynchronous HTTP communication with Livy.

Agent activity

28 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/jupyter-incubator/sparkmagic ↗

packagehttps://pypi.org/project/sparkmagic/ ↗