Registry / data / pyspark-dist-explore

pyspark-dist-explore

library0.1.8pypypi✓ verified 88d ago

PySpark Distribution Explorer (pyspark-dist-explore, current version 0.1.8) is a Python library that enables creating histogram and density plots directly from PySpark DataFrames. It simplifies exploratory data analysis (EDA) for large datasets by leveraging Matplotlib and Pandas to visualize distributions. The project is currently in maintenance mode with infrequent updates.

data

pip install pyspark-dist-explore

package ↗

INSTALL

IMPORT

SIG · PYSPARK-DIST-EXPLO

pyspark-dist-explore

datapythonv0.1.8

Install

15.0s avg

Import

4177ms

Disk

396MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.1.8 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.910 runs

installs and imports cleanly · install 0.0s · import 4.281s · 393.3MB

glibc

py 3.10–3.910 runs

installs and imports cleanly · install 15.0s · import 4.073s · 377MB

396MB installed

● package 396MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

hist

✓ from pyspark_dist_explore import hist

density_plot

✓ from pyspark_dist_explore import density_plot

describe_pd

✓ from pyspark_dist_explore import describe_pd

This quickstart demonstrates how to initialize a SparkSession, create a sample DataFrame, and then use `hist`, `density_plot`, and `describe_pd` to visualize and summarize numerical distributions. Remember to call `plt.show()` to display the plots.

from pyspark_dist_explore import hist, density_plot, describe_pd from pyspark.sql import SparkSession import matplotlib.pyplot as plt import os # Ensure SparkSession is available (replace with your actual Spark setup) # For local testing, ensure pyspark is installed: pip install pyspark spark = SparkSession.builder.appName("DistExploreQuickstart").getOrCreate() # Create a sample PySpark DataFrame data = [ (1, "A", 10.5), (2, "B", 12.0), (3, "A", 11.2), (4, "C", 9.8), (5, "B", 13.1), (6, "A", 10.8), (7, "C", 9.5), (8, "B", 12.5), (9, "A", 11.0), (10, "C", 10.0) ] columns = ["id", "category", "value"] df = spark.createDataFrame(data, columns) print("Original DataFrame:") df.show() # 1. Generate a histogram fig_hist, ax_hist = plt.subplots() hist(ax_hist, df.select('value'), bins=5, color='skyblue', edgecolor='black') ax_hist.set_title('Histogram of Value') ax_hist.set_xlabel('Value') ax_hist.set_ylabel('Frequency') plt.tight_layout() plt.show() # Display the plot # 2. Generate a density plot fig_density, ax_density = plt.subplots() density_plot(ax_density, df.select('value'), color='green', fill=True, alpha=0.5) ax_density.set_title('Density Plot of Value') ax_density.set_xlabel('Value') ax_density.set_ylabel('Density') plt.tight_layout() plt.show() # Display the plot # 3. Get descriptive statistics as a Pandas DataFrame desc_df = describe_pd(df.select('value')) print("\nDescriptive Statistics (Pandas DataFrame):") print(desc_df) # Stop the SparkSession spark.stop()

Debug

Known issues

gotchaPySpark-dist-explore functions (`hist`, `density_plot`) require a `matplotlib.axes.Axes` object as their first argument. You must create a Matplotlib figure and axes explicitly before calling these functions.

fix

Initialize `fig, ax = plt.subplots()` before calling plotting functions like `hist(ax, ...)`.

affects: 0.1.x

gotchaThe plotting functions (`hist`, `density_plot`) expect a PySpark DataFrame containing *only* the numerical column(s) you wish to plot. Do not pass the entire DataFrame if it contains non-numerical columns or multiple columns.

fix

Use `df.select('column_name')` to pass only the relevant numerical column to the plotting function, e.g., `hist(ax, df.select('my_numeric_column'))`.

affects: 0.1.x

gotchaIf plots are not displaying in non-interactive environments (e.g., scripts, remote servers), it might be due to Matplotlib's backend. `plt.show()` is crucial, but an interactive backend might also be needed.

fix

Always call `plt.show()` after generating plots. For non-interactive environments, consider saving the figure: `plt.savefig('my_plot.png')` or configuring a non-interactive backend like `agg` (though this won't show plots interactively). For Jupyter/IPython, ensure `%matplotlib inline` or `%matplotlib notebook` is set.

affects: 0.1.x

deprecatedThe library is in maintenance mode with its last release in 2019 (0.1.8). While functional, it might not receive updates for newer PySpark versions or advanced features, and bug fixes are unlikely.

fix

Be aware of potential compatibility issues with very recent PySpark versions. For critical new projects, consider alternative, more actively maintained PySpark visualization libraries if available or roll your own using PySpark's RDD/DataFrame operations combined with Matplotlib/Seaborn.

affects: < 0.1.8

Errors

Common errors & fixes

UserWarning: Matplotlib is currently using agg, which is a non-interactive backend, so figures will not be shown.

Matplotlib is configured to use a non-interactive backend (like 'agg') which doesn't display plots to the screen automatically, and `plt.show()` was likely not called.

fix

Ensure you call `plt.show()` after generating your plot. If running in an interactive environment (like Jupyter), use `%matplotlib inline` or `%matplotlib notebook`. Otherwise, save the figure with `plt.savefig('plot.png')`.

TypeError: cannot convert 'StringType' object to float

You passed a PySpark DataFrame column with a non-numeric data type (e.g., StringType) to a plotting function that expects numerical data.

fix

Ensure the column you are plotting is of a numeric type (IntegerType, FloatType, DoubleType). Cast the column if necessary: `df.withColumn('numeric_col', df['string_col'].cast('double')).select('numeric_col')`.

AttributeError: 'DataFrame' object has no attribute 'plot'

You are trying to call a `.plot()` method directly on a PySpark DataFrame, which is not supported by PySpark itself.

fix

PySpark-dist-explore functions are standalone. Instead of `df.plot()`, use `hist(ax, df.select('column_name'))` or `density_plot(ax, df.select('column_name'))` after initializing `fig, ax = plt.subplots()`.

NameError: name 'spark' is not defined

The `SparkSession` object named `spark` was not created or is out of scope before being used.

fix

Make sure to initialize your SparkSession: `from pyspark.sql import SparkSession; spark = SparkSession.builder.appName("MyApp").getOrCreate()`.

Upgrade

Version history

0.1.8latest on PyPI · released Aug 20, 2019

Audit

Dependencies

pysparkrequiredCore functionality relies on PySpark DataFrames.

matplotlibrequiredUsed for generating histogram and density plots.

pandasrequiredUsed for `describe_pd` functionality and internal data handling for plotting.

Agent activity

7 hits · last 30 days

node

Resources

packagehttps://pypi.org/project/pyspark-dist-explore/ ↗