Registry / data / pyspark-pandas

pyspark-pandas

library0.0.7pypypiunverified

PySpark-Pandas (version 0.0.7) is an early project that aimed to provide tools and algorithms for pandas DataFrames distributed on PySpark. Its last release was in 2016, and the project has since been abandoned. The PyPI description itself advises users to consider alternatives like SparklingPandas, and the official Apache Spark project now includes its own 'Pandas API on Spark' (formerly Koalas) for similar functionality, which is the recommended modern solution.

data aws gcp azure

pip install pyspark-pandas

github ↗package ↗

INSTALL

IMPORT

SIG · PYSPARK-PANDAS

pyspark-pandas

datapythonv0.0.7

Install

8.7s avg

Import

—

Disk

165MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.0.7 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.920 runs

installs and imports cleanly · install 0.0s · import 0.000s · 166MB

glibc

py 3.10–3.920 runs

installs and imports cleanly · install 8.7s · import 0.000s · 159MB

165MB installed

● package 165MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

DataFrame

✓ from pyspark_pandas import DataFrame

✗ from pyspark_pandas import DataFrame

The `pyspark-pandas` library (version 0.0.7) is largely unmaintained and does not offer a readily available, functional quickstart. The provided code demonstrates a quickstart using the official 'Pandas API on Spark' (`pyspark.pandas`), which is the recommended alternative for distributed pandas-like operations in a modern PySpark environment.

# The 'pyspark-pandas' (0.0.7) library is abandoned and lacks a functional, self-contained quickstart example # compatible with modern Spark/Python environments. # Its primary functionality would have involved wrapping Spark RDDs or DataFrames with a pandas-like interface. # # For modern 'Pandas API on Spark' functionality, use pyspark.pandas: from pyspark.sql import SparkSession import pyspark.pandas as ps import pandas as pd # Create a SparkSession spark = SparkSession.builder.appName("PandasOnSparkQuickstart").getOrCreate() # Create a pandas-on-Spark DataFrame from a pandas DataFrame pd_df = pd.DataFrame({"col1": [1, 2, 3], "col2": [4, 5, 6]}) ps_df = ps.from_pandas(pd_df) print("Pandas-on-Spark DataFrame:") print(ps_df) print(f"Type: {type(ps_df)}") # Perform a simple operation ps_df['col3'] = ps_df['col1'] + ps_df['col2'] print("\nDataFrame after operation:") print(ps_df) # Convert back to a pandas DataFrame (collects data to driver) pandas_result = ps_df.to_pandas() print("\nResult as pandas DataFrame:") print(pandas_result) spark.stop()

Debug

Known issues

breakingThe `pyspark-pandas` (0.0.7) library is effectively abandoned since its last commit in 2016. It is highly unlikely to be compatible with modern versions of PySpark or Python, and its functionality has been superseded by the official 'Pandas API on Spark' (formerly Koalas) integrated directly into PySpark as `pyspark.pandas`.

fix

Do not use `pyspark-pandas`. Instead, use `pyspark.pandas` which is included with PySpark (PySpark 3.2+). Install `pyspark` and import `pyspark.pandas as ps`.

affects: <=0.0.7

deprecatedThe PyPI description for `pyspark-pandas` explicitly advises users to 'Please consider the SparklingPandas project before this one'. This indicates the project was considered superseded even at the time of its last update.

fix

For distributed pandas-like functionality, use `pyspark.pandas` (the official Pandas API on Spark) which is actively maintained and integrated into Apache Spark.

affects: All (0.0.7)

gotchaConfusing `pyspark-pandas` (the abandoned PyPI package) with `pyspark.pandas` (the official Pandas API on Spark) is a common mistake. They are distinct projects with different import paths and maintenance statuses.

fix

Always import `pyspark.pandas as ps` for the official Pandas API on Spark. The `pyspark-pandas` package should not be used.

affects: All

Upgrade

Version history

0.0.7latest on PyPI · released Oct 14, 2014

Audit

Dependencies

pysparkrequiredCore dependency for distributed DataFrame processing.

pandasrequiredCore dependency for pandas DataFrame compatibility.

Agent activity

50 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/adgaudio/pyspark_pandas ↗

packagehttps://pypi.org/project/pyspark-pandas/ ↗