pysparkling

library0.6.2pypypi✓ verified 86d ago

Pure Python implementation of the Spark RDD interface, providing a lightweight alternative to PySpark for local or small-scale distributed computing. Current version: 0.6.2. Release cadence is sporadic, with the last release in 2020.

data workflow

pip install pysparkling

github ↗package ↗

INSTALL

IMPORT

SIG · PYSPARKLING

pysparkling

datapythonv0.6.2

harness data pending

Install & Compatibility

Where this runs

No compatibility data collected yet for this library.

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

pysparkling

✓ import pysparkling

✗ import pyspark

pysparkling is a separate library, not PySpark.

Context

✓ from pysparkling import Context

✗ from pyspark import SparkContext

Use pysparkling's Context instead of SparkContext.

Create a local context and run a simple RDD sum operation.

from pysparkling import Context sc = Context() rdd = sc.parallelize([1, 2, 3, 4, 5]) print(rdd.sum()) # 15 sc.stop()

Debug

Known issues

deprecatedpysparkling is no longer actively maintained. It may not work with newer Python versions or have security updates.

fix

Consider migrating to PySpark or Dask for production use.

affects: >=0.6.2

gotchapysparkling's API is similar but not identical to PySpark. Some methods may have different signatures or missing features.

fix

Check the official documentation for exact API differences.

affects: all

gotchapysparkling does not support distributed execution across multiple machines; it runs in a single process with simulated parallelism.

fix

Use PySpark or Dask for true distributed computing.

affects: all

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'pyspark'

Trying to import pysparkling as pyspark.

fix

Install pysparkling with 'pip install pysparkling' and import as 'import pysparkling'.

AttributeError: module 'pysparkling' has no attribute 'SparkContext'

pysparkling uses Context instead of SparkContext.

fix

Use 'from pysparkling import Context' and create 'sc = Context()'.

ImportError: cannot import name 'SparkConf'

pysparkling does not support SparkConf.

fix

Use pysparkling's Context without configuration, or pass options via keyword arguments if supported.

Upgrade

Version history

0.6.2latest on PyPI · released Nov 13, 2022

Audit

Dependencies

No dependency data recorded yet.

Agent activity

38 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/svenkreiss/pysparkling ↗

packagehttps://pypi.org/project/pysparkling/ ↗