Registry / data / pyspark-data-sources

pyspark-data-sources

library0.1.11pypypiunverified

Custom Spark data sources for reading and writing data in Apache Spark, using the Python Data Source API. Current version: 0.1.11. Release cadence: irregular, low activity.

data database

pip install pyspark-data-sources

package ↗

INSTALL

IMPORT

SIG · PYSPARK-DATA-SOURC

pyspark-data-sources

datapythonv0.1.11

Install

7.4s avg

Import

—

Disk

187MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.1.11 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.95 runs

installs and imports cleanly · install 0.0s · import 0.000s · 241MB

glibc

py 3.10–3.95 runs

installs and imports cleanly · install 7.4s · import 0.000s · 218MB

187MB installed

● package 187MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

DataSource

✓ from pyspark_datasources import DataSource

✗ from pyspark_datasources import DataSource

Create a Spark session and use a custom data source via format(). Replace 'custom_source' with the actual source name.

from pyspark.sql import SparkSession from pyspark_datasources import DataSource spark = SparkSession.builder.appName('example').getOrCreate() df = spark.read.format('custom_source').load('path/to/data') df.show()

Debug

Known issues

gotchaThe library is in early development (0.1.x). API may change without notice.

fix

Pin to a specific version and test upgrades carefully.

affects: >=0.1.0,<1.0.0

deprecatedPython 3.13 is not supported (requires <3.13). Users on 3.13 must downgrade or use alternatives.

fix

Use Python 3.9, 3.10, 3.11, or 3.12.

affects: >=0.1.0

gotchaThe package name uses hyphens (pyspark-data-sources) but the import uses underscores (pyspark_datasources). This mismatch can cause ImportError.

fix

Use `import pyspark_datasources` (note: singular 'datasources' in import).

affects: all

gotchaThe library requires Java and a Spark installation. Not a pure Python solution.

fix

Ensure Java 8/11 and Spark are installed and SPARK_HOME is set.

affects: all

breakingThe DataSource class API may change; custom data source implementations depend on internal interfaces.

fix

Check the source code for the current DataSource abstract methods before implementing.

affects: >=0.1.0

Upgrade

Version history

0.1.11latest on PyPI · released Jan 26, 2026

Audit

Dependencies

pysparkrequiredRequired to use Spark DataFrame API and DataSource API

Agent activity

12 hits · last 30 days

node

Resources

packagehttps://pypi.org/project/pyspark-data-sources/ ↗