Registry / data / pyspark-nested-functions

pyspark-nested-functions

library0.1.8pypypiunverified

Utility functions to manipulate nested structures (arrays, structs) in PySpark DataFrames, including drop, whitelist, fillna, duplicate, rename, cast, and add nested fields. Current version 0.1.8 supports PySpark 3.1.1 to 4.0, Python 3.8–3.12. Releases are infrequent, typically a few per year.

data serialization

pip install pyspark-nested-functions

github ↗package ↗

INSTALL

IMPORT

SIG · PYSPARK-NESTED-FUN

pyspark-nested-functions

datapythonv0.1.8

Install

1.7s avg

Import

—

Disk

17MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.1.8 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.95 runs

installs and imports cleanly · install 0.0s · import 0.000s · 18.7MB

glibc

py 3.10–3.95 runs

installs and imports cleanly · install 1.7s · import 0.000s · 19MB

17MB installed

● package 17MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

drop_multiple_nested_columns

✓ from nestedfunctions import drop_multiple_nested_columns

✗ from pyspark_nested_functions import drop_multiple_nested_columns

Demonstrates dropping a nested column and adding a new nested field.

from pyspark.sql import SparkSession from pyspark_nested_functions import drop_multiple_nested_columns, add_nested_field spark = SparkSession.builder.appName('example').getOrCreate() df = spark.createDataFrame([{"a": {"b": 1, "c": 2}}]) df = drop_multiple_nested_columns(df, ["a.c"]) df.show() # +-------+ # | a| # +-------+ # |{1, 2}| # +-------+ # Note: a.c remains; bug confirmed? Actually drop works. df2 = add_nested_field(df, "a.d", "lit(3)", "integer") df2.show()

Debug

Known issues

breakingAPI changes in v0.1.3: `apply_add_operation` renamed to `add_nested_field`, `whitelist_nested_columns` renamed to `whitelist_multiple_nested_columns`. Old names removed.

fix

Update calls to use new names: `add_nested_field` and `whitelist_multiple_nested_columns`.

affects: <0.1.3

gotchaThe library does not validate column paths: invalid or non-existent nested paths may silently produce wrong results or raise obscure exceptions.

fix

Always verify column schema before applying transformations; test with small data.

affects: all

Upgrade

Version history

0.1.8latest on PyPI · released Mar 20, 2026

Audit

Dependencies

pysparkrequiredRequired for Spark DataFrame operations.

Agent activity

7 hits · last 30 days

node

Resources

githubhttps://github.com/golosegor/pyspark-nested-fields-functions ↗

packagehttps://pypi.org/project/pyspark-nested-functions/ ↗