repartipy is a Python library designed to assist with managing PySpark DataFrame partition sizes. It provides a function to repartition a DataFrame based on a target partition size in megabytes, aiming to optimize storage and processing efficiency. As of version 0.1.8, it's a relatively stable and focused utility, with updates likely driven by PySpark compatibility or feature requests rather than a fixed cadence.
pip install repartipyVerified import paths — ran on the pinned version, not inferred.
This quickstart demonstrates how to initialize a SparkSession, create a sample DataFrame, and then use `repartition_by_size` to optimize its partitions. The example aims for 10MB partitions, showing the initial and resulting partition counts. Remember that repartitioning creates a new DataFrame and requires an action (like writing data or collecting) to trigger actual computation.
Always use `new_df = repartition_by_size(old_df, ...)`
Profile your Spark jobs and understand your data distribution. Only repartition when necessary and consider the impact of the `target_partition_size_mb` on overall job performance.
Use the target size as a guideline. Monitor actual partition sizes after repartitioning using Spark UI or metrics to confirm the desired effect. Adjust the target size based on observed results and data characteristics.