Install & Compatibility
Where this runs
tested against v0.11.0 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
py 3.13
✕ no_wheel
✕ no_wheel
53MB installed
● package 53MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
KoheesioSparkSession
✓ from koheesio.spark import KoheesioSparkSession
Use this to get or create the managed Spark session.
Step
✓ from koheesio.steps import Step
Base class for defining custom data processing steps.
Pipeline
✓ from koheesio.pipelines import Pipeline
Base class for orchestrating multiple steps.
CsvReader
✓ from koheesio.steps.readers import CsvReader
Example of a common built-in reader step.
This example demonstrates how to initialize a Koheesio Spark session, define a custom `Step` using Pydantic for parameters, create a dummy DataFrame, and execute the step. Ensure a Spark environment is available or PySpark is installed.
import os
from koheesio.spark import KoheesioSparkSession
from koheesio.steps import Step
from pyspark.sql import DataFrame
from pyspark.sql.functions import lit
# 1. Define your Spark Koheesio Session
spark_session_name = "koheesio-example"
spark = KoheesioSparkSession.get_or_create(spark_session_name=spark_session_name)
# 2. Define a simple Step
class AddColumnStep(Step):
"""Adds a new column to the DataFrame."""
value: str
def execute(self, df: DataFrame) -> DataFrame:
self.log.info(f"Adding column 'new_column' with value '{self.value}'")
return df.withColumn("new_column", lit(self.value))
# 3. Create a dummy DataFrame
df_input = spark.createDataFrame([(1, "a"), (2, "b")], ["id", "col1"])
df_input.show()
# 4. Run your step
df_output = AddColumnStep(value="koheesio-rocks").execute(df_input)
df_output.show()
# 5. Stop the Spark session (optional in many environments)
# spark.stop()
Debug
Known issues
breakingKoheesio has strict `pyspark` version requirements (`>=3.3.0,<3.6.0`). Using Spark versions outside this range, especially attempting 'Spark Connect' with unsupported versions, will lead to errors.fixEnsure your environment's `pyspark` version is within the specified range. For example, `pip install pyspark==3.5.0`.
affects: All versions 0.10.x
gotchaSpecific functionalities like Snowflake, Databricks, or Delta Lake integration require installing Koheesio with their respective optional dependencies (e.g., `pip install 'koheesio[snowflake]'`).fixInstall Koheesio with the required extras, for example: `pip install 'koheesio[snowflake]'` or `pip install 'koheesio[all]'`.
affects: All versions
gotchaKoheesio requires Python versions `>=3.9` and `<3.13`. Ensure your Python environment meets these requirements to avoid compatibility issues.fixUpgrade or downgrade your Python interpreter to a compatible version (e.g., Python 3.9, 3.10, 3.11, or 3.12).
affects: All versions 0.10.x
breakingKoheesio now explicitly requires Pydantic V2 (`>=2.0.0`). If you are migrating from an older Koheesio version that used Pydantic V1, you may need to update your custom `Step` definitions due to Pydantic's breaking API changes between major versions.fixReview Pydantic V2 migration guides. Update `Step` parameter definitions and ensure correct Pydantic V2 syntax is used.
affects: Likely from 0.x.x to 0.10.x if Pydantic V1 was previously used.
Upgrade
Version history
0.11.0latest on PyPI · released Jul 27, 2026
Audit
Dependencies
pysparkrequiredCore dependency for all Spark-based operations. Requires version >=3.3.0,<3.6.0.
pydanticrequiredUsed for defining step parameters and validation. Requires version >=2.0.0.
snowflake-connector-pythonoptionalRequired for Snowflake integration features.
delta-sparkoptionalRequired for Delta Lake integration features.
databricks-sql-connectoroptionalRequired for Databricks SQL connectivity.