Install & Compatibility
Where this runs
tested against v? · pip install
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.910 runs
build_error
glibcpy 3.10–3.910 runs
build_error
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
GridSearchCV
✓ from spark_sklearn import GridSearchCV
RandomizedSearchCV
✓ from spark_sklearn import RandomizedSearchCV
SparkContext
✓ from pyspark import SparkContext
✗ from spark_sklearn import SparkContext
SparkContext comes from PySpark, not spark-sklearn directly.
This quickstart demonstrates how to use spark-sklearn's GridSearchCV to perform hyperparameter tuning for a scikit-learn SVC model, distributing the computation across a Spark cluster (or locally). It covers SparkContext initialization, data preparation, defining the estimator and parameter grid, fitting the model, and retrieving results.
import os
from pyspark import SparkContext
from spark_sklearn import GridSearchCV
from sklearn.svm import SVC
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
# Initialize SparkContext
# For local testing, 'local[*]' works. For a cluster, set SPARK_MASTER env var.
if os.environ.get('SPARK_MASTER') is None:
os.environ['SPARK_MASTER'] = 'local[*]'
sc = None
try:
sc = SparkContext(appName="SparkSklearnExample")
# Generate some synthetic data
X, y = make_classification(n_samples=1000, n_features=20, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Define the estimator and parameter grid
estimator = SVC(gamma='auto', random_state=42)
param_grid = {'C': [0.1, 1, 10], 'kernel': ['linear', 'rbf']}
# Use Spark-backed GridSearchCV
clf = GridSearchCV(sc, estimator, param_grid, cv=3)
clf.fit(X_train, y_train)
print("Best parameters found:", clf.best_params_)
print("Best cross-validation score:", clf.best_score_)
print("Test set accuracy:", clf.score(X_test, y_test))
except Exception as e:
print(f"An error occurred: {e}")
finally:
if sc:
sc.stop()
Debug
Known issues
breakingProject is abandoned and unmaintained. The last commit was in 2017, meaning it does not receive bug fixes, security updates, or compatibility patches for newer Python, Spark, or scikit-learn versions.fixConsider using native Spark MLlib for distributed machine learning, or alternative frameworks designed for distributed scikit-learn (e.g., dask-ml for smaller clusters/different paradigms) if compatibility issues arise.
affects: 0.3.0 and older
breakingStrict compatibility with older Spark and scikit-learn versions. spark-sklearn officially supports Spark 2.x and scikit-learn 0.18.x. Using it with newer versions will likely lead to runtime errors or unexpected behavior.fixDowngrade your Spark and scikit-learn installations to the officially supported versions or migrate to more actively maintained distributed ML solutions.
affects: All versions, when used with Spark > 2.x or scikit-learn > 0.18.x
gotchaPotential performance overhead due to data serialization/deserialization. Data is often converted between Spark RDD/DataFrame and scikit-learn's numpy arrays, which can incur significant overhead for very large datasets.fixMonitor Spark UI for serialization/deserialization times. For optimal performance with large datasets on Spark, consider using Spark MLlib, which operates natively on Spark DataFrames.
affects: All versions
gotchaLimited functionality to GridSearchCV and RandomizedSearchCV. spark-sklearn does not provide broader integration with other scikit-learn functionalities or a direct bridge to Spark's native MLlib estimators.fixUnderstand that its scope is narrow. For other distributed scikit-learn tasks, look into libraries like dask-ml, or for full Spark integration, use Spark MLlib.
affects: All versions
Upgrade
Version history
0.3.0latest on PyPI · released Jan 30, 2019
Audit
Dependencies
pysparkrequiredRequired to interact with Apache Spark. spark-sklearn officially supports Spark 2.x.
scikit-learnrequiredThe core machine learning library spark-sklearn integrates with. Officially supports scikit-learn 0.18.x.