Install & Compatibility
Where this runs
tested against v1.1.3 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.95 runs
installs and imports cleanly · install 0.0s · import 0.502s · 509.2MB
glibcpy 3.10–3.95 runs
installs and imports cleanly · install 31.3s · import 0.468s · 510MB
506MB installed
● package 506MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
SparkSession
✓ from pyspark.sql import SparkSession
OpenAIPrompt
✓ from synapse.ml.services.openai import OpenAIPrompt
LightGBMClassifier
✓ from synapse.ml.lightgbm import LightGBMClassifier
TextFeaturizer
✓ from synapse.ml.featurize.text import TextFeaturizer
AnalyzeText
✓ from synapse.ml.services.language import AnalyzeText
find_secret
✓ from synapse.ml.core.platform import find_secret
✗ import mmlspark.core.platform.find_secret
SynapseML was formerly MMLSpark; namespaces have changed.
This quickstart demonstrates how to set up a Spark session with SynapseML, create a DataFrame with sample prompts, and use the `OpenAIPrompt` transformer to interact with an Azure OpenAI service for chat completions. It includes placeholders for API key and deployment name, which should be configured via environment variables or a secure secret management system in production.
import os
from pyspark.sql import SparkSession
from synapse.ml.services.openai import OpenAIPrompt
# Initialize Spark Session with SynapseML package
spark = SparkSession.builder \
.appName("SynapseML_OpenAI_Quickstart") \
.config("spark.jars.packages", "com.microsoft.azure:synapseml_2.12:1.1.3") \
.config("spark.jars.repositories", "https://mmlspark.azureedge.net/maven") \
.getOrCreate()
# Prepare sample data
df = spark.createDataFrame([
("Explain quantum computing in simple terms.",),
("What are the benefits of exercise?",),
("Describe the water cycle.",)
]).toDF("prompt")
# Configure Azure OpenAI service details
# Replace with your actual deployment name and API key.
# For local testing, set OPENAI_API_KEY environment variable.
# In a Synapse/Databricks environment, use secret management (e.g., find_secret)
openai_api_key = os.environ.get('OPENAI_API_KEY', 'YOUR_OPENAI_API_KEY')
openai_deployment_name = os.environ.get('OPENAI_DEPLOYMENT_NAME', 'gpt-4.1')
if openai_api_key == 'YOUR_OPENAI_API_KEY':
print("WARNING: Please set OPENAI_API_KEY environment variable or replace 'YOUR_OPENAI_API_KEY' with your actual key.")
print("Skipping OpenAI interaction due to missing API key.")
else:
# Configure OpenAIPrompt for chat completions
prompt_completion = (
OpenAIPrompt()
.setSubscriptionKey(openai_api_key) # Use subscriptionKey for Azure OpenAI API Key
.setDeploymentName(openai_deployment_name)
.setApiType("chat_completions")
.setPromptCol("prompt")
.setUsageCol("usage")
.setOutputCol("completions")
)
# Transform and display results
result_df = prompt_completion.transform(df.repartition(1)).select("prompt", "completions.choices.message.content", "usage")
result_df.show(truncate=False)
# Stop Spark session
spark.stop()
Debug
Known issues
breakingSynapseML was formerly known as MMLSpark. Major package and namespace changes occurred during this renaming, requiring updates to import statements (e.g., `mmlspark.foo` became `synapse.ml.foo`).fixUpdate all import statements from `mmlspark.xyz` to `synapse.ml.xyz`.
affects: <=0.18 (MMLSpark) to >=1.0.0 (SynapseML)
gotchaSynapseML has specific Apache Spark and Python version requirements. For example, SynapseML v1.1.3 typically requires Spark 3.4+ and Python 3.8+. Using incompatible versions can lead to installation failures or runtime errors.fixAlways consult the official SynapseML documentation or GitHub README for the exact Spark and Python version compatibility matrix for your chosen SynapseML version.
affects: All versions
gotchaLightGBM training via SynapseML can be unstable on Spark clusters with dynamic resource allocation (e.g., autoscaling). Changes in executors during data processing can cause the training to hang or fail, as LightGBM's native distributed mode does not gracefully handle such networking changes.fixFor critical LightGBM training jobs, consider disabling cluster autoscaling, setting a fixed (smaller) number of executors, or increasing Spark's executor heartbeat interval and network timeouts. Splitting data into smaller batches with `numBatches` can also improve reliability at the cost of increased total processing time.
affects: All versions using LightGBM
gotchaWhen training LightGBM with large datasets, especially in 'bulk execution mode' (default), users may encounter Java `OutOfMemoryError` (OOM) exceptions. This can occur even with seemingly sufficient executor memory, indicating potential issues with Spark's `spark.driver.maxResultSize` or the way data is transferred.fixIncrease `spark.driver.maxResultSize` and executor memory, and consider adjusting the `numBatches` parameter in LightGBM, although this might impact performance. Monitor Spark UI metrics to understand actual memory utilization.
affects: All versions using LightGBM with large datasets
deprecatedOn Microsoft Fabric, while SynapseML is preinstalled, installing or changing SynapseML versions using the `%%configure -f` magic command in notebooks is not officially supported and does not guarantee service-level agreement or future compatibility with official releases.fixFor production workloads on Fabric, rely on the preinstalled SynapseML version or consult official Microsoft Fabric documentation for supported methods of library management. Use `%%configure` at your own risk for experimental purposes.
affects: All versions on Microsoft Fabric
Upgrade
Version history
1.1.3latest on PyPI · released Apr 7, 2026
Audit
Dependencies
pysparkrequiredSynapseML is built on Apache Spark and requires pyspark for Python environments.