Install & Compatibility
Where this runs
tested against v0.7.0 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.95 runs
installs and imports cleanly · install 0.0s · import 0.000s · 45.5MB
glibcpy 3.10–3.95 runs
installs and imports cleanly · install 4.0s · import 0.000s · 46MB
45MB installed
● package 45MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
SparkClient
✓ from azure.synapse.spark import SparkClient
This is the primary client for interacting with Synapse Spark.
DefaultAzureCredential
✓ from azure.identity import DefaultAzureCredential
Recommended credential for authenticating with Azure services.
This quickstart demonstrates how to authenticate with Azure Synapse and list Spark batch jobs within a specified Spark pool. You'll need to set `SYNAPSE_WORKSPACE_NAME` and `SYNAPSE_SPARK_POOL_NAME` environment variables (or hardcode them) and ensure your Azure credentials are configured (e.g., via `az login`).
import os
from azure.identity import DefaultAzureCredential
from azure.synapse.spark import SparkClient
# Replace with your Synapse workspace name and a Spark pool name
synapse_workspace_name = os.environ.get("SYNAPSE_WORKSPACE_NAME", "your_synapse_workspace_name")
spark_pool_name = os.environ.get("SYNAPSE_SPARK_POOL_NAME", "your_spark_pool_name")
endpoint = f"https://{synapse_workspace_name}.dev.azuresynapse.net"
if synapse_workspace_name == "your_synapse_workspace_name" or spark_pool_name == "your_spark_pool_name":
print("Please set SYNAPSE_WORKSPACE_NAME and SYNAPSE_SPARK_POOL_NAME environment variables ",
"or replace the placeholder values in the code.")
else:
try:
# Obtain a credential from Azure Identity. Ensure you're logged in via Azure CLI/VS Code, or env vars are set.
credential = DefaultAzureCredential()
# Create a SparkClient
spark_client = SparkClient(endpoint=endpoint, credential=credential)
# List Spark batch jobs in a specific pool (example operation)
print(f"Listing Spark batch jobs for Spark Pool '{spark_pool_name}' in workspace '{synapse_workspace_name}'...")
batch_jobs_collection = spark_client.spark_batch.get_spark_batch_jobs(
workspace_name=synapse_workspace_name,
spark_pool_name=spark_pool_name
)
print(f"Found {len(batch_jobs_collection.value)} Spark batch jobs:")
for job in batch_jobs_collection.value:
print(f" - Job ID: {job.id}, Name: {job.name}, State: {job.state}")
except Exception as e:
print(f"Error interacting with Azure Synapse Spark: {e}")
print("Ensure your Azure credentials are set up and you have permissions to the Synapse workspace and Spark pool.")
Debug
Known issues
breakingAs a library in an early preview version (0.x.x), `azure-synapse-spark` may introduce breaking changes in minor version updates. Always review release notes when upgrading.fixConsult the official Azure SDK for Python changelog for `azure-synapse-spark` before upgrading to new preview versions.
affects: 0.x.x
gotchaMany client methods, like `get_spark_batch_jobs`, require explicit `workspace_name` and `spark_pool_name` arguments, even if the workspace name is implicitly part of the client's `endpoint` URL. Ensure these parameters are consistently provided.fixAlways check the method signatures and examples. Pass `workspace_name` and `spark_pool_name` explicitly where required by the method, typically derived from environment variables or configuration.
affects: 0.7.0 and likely later
gotchaAuthentication with Azure services using `DefaultAzureCredential` relies on specific environment variables (e.g., `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`, `AZURE_TENANT_ID`), being logged in via Azure CLI (`az login`), or other Azure Identity sources. Without proper setup, authentication will fail.fixEnsure `azure-identity` is installed (`pip install azure-identity`) and that your environment is correctly configured for `DefaultAzureCredential` to find credentials (e.g., `az login` or appropriate environment variables).
affects: All versions using `azure-identity`
breakingThe `azure-synapse-spark` library depends on `six`, but it appears not to be installed automatically, leading to a `ModuleNotFoundError` during import. This indicates a potential issue with the library's declared dependencies.fixEnsure the `six` library is installed in your environment by running `pip install six`.
affects: All versions
gotchaThe `azure-synapse-spark` library depends on `six`. If `six` is not installed, you will encounter a `ModuleNotFoundError`. This can happen if transitive dependencies are not correctly resolved by your package manager or if using minimal Python environments.fixEnsure `six` is installed in your environment by including it in your `requirements.txt` or explicitly running `pip install six`.
affects: All versions
Upgrade
Version history
0.7.0latest on PyPI · released Oct 5, 2021
Audit
Dependencies
azure-corerequiredProvides shared primitives, HTTP pipeline, and error handling for Azure SDK client libraries.
msrestrequiredUsed for serialization/deserialization of REST API models.
azure-identityoptionalRequired for authentication using DefaultAzureCredential and other credential types.