sparkMeasure is a Python API for the core Scala library, designed for performance troubleshooting of Apache Spark workloads. It simplifies the collection and analysis of Spark metrics, making it suitable for interactive analysis, testing, and production monitoring. The library focuses on easing metric collection and analysis for both developers and data engineers. Releases are frequent, typically on a quarterly to half-yearly cadence, with the current stable version being 0.27.0.
pip install sparkmeasureVerified import paths — ran on the pinned version, not inferred.
This quickstart demonstrates how to initialize SparkMeasure and use the `runandmeasure` method to collect and report performance metrics for a Spark SQL query. It sets up a local SparkSession, loads the necessary SparkMeasure JAR, and then uses `StageMetrics` to instrument a simple operation.
Update Spark configuration to use the new prefix for Kafka Producer properties (e.g., `spark.sparkmeasure.kafka.bootstrap.servers`).
Be aware of these limitations when using sparkMeasure with Spark Connect. Consider Flight Recorder mode for application-level metrics or alternative monitoring for client-specific details.
Monitor driver memory usage. For large-scale data, consider saving metrics to file or external sinks (Kafka, InfluxDB) using Flight Recorder mode, which directly writes collected metrics.
Prefer `StageMetrics` for general performance overview and troubleshooting, and switch to `TaskMetrics` only when detailed task-level analysis is required.
When analyzing job failures, be aware that reported metrics might not fully account for all resource usage leading up to the failure. Complement with Spark UI event logs for full failure context.