Install & Compatibility
Where this runs
tested against v1.1.0 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.95 runs
installs and imports cleanly · install 0.0s · import 0.000s · 18MB
glibcpy 3.10–3.95 runs
installs and imports cleanly · install 1.6s · import 0.000s · 18MB
16MB installed
● package 16MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
HnswIndex
✓ from pyspark_hnsw import HnswIndex
✗ from pyspark_hnsw import HnswIndex
This quickstart demonstrates how to initialize a Spark session, create a sample DataFrame with vector data, build an HNSW index using `HnswIndex`, and then perform a nearest neighbor search. Remember to configure Spark appropriately for your environment (e.g., local, YARN, Kubernetes) and ensure the `index_path` is accessible by all Spark workers.
from pyspark import SparkConf, SparkContext
from pyspark.sql import SparkSession
from pyspark_hnsw import HnswIndex
import numpy as np
import os
# Configure Spark (local mode for example)
conf = SparkConf().setAppName("HnswQuickstart").setMaster("local[*]")
sc = SparkContext(conf=conf)
spark = SparkSession(sc)
# Create some sample data with 128-dimensional vectors
data = [(i, [float(x) for x in np.random.rand(128)]) for i in range(1000)]
df = spark.createDataFrame(data, ["id", "vector"])
# Define a path for the index (local or distributed filesystem like HDFS/S3)
# Ensure this path is writable and accessible by Spark workers
index_path = "hnsw_index_test_dir"
# Clean up previous index if it exists for repeatable runs
if os.path.exists(index_path):
import shutil
shutil.rmtree(index_path)
# Build the HNSW index
hnsw_index = HnswIndex(spark, "id", "vector", index_path) \
.setM(16) \
.setEf(100) \
.setNumPartitions(10) \
.setDistanceType("cosine") \
.build(df)
# Define a query vector
query_vector = [float(x) for x in np.random.rand(128)]
num_neighbors = 5
# Find nearest neighbors
result = hnsw_index.findNearestNeighbors(query_vector, num_neighbors)
result.show()
# Stop the Spark session
spark.stop()
Debug
Known issues
gotchaThere is a discrepancy between the latest PyPI version (1.1.0) and the latest GitHub release (1.2.1). Ensure you are aware of which version you are installing and its associated features/fixes.fixCheck both PyPI and GitHub for the most up-to-date information. If installing from source, pin to a specific commit or tag.
affects: 1.1.0 (PyPI) vs 1.2.1 (GitHub)
gotchaBuilding and querying HNSW indices, especially with high dimensionality or large datasets, can be memory and CPU intensive. Adjust Spark executor memory (`spark.executor.memory`), number of partitions (`setNumPartitions`), and HNSW parameters (`setM`, `setEf`) carefully.fixMonitor Spark UI for memory and CPU usage. Start with smaller datasets and conservative HNSW parameters, then scale up. Increase Spark memory allocations if OutOfMemory errors occur.
affects: All versions
breakingVersion 1.2.0 (not yet on PyPI as of 1.1.0) includes a repackaging of classes (`Repackage classes to avoid JPMS issues`). While primarily affecting Java Module System users, this change might alter internal class paths or dependencies that could indirectly impact complex PySpark setups or users relying on specific internal JAR references.fixIf upgrading to 1.2.x or later from source, test thoroughly, especially if you have custom Spark configurations or Java dependencies. Verify that no existing Spark job configurations rely on old internal class names.
affects: >=1.2.0 (GitHub releases)
gotchaThe `index_path` used to build the index must be accessible and writable by all Spark executors, and it should typically point to a distributed file system like HDFS, S3, or similar. Using a local path will store the index only on the driver or the first executor, which is not suitable for distributed use.fixAlways use a distributed file system path (e.g., `s3a://your-bucket/index_name`, `hdfs://namenode/user/index_name`) for `index_path` when running on a Spark cluster. For local testing, ensure the path exists and has write permissions.
affects: All versions
gotchaEnsure your vectors are in a format compatible with `pyspark-hnsw`, typically `ArrayType(FloatType)`. Mismatched data types can lead to errors during index building or querying.fixVerify the schema of your input DataFrame, especially the vector column. Convert vectors to `list[float]` or `np.array` before creating the DataFrame if needed.
affects: All versions
Upgrade
Version history
1.1.0latest on PyPI · released Dec 30, 2022
Audit
Dependencies
pysparkrequiredRequired for distributed processing and Spark integration.
hnswlibrequiredThe underlying C++ HNSW library with Python bindings.