Registry / data / sparse-dot-topn

sparse-dot-topn

library1.2.0pypypi✓ verified 84d ago

sparse-dot-topn is a Python package designed to accelerate sparse matrix multiplication followed by the selection of the top-N results. It significantly reduces memory footprint and improves performance for operations common in tasks like large-scale string comparison and entity matching. Developed by ING Wholesale Banking Advanced Analytics, it is currently at version 1.2.0 and receives regular updates with a focus on performance and Python version compatibility.

data ai-ml

pip install sparse-dot-topn

github ↗package ↗

INSTALL

IMPORT

SIG · SPARSE-DOT-TOPN

sparse-dot-topn

datapythonv1.2.0

Install

7.5s avg

Import

969ms

Disk

227MB

Pass rate

5/ 10

Env Coverage5 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v1.2.0 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.920 runs

build_error

glibc

py 3.10–3.920 runs

installs and imports cleanly · install 7.5s · import 0.969s · 223MB

227MB installed

● package 227MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

sp_matmul_topn

✓ from sparse_dot_topn import sp_matmul_topn

✗ from sparse_dot_topn import awesome_cossim_topn

`awesome_cossim_topn` was deprecated in v1.0.0 and will be removed in future versions; use `sp_matmul_topn` instead.

sp_matmul

✓ from sparse_dot_topn import sp_matmul

zip_sp_matmul_topn

✓ from sparse_dot_topn import zip_sp_matmul_topn

Introduced in v1.1.0 for chunked matrix multiplication.

This example demonstrates how to perform a sparse matrix multiplication with top-N result selection using `sp_matmul_topn`. It creates two random CSR sparse matrices and computes their product, keeping only the top 10 values for each row in the result matrix. Ensure `scipy` and `numpy` are installed.

import scipy.sparse as sparse from sparse_dot_topn import sp_matmul_topn import numpy as np # Create two sample sparse matrices (CSR format is recommended for performance) A = sparse.random(1000, 100, density=0.1, format="csr", random_state=42) B = sparse.random(100, 2000, density=0.1, format="csr", random_state=42) # Compute C = A * B and retain the top 10 values per row in C # sp_matmul_topn also supports `n_threads` for parallel execution C = sp_matmul_topn(A, B, top_n=10, n_threads=None, threshold=0.0) print(f"Shape of A: {A.shape}") print(f"Shape of B: {B.shape}") print(f"Shape of result C: {C.shape}") print(f"Number of non-zero elements in C: {C.nnz}") # print(C)

Debug

Known issues

breakingPython 3.8 support was dropped in version 1.2.0. Ensure you are using Python 3.9 or higher.

fix

Upgrade your Python environment to 3.9 or newer.

affects: >=1.2.0

breakingMajor API changes in v1.0.0: `awesome_cossim_topn` was deprecated (use `sp_matmul_topn`), `ntop` parameter renamed to `topn`, `lower_bound` to `threshold`, and `use_threads`/`n_jobs` combined into `n_threads`.

fix

Migrate your code to use the new `sp_matmul_topn` function and updated parameter names. Refer to the migration guide in the GitHub README.

affects: >=1.0.0

gotchaOpenMP initialization issues, especially on MacOS, can lead to crashes or unexpected behavior due to double initialization or incorrect `rpath` settings.

fix

If experiencing OpenMP errors, try installing from source with `pip install sparse_dot_topn --no-binary sparse_dot_topn` to ensure architecture-specific optimizations, or run functions without explicitly specifying the `n_threads` argument. Check the `INSTALLATION.md` for platform-specific troubleshooting.

affects: All

gotchaSetting `threshold=None` (the default since v1.0.0) enables pre-computation of non-zero entries, which can reduce memory at a mild performance penalty (~10%). If performance is critical and memory is not an issue, consider setting an explicit `threshold` (e.g., `0.0`).

fix

Profile your application with `threshold=None` versus a specific `threshold` value to determine the optimal setting for your use case.

affects: >=1.0.0

Errors

Common errors & fixes

cannot import name 'awesome_cossim_topn' from 'sparse_dot_topn'

The function `awesome_cossim_topn` was deprecated and replaced by `sp_matmul_topn` in version 1.0.0. It has likely been removed in newer versions.

fix

Update your import statement and function calls to use `sp_matmul_topn` instead. For example: `from sparse_dot_topn import sp_matmul_topn` and `C = sp_matmul_topn(A, B, top_n=N)`.

Could not build wheels for sparse-dot-topn, which is required to install pyproject.toml-based projects

This error typically indicates missing build dependencies (like a C++ compiler, Cython, or compatible NumPy/SciPy versions) required to compile the underlying C++ extension if a pre-built wheel is not available for your system/Python version.

fix

Ensure you have a C++17 compatible compiler installed (e.g., `build-essential` on Debian/Ubuntu, Xcode Command Line Tools on MacOS, Visual Studio Build Tools on Windows). Also, explicitly install `cython` and `numpy` before `sparse-dot-topn`: `pip install cython numpy scipy sparse-dot-topn`. If issues persist, try specifying a known working NumPy version or installing with `--no-binary sparse_dot_topn`.

MemoryError: Unable to allocate ...

Performing sparse matrix multiplication on extremely large matrices can still exhaust available memory, even with `sparse-dot-topn`'s optimizations.

fix

For very large matrices (e.g., O(10M+) rows), consider breaking them into smaller chunks and using the `zip_sp_matmul_topn` function to process them distributedly. Adjusting `top_n` and `threshold` can also help reduce memory footprint.

Upgrade

Version history

1.2.0latest on PyPI · released Dec 16, 2025

Audit

Dependencies

numpyrequiredCore dependency for numerical operations and array handling.

psutilrequiredUsed for system and process utilities, possibly for resource monitoring or thread management.

scipyrequiredProvides sparse matrix data structures (e.g., csr_matrix) which are fundamental to sparse-dot-topn's functionality.

Agent activity

14 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/ing-bank/sparse_dot_topn ↗

packagehttps://pypi.org/project/sparse-dot-topn/ ↗