Registry / data / datasketches

datasketches

library5.2.0pypypi✓ verified 23d ago

The Apache DataSketches Library for Python provides a collection of high-performance, stochastic streaming algorithms (sketches) for approximate queries on massive datasets. These sketches offer mathematically proven error bounds and are designed for problems like count distinct, quantiles, most-frequent items, joins, matrix computations, and graph analysis. The current version is 5.2.0, with a regular release cadence as part of the Apache DataSketches project.

data serialization ai-ml

pip install datasketches

package ↗homepage ↗

INSTALL

IMPORT

SIG · DATASKETCHES

datasketches

datapythonv5.2.0

Install

3.8s avg

Import

257ms

Disk

91MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v5.2.0 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.95 runs

installs and imports cleanly · install 0.0s · import 0.238s · 92.3MB

glibc

py 3.10–3.95 runs

installs and imports cleanly · install 3.8s · import 0.276s · 87MB

91MB installed

● package 91MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

datasketches

✓ import datasketches

The primary module for accessing all sketch classes and utilities.

kll_ints_sketch

✓ from datasketches import kll_ints_sketch

✗ import kll_ints_sketch

Specific sketch classes are exposed directly under the top-level 'datasketches' module.

This quickstart demonstrates how to create and use a KLL (Kaplan-Meier-Louis) integer sketch to estimate quantiles and ranks from a stream of data. The KLL sketch is an efficient way to get approximate quantile information with strong error guarantees.

import datasketches # Create a KLL sketch for integers kll_sketch = datasketches.kll_ints_sketch() # Update the sketch with data for i in range(1000): kll_sketch.update(i) # Get quantiles median = kll_sketch.get_quantile(0.5) rank_99 = kll_sketch.get_rank(99) print(f"Estimated median: {median}") print(f"Estimated rank for value 99: {rank_99}") print(f"Estimated number of distinct items: {kll_sketch.get_num_retained()}")

Debug

Known issues

breakingVersion 5.0.0 introduced significant API changes, including the migration from pybind11 to nanobind for C++ bindings. This also led to more 'pythonic' API patterns, such as using `.copy()` instead of C++-style copy constructors and `str()` taking no arguments.

fix

Review your code for C++-style copy constructors and `str()` calls. Adapt to Pythonic `obj.copy()` methods and argument-less `str()` for object representation. Ensure `nanobind` is installed instead of `pybind11`.

affects: 5.0.0 and later

gotchaPython's native integer types do not support unsigned integers or numeric values with fewer than 64 bits directly. This can result in sketches created within Python being non-identical to those created in Java or C++ versions of DataSketches for certain data types or configurations.

fix

Be aware of potential discrepancies when comparing sketch results or binary serializations across different language implementations. Loading sketches serialized from other languages into Python will work as expected, but the creation process may differ.

affects: All versions

deprecatedThe 'Quantiles Sketch' (e.g., `quantiles_ints_sketch`) is considered an inferior algorithm compared to the KLL sketch and is officially deprecated in favor of KLL and REQ sketches.

fix

Migrate existing code using 'Quantiles Sketch' to `kll_ints_sketch`, `kll_floats_sketch`, or `req_ints_sketch`, `req_floats_sketch` for better accuracy and performance.

affects: 3.4.0 and later

gotchaWhen integrating `datasketches` with Apache Spark, especially with Spark 3.5+ and Java 17+, specific Spark configurations and Java options (`--add-modules=jdk.incubator.foreign`) are required for the driver and executors. Incorrect configuration can lead to runtime errors.

fix

Consult the `datasketches-spark` documentation for detailed Spark configuration settings, including `spark.driver.userClassPathFirst`, `spark.executor.userClassPathFirst`, and the necessary Java options for module exports.

affects: All versions (specific to Spark 3.5+ / Java 17+)

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'datasketches'

The 'datasketches' library is not installed in the Python environment, or the Python interpreter cannot locate the installed package.

fix

Ensure the library is correctly installed using pip: `pip install datasketches`

AttributeError: module 'datasketches' has no attribute 'KLLSketch'

Users often attempt to import or instantiate sketch classes directly as top-level attributes of the `datasketches` module (e.g., `datasketches.KLLSketch`). However, the specific sketch implementations are typically exposed as factory functions or specific sub-module attributes, often named by their data type (e.g., `kll_doubles_sketch`).

fix

Access the specific sketch type using its correct factory function, such as `datasketches.kll_doubles_sketch()` or `datasketches.kll_ints_sketch()`.

TypeError: update(): incompatible function arguments. The following argument types are supported: ...

This error occurs when an incorrect data type is passed to a sketch's `update()` method. For example, trying to update a `kll_ints_sketch` with a floating-point number or a `kll_floats_sketch` with a string, when the sketch expects a different numeric type or a comparable item type.

fix

Ensure the data type being passed to the `update()` method matches the expected type of the specific sketch instance (e.g., `kll_ints_sketch` expects integers, `kll_doubles_sketch` expects doubles/floats, `kll_items_sketch` expects comparable Python objects).

Upgrade

Version history

5.2.0latest on PyPI · released Mar 1, 2025

Audit

Dependencies

numpyrequiredRequired for numerical operations and array handling.

nanobindrequiredUsed for Python-C++ bindings. Replaced pybind11 in version 5.0.0 and later.

Agent activity

24 hits · last 30 days

node

OpenAI (training)

Resources

packagehttps://pypi.org/project/datasketches/ ↗

homepagehttp://datasketches.apache.org ↗