pyddq

library5.0.0pypypiunverified

Python API for Drunken Data Quality (DDQ), a data quality validation library for Apache Spark DataFrames. Current version 5.0.0, supports Spark 2.2.1 and Python 3. Last release in 2017; project appears stable with no recent updates.

data database

pip install pyddq

github ↗package ↗

INSTALL

IMPORT

SIG · PYDDQ

pyddq

datapythonv5.0.0

harness data pending

Install & Compatibility

Where this runs

No compatibility data collected yet for this library.

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

Constraint

✓ from pyddq import Constraint

✗ import pyddq; pyddq.Constraint

Direct import from pyddq is the standard method.

Check

✓ from pyddq.runner import Check

✗ from pyddq import Check

Check class is inside the runner module, not top-level.

Runner

✓ from pyddq.runner import Runner

✗ import pyddq; pyddq.Runner

Runner must be imported from the runner submodule.

Quickstart: create a Spark DataFrame, define a constraint, run a check with Runner.

from pyddq import Constraint from pyddq.runner import Runner, Check from pyspark.sql import SparkSession spark = SparkSession.builder.appName('ddq_example').getOrCreate() df = spark.createDataFrame([('Alice', 34), ('Bob', 45), ('Charlie', 28)], ['name', 'age']) constraint = Constraint(name='age_not_null', condition="age IS NOT NULL") check = Check(df, [constraint]) runner = Runner() results = runner.run(check) for r in results: print(r.result, r.constraint_name)

Debug

Known issues

gotchapyddq requires a running SparkSession. Without initializing Spark, imports will fail or hang.

fix

Ensure SparkSession is created before using pyddq functions.

affects: all

breakingThe API uses Spark 2.0+ SparkSession; older SparkContext-based code will not work.

fix

Use SparkSession instead of SparkContext when creating DataFrame.

affects: 4.0.0+

deprecatedThe library is no longer actively maintained. No updates since 2017; may not work with newer Spark versions.

fix

Consider alternatives like great_expectations or Deequ for modern Spark setups.

affects: 5.0.0

Upgrade

Version history

5.0.0latest on PyPI · released Feb 28, 2020

Audit

Dependencies

No dependency data recorded yet.

Agent activity

10 hits · last 30 days

node

Resources

githubhttps://github.com/FRosner/drunken-data-quality#python-api-1 ↗

packagehttps://pypi.org/project/pyddq/ ↗