csv-diff

library1.2pypypi✓ verified 24d ago

csv-diff is a Python CLI tool and library for efficiently comparing the semantic contents of two CSV, TSV, or JSON files. It identifies added, removed, and changed rows based on a specified key, ignoring cosmetic differences like row and column ordering. The library is actively maintained with regular updates addressing features and bug fixes, with its current version being 1.2.

data devops

pip install csv-diff

github ↗package ↗

INSTALL

IMPORT

SIG · CSV-DIFF

csv-diff

datapythonv1.2

Install

1.8s avg

Import

13ms

Disk

17MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v1.2 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.95 runs

installs and imports cleanly · install 0.0s · import 0.014s · 18.9MB

glibc

py 3.10–3.95 runs

installs and imports cleanly · install 1.8s · import 0.012s · 19MB

17MB installed

● package 17MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

load_csv

✓ from csv_diff import load_csv, compare

Used to load CSV/TSV/JSON data from a file-like object or path into a standardized dictionary format for comparison.

compare

✓ from csv_diff import load_csv, compare

The main function for performing the diff between two loaded data structures.

This quickstart demonstrates how to use `csv-diff` programmatically to compare two in-memory CSV datasets. It loads the data using `load_csv`, specifying 'id' as the unique key, and then uses `compare` to generate a dictionary detailing added, removed, and changed rows and columns.

import io from csv_diff import load_csv, compare # Simulate two CSV files as in-memory strings csv1_data = """id,name,age 1,Alice,30 2,Bob,24 3,Charlie,35""" csv2_data = """id,name,age 1,Alice,31 3,Charlie,35 4,David,28""" # Load the CSV data, specifying the key column csv1 = load_csv(io.StringIO(csv1_data), key="id") csv2 = load_csv(io.StringIO(csv2_data), key="id") # Compare the two CSVs diff = compare(csv1, csv2) # Print the detected differences print(f"Added rows: {diff.get('added')}") print(f"Removed rows: {diff.get('removed')}") print(f"Changed rows: {diff.get('changed')}") print(f"Columns added: {diff.get('columns_added')}") print(f"Columns removed: {diff.get('columns_removed')}")

csv-diff --version

Debug

Known issues

gotchaProviding a `key_columns` (or `key` parameter for `load_csv`) is crucial. Without a specified unique key, `csv-diff` cannot accurately identify matching rows for comparison, potentially leading to incorrect diff results or errors. This was explicitly addressed with a fix in version 1.0.

fix

Always provide the `--key` option when using the CLI, or the `key` parameter to `load_csv` when using the library.

affects: < 1.0

breakingPrior to version 1.0, column names containing a `.` character could cause bugs. This was fixed in 1.0, potentially changing diff results for users who encountered this issue in earlier versions.

fix

Ensure your `csv-diff` version is 1.0 or newer. If you relied on the old (buggy) behavior, review your diffs after updating.

affects: < 1.0

gotchaWhile `csv-diff` automatically detects CSV/TSV/JSON formats, it's safer to explicitly specify the input format using `--format=csv`, `--format=tsv`, or `--format=json` for the CLI, or appropriate handling when loading data programmatically, especially for ambiguous files.

fix

Use the `--format` CLI option or ensure your data loading logic explicitly handles the expected input format to avoid unexpected parsing behavior.

affects: All versions

deprecatedThe format of the human-readable CLI output changed significantly in versions 0.3.1 and 0.2 (e.g., order of output, inclusion of more detail). Scripts that parsed the CLI's plain text output in older versions might break or produce incorrect results with newer versions.

fix

For programmatic consumption of diff results, use the dictionary output from the `compare` function rather than parsing the CLI's human-readable text output. If you must parse CLI output, ensure your parsing logic is robust to format changes or use `--json` output.

affects: < 0.3.1

Errors

Common errors & fixes

KeyError: 'YourColumnName'

The specified key column ('YourColumnName') is not found in one or both of the input CSV files. This can be due to typos, case sensitivity, leading/trailing whitespace in the header, or Byte Order Mark (BOM) characters affecting how the header is read.

fix

Ensure the key column name provided to `csv-diff` (using `--key` in CLI or `index_columns` in the library) exactly matches the header in both CSV files. Inspect column names for invisible characters or unexpected encoding (e.g., check for UTF-8 BOM if encountering issues, by opening the file with `encoding='utf-8-sig'` if using the library).

ERROR: CSV parse error on line X

The CSV file is malformed around the specified line number, often due to inconsistent delimiters, unescaped quotes within fields, or varying numbers of columns per row, preventing `csv-diff` from parsing the file correctly.

fix

Examine the CSV file, particularly near line 'X', for structural inconsistencies. Use a text editor to identify and correct issues like misplaced commas, unclosed quotes, or unexpected characters. If using a non-comma delimiter, ensure it's consistently applied.

FileNotFoundError: [Errno 2] No such file or directory: 'path/to/your/file.csv'

The input CSV file specified in the command or library call does not exist at the given path, or `csv-diff` does not have the necessary permissions to access it.

fix

Verify that the file path is correct, including the file name and extension. Ensure the file exists at the specified location and that `csv-diff` has read permissions for the file and its parent directories. Use absolute paths to avoid issues with relative working directories.

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x... in position ...: invalid start byte

The CSV file is encoded in a format other than UTF-8 (e.g., Windows-1252 or ISO-8859-1), but the Python environment or `csv-diff`'s underlying file reading is attempting to decode it as UTF-8.

fix

Pre-convert the CSV file to UTF-8 encoding using a text editor or another script. If using `csv-diff` as a Python library, open the files with the correct `encoding` parameter (e.g., `open(filename, 'r', encoding='latin1')` or `encoding='windows-1252'`) before passing the file-like objects to `csvdiff.diff_files` or `csvdiff.diff_records`.

Upgrade

Version history

1.2latest on PyPI · released Sep 6, 2024

Audit

Dependencies

clickrequiredPowers the command-line interface (CLI) interactions.

python-benedictrequiredLikely used for flexible, key-based data access across different data structures (e.g., nested JSON objects).

tabulaterequiredUsed for formatting and rendering tabular data in human-readable output.

richrequiredProvides rich text and beautiful formatting in the terminal output.

Agent activity

16 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/simonw/csv-diff ↗

packagehttps://pypi.org/project/csv-diff/ ↗