Registry / data / detect-delimiter

detect-delimiter

library0.1.1pypypi✓ verified 87d ago

The `detect-delimiter` Python library, currently at version 0.1.1 and last released in July 2018, provides a simple function to automatically identify the delimiter used in various ad-hoc file formats like CSV or TSV. It primarily operates by counting character frequencies within an input string. The library exposes a single `detect()` function, making it straightforward to use for basic delimiter detection needs. Its release cadence appears to be sporadic or ceased, indicating a stable but not actively developed state.

data serialization

pip install detect-delimiter

package ↗homepage ↗

INSTALL

IMPORT

SIG · DETECT-DELIMITER

detect-delimiter

datapythonv0.1.1

Install

1.5s avg

Import

14ms

Disk

16MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.1.1 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.920 runs

installs and imports cleanly · install 0.0s · import 0.014s · 17.8MB

glibc

py 3.10–3.920 runs

installs and imports cleanly · install 1.5s · import 0.014s · 18MB

16MB installed

● package 16MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

detect

✓ from detect_delimiter import detect

The `detect()` function is the primary entry point. It takes the text as a string and can optionally take `whitelist` (a list of characters to prioritize), `blacklist` (characters to ignore), and `default` (a value to return if no delimiter is found) parameters.

from detect_delimiter import detect # Example 1: Basic comma-separated data text1 = "apple,banana,cherry" delimiter1 = detect(text1) print(f"Delimiter for '{text1}': '{delimiter1}'") # Example 2: Tab-separated data text2 = "name\tage\tcity" delimiter2 = detect(text2) print(f"Delimiter for '{text2}': '{delimiter2}'") # Example 3: Semicolon-separated with a custom whitelist text3 = "one;two;three" delimiter3 = detect(text3, whitelist=[';', ',', '|']) print(f"Delimiter for '{text3}': '{delimiter3}'") # Example 4: No common delimiter found, returning a default value text4 = "hello world" delimiter4 = detect(text4, default='NA') print(f"Delimiter for '{text4}': '{delimiter4}'") # Example 5: Period as delimiter, which is blacklisted by default text5 = "file.name.txt" delimiter5 = detect(text5) print(f"Delimiter for '{text5}': '{delimiter5}'") # Expected: None (as '.' is blacklisted by default)

Debug

Known issues

gotchaThe `detect()` function, by default, will not check alphanumeric characters or the period/full stop character ('.') as delimiters. If your files use these as actual delimiters (e.g., a custom file format with `.` as a separator), they will be ignored.

fix

Use the `blacklist` parameter to explicitly include characters for consideration, for example: `detect(text, blacklist=[])` to remove all default blacklisted characters, or `detect(text, whitelist=['.'])` to force checking for a period.

affects: 0.1.1

gotchaThe library does not handle CSV quoting rules (e.g., delimiters within double quotes `"field, with, commas"`). It primarily relies on simple character frequency counting. This can lead to incorrect delimiter detection in malformed CSVs or when data fields contain characters that are also common delimiters.

fix

For robust CSV parsing that respects quoting and escaping, consider using Python's built-in `csv.Sniffer` or a more advanced library like `CleverCSV`.

affects: 0.1.1

gotchaThe `detect-delimiter` library is designed for single-character delimiters and does not support multi-character delimiters (e.g., `##`, `|||`).

fix

For files with multi-character delimiters, manual parsing or a custom solution will be required, as this library is not suitable.

affects: 0.1.1

Errors

Common errors & fixes

detect_delimiter doesn't consider quotation and escaping, and hence can easily miss the correct separator if it occurs more often because it's escaped.

The `detect()` function performs a basic character frequency count without a full understanding of CSV format rules, such as quoted fields. If a common delimiter like a comma appears frequently within quoted text, it may be incorrectly identified as the primary delimiter.

fix

For files known to follow CSV standards with quoting, use Python's `csv.Sniffer`. If a simplified approach is still desired and the problem persists, use the `whitelist` parameter with only the *true* expected delimiters, e.g., `detect(text, whitelist=[';'])`.

None returned as delimiter when an expected delimiter is clearly present in the text.

The expected delimiter might not be in the default `whitelist` `[',', ';', ':', '|', '\t']` or it might be a character that is blacklisted by default (e.g., alphanumeric, period).

fix

Explicitly provide a `whitelist` parameter with the characters you expect to be delimiters (e.g., `detect(text, whitelist=['|', '~'])`) or adjust the `blacklist` if characters are being incorrectly ignored (e.g., `detect(text, blacklist=[])`).

Incorrect delimiter detected (e.g., returns ',' but file is ';'-separated).

The library's frequency-based detection can be misled if a character that is *not* the true delimiter appears more often in the sample text. This is common if data fields contain frequent commas in a semicolon-delimited file.

fix

Narrow down the possibilities using the `whitelist` parameter, for example, `detect(text, whitelist=[';', '|'])`. For highly ambiguous cases, manual inspection or a more context-aware parsing library might be needed.

Upgrade

Version history

0.1.1latest on PyPI · released Jul 29, 2018

Audit

Dependencies

No dependency data recorded yet.

Agent activity

17 hits · last 30 days

node

OpenAI (training)

Resources

packagehttps://pypi.org/project/detect-delimiter/ ↗

homepagehttps://git.nzoss.org.nz/tim-mcnamara/detect-delimiter ↗