This Python module provides an `xopen` function that works like Python's built-in `open` function but also transparently deals with compressed files. `xopen` selects the most efficient method for reading or writing a compressed file, often leveraging external tools like `pigz` for parallel processing. It supports gzip (.gz), bzip2 (.bz2), xz (.xz), and optionally Zstandard (.zst) formats. The library is actively maintained, currently at version 2.0.2, and has a consistent release cadence.
pip install xopenVerified import paths — ran on the pinned version, not inferred.
Demonstrates reading from a gzipped file (auto-detected) and writing to an xz-compressed file with a specified compression level. `xopen` behaves like the built-in `open` but handles compression transparently based on file extension or magic numbers.
Upgrade Python to 3.8 or later, or downgrade `xopen` to `<1.8.0`.
Explicitly specify the correct `encoding` parameter (e.g., `encoding='latin-1'` or `encoding=locale.getpreferredencoding(False)`) when working with non-UTF-8 text files.
Understand your performance requirements. For maximum speed on multi-core systems, leave `threads=None`. For predictable single-threaded behavior (e.g., in resource-constrained environments or when external tools are undesirable), use `threads=0`.
Ensure `xopen` is updated to at least v2.0.0. If using `compresslevel=0` with gzip, test thoroughly, especially if external gzip utilities are involved.
pip install xopen
pip install python-zstandard
import xopen # then use xopen.xopen() # or from xopen import xopen # then use xopen()
Ensure the file truly is a gzip-compressed file, verify its integrity, or remove the '.gz' extension if it is a plain text file.