The `indexed-gzip` project is a Python extension providing fast random access to gzip files by building an index of seek points. It acts as a drop-in replacement for Python's built-in `gzip.GzipFile` class, significantly improving performance for `seek` operations on large gzipped files. It is currently at version 1.10.3 and is actively maintained with regular releases.
pip install indexed-gzipVerified import paths — ran on the pinned version, not inferred.
This example demonstrates how to open a gzip file with `IndexedGzipFile`, perform random `seek` and `read` operations, and explicitly build a full index. It shows how `indexed-gzip` acts as a drop-in replacement for `gzip.GzipFile` for read operations, but with significantly improved performance for non-sequential access.
Use Python's built-in `gzip` module or another library for writing gzipped files. Then, open the saved file with `indexed-gzip` for fast random access.
Tune the `spacing` parameter (e.g., `IndexedGzipFile(filename, spacing=65536)`) based on your application's read patterns and memory constraints. For very fine-grained seeking, a smaller value might be beneficial, while for large files with infrequent seeks, a larger value saves memory.
Upgrade to `indexed-gzip` version 1.10.2 or newer. If upgrading is not possible, convert `pathlib.Path` objects to strings using `str(path_obj)` before passing them to `IndexedGzipFile`.
Upgrade to `indexed-gzip` version 1.10.0 or newer to avoid this specific data corruption/read error. If unable to upgrade, ensure CRC validation is enabled (`crc_check=True`, default) or be aware of potential issues with malformed (but technically valid) GZIP files.
The `IndexedGzipFile` is strictly read-only. For writing gzipped files, use Python's built-in `gzip` module (e.g., `gzip.GzipFile('file.gz', 'wb')`) or another appropriate library. Once written, the file can be opened with `indexed-gzip` for efficient random access.Upgrade `indexed-gzip` to version 1.10.2 or later. Alternatively, convert the `pathlib.Path` object to a string before passing it: `igzip.IndexedGzipFile(str(my_path_obj))`.
Replace `gzip.GzipFile` with `indexed_gzip.IndexedGzipFile`. This library builds an internal index allowing for much faster random `seek` operations. `import indexed_gzip as igzip` and use `igzip.IndexedGzipFile` instead of `gzip.GzipFile`.