Registry / data / indexed-gzip

indexed-gzip

JSON →
library1.10.3pypypi✓ verified 84d ago

The `indexed-gzip` project is a Python extension providing fast random access to gzip files by building an index of seek points. It acts as a drop-in replacement for Python's built-in `gzip.GzipFile` class, significantly improving performance for `seek` operations on large gzipped files. It is currently at version 1.10.3 and is actively maintained with regular releases.

pip install indexed-gzip
INSTALL
IMPORT
SIG · INDEXED-GZIP
I
indexed-gzip
datapythonv1.10.3
Install
1.7s avg
Import
122ms
Disk
19MB
Pass rate
10/ 10
Env Coverage10 / 10
glibc
3.93.13
musl
3.93.13
Install & Compatibility
Where this runs
tested against v1.10.3 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
musl
py 3.103.920 runs
installs and imports cleanly · install 0.0s · import 0.128s · 21MB
glibc
py 3.103.920 runs
installs and imports cleanly · install 1.7s · import 0.115s · 22MB
19MB installed
● package 19MB
Code
Verified usage

Verified import paths — ran on the pinned version, not inferred.

IndexedGzipFile
from indexed_gzip import IndexedGzipFile
Commonly imported as 'igzip' for brevity: `import indexed_gzip as igzip` then `igzip.IndexedGzipFile`.

This example demonstrates how to open a gzip file with `IndexedGzipFile`, perform random `seek` and `read` operations, and explicitly build a full index. It shows how `indexed-gzip` acts as a drop-in replacement for `gzip.GzipFile` for read operations, but with significantly improved performance for non-sequential access.

import indexed_gzip as igzip import os # Create a dummy gzip file for demonstration dummy_data = b"This is some sample data for a gzipped file.\nRepeat this line many times to make it bigger.\n" * 10000 with open('test_file.gz', 'wb') as f: import gzip g = gzip.GzipFile(fileobj=f, mode='wb') g.write(dummy_data) g.close() # Open the indexed gzip file try: with igzip.IndexedGzipFile('test_file.gz') as fobj: print(f"Original file size: {len(dummy_data)} bytes") # Seek to an arbitrary position fobj.seek(15000) data = fobj.read(100) print(f"Read 100 bytes from offset 15000: {data.decode(errors='ignore')[:50]}...") # Seek to another position fobj.seek(5000) data = fobj.read(50) print(f"Read 50 bytes from offset 5000: {data.decode(errors='ignore')[:50]}...") # Build a full index explicitly (optional, often done on demand) fobj.build_full_index() print(f"Index built with {fobj.tell()} bytes processed.") finally: # Clean up the dummy file if os.path.exists('test_file.gz'): os.remove('test_file.gz')
Debug
Known issues
breakingThe `IndexedGzipFile` class currently does not support writing data. It is a read-only interface. Attempting to open in write mode or call write methods will result in an error.
fix
Use Python's built-in `gzip` module or another library for writing gzipped files. Then, open the saved file with `indexed-gzip` for fast random access.
affects: All versions
gotchaThe `spacing` parameter during `IndexedGzipFile` initialization (or implicitly during index building) controls the density of seek points. A smaller `spacing` improves seek performance but increases memory usage for the index, and vice-versa. The default is 1MB.
fix
Tune the `spacing` parameter (e.g., `IndexedGzipFile(filename, spacing=65536)`) based on your application's read patterns and memory constraints. For very fine-grained seeking, a smaller value might be beneficial, while for large files with infrequent seeks, a larger value saves memory.
affects: All versions
deprecatedPrior to version 1.10.2, passing `pathlib.Path` objects directly to `IndexedGzipFile` for the filename argument might not have been fully supported, potentially leading to errors or unexpected behavior.
fix
Upgrade to `indexed-gzip` version 1.10.2 or newer. If upgrading is not possible, convert `pathlib.Path` objects to strings using `str(path_obj)` before passing them to `IndexedGzipFile`.
affects: <1.10.2
gotchaA bug in versions prior to 1.10.0 could occur when CRC validation was disabled, particularly on GZIP streams where the stream footer contained bytes matching the GZIP magic bytes `0x1f8b`.
fix
Upgrade to `indexed-gzip` version 1.10.0 or newer to avoid this specific data corruption/read error. If unable to upgrade, ensure CRC validation is enabled (`crc_check=True`, default) or be aware of potential issues with malformed (but technically valid) GZIP files.
affects: <1.10.0
Errors
Common errors & fixes
IOError: No write support for IndexedGzipFile
Attempting to open an `IndexedGzipFile` in write mode ('w', 'wb', 'a', etc.) or calling write methods on it.
fix
The `IndexedGzipFile` is strictly read-only. For writing gzipped files, use Python's built-in `gzip` module (e.g., `gzip.GzipFile('file.gz', 'wb')`) or another appropriate library. Once written, the file can be opened with `indexed-gzip` for efficient random access.
TypeError: argument of type 'Path' is not iterable
Passing a `pathlib.Path` object directly as the `filename` argument to `IndexedGzipFile` in an older version (<1.10.2) that did not explicitly support `pathlib.Path`.
fix
Upgrade `indexed-gzip` to version 1.10.2 or later. Alternatively, convert the `pathlib.Path` object to a string before passing it: `igzip.IndexedGzipFile(str(my_path_obj))`.
Extremely slow seek() operations when using `gzip.GzipFile` on large files.
The standard `gzip.GzipFile` class must decompress from the beginning of the file up to the desired seek point, making random access inefficient, especially for large files.
fix
Replace `gzip.GzipFile` with `indexed_gzip.IndexedGzipFile`. This library builds an internal index allowing for much faster random `seek` operations. `import indexed_gzip as igzip` and use `igzip.IndexedGzipFile` instead of `gzip.GzipFile`.
Upgrade
Version history
1.10.3latest on PyPI · released Dec 8, 2025
Audit
Dependencies
CythonoptionalRequired for building from source, but not a runtime dependency for pre-built wheels.
Agent activity
4 hits · last 30 days
node
4
Resources
indexed-gzip — pip install indexed-gzip · libregistry