Lilcom is a Python library that provides lossy compression for sequence data stored in NumPy arrays. It efficiently compresses floating-point or 16-bit integer NumPy arrays into byte strings, typically used in machine learning applications for storing training data and models. The current version is 1.8.2, and it has a fairly active release cadence, with updates addressing compatibility and functionality.
pip install lilcomVerified import paths — ran on the pinned version, not inferred.
This quickstart demonstrates how to compress a NumPy array using `lilcom.compress()` and then decompress it using `lilcom.decompress()`. Note that `lilcom` is a lossy compression utility, so the decompressed array will not be exactly identical to the original. The `tick_power` argument controls the precision of the compression.
Be aware of the lossy nature and choose an appropriate `tick_power` for your application's accuracy requirements. `2**(tick_power-1)` gives the maximum error per element.
If migrating from older code, replace `bits_per_sample` with `tick_power`. The `tick_power` argument (default: -8) is the power of 2 used for the step size between discretized values.
Ensure your environment uses Python 3.6 or higher. Upgrade Python if necessary.
Implement robust error handling, retransmission, or checksums if storing or transmitting compressed data in environments prone to bit errors and where data recovery is critical.
Ensure a C++ compiler is installed and accessible in your system's PATH before attempting a source installation (e.g., `apt-get install build-essential` on Debian/Ubuntu or Xcode Command Line Tools on macOS).
Ensure your Python environment is version 3.6 or higher. Upgrade Python if necessary, for example, by using `python3` instead of `python` or updating your virtual environment.
Try downgrading your `setuptools` package before installing `lilcom`: `pip install 'setuptools<60'` or ensure you are using a more recent Python 3.x version (e.g., 3.8+).
Verify the integrity of the compressed byte string. Ensure the entire compressed data was successfully stored and retrieved. If the data was transmitted, check for transmission errors. Also, confirm that the data being decompressed was indeed compressed using `lilcom.compress()`.
Replace `bits_per_sample` with `tick_power` in your call to `lilcom.compress()`. For example, change `lilcom.compress(data, bits_per_sample=N)` to `lilcom.compress(data, tick_power=M)`.