warc3-wet

library0.2.5pypypi✓ verified 23d ago

warc3-wet is a Python library designed to work with ARC and WARC (Web ARChive) files, which are formats for storing web crawls. It is a fork of the original `warc` repository, updated for Python 3 compatibility and to handle issues with specific datasets like ClueWeb09. The current version is 0.2.5, released on July 17, 2024, indicating an active, though not rapid, release cadence for maintenance and compatibility updates.

http-networking serialization data

pip install warc3-wet

github ↗package ↗

INSTALL

IMPORT

SIG · WARC3-WET

warc3-wet

http-networkingpythonv0.2.5

Install

1.6s avg

Import

98ms

Disk

16MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.2.5 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.95 runs

installs and imports cleanly · install 0.0s · import 0.102s · 17.9MB

glibc

py 3.10–3.95 runs

installs and imports cleanly · install 1.6s · import 0.094s · 18MB

16MB installed

● package 16MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

warc

✓ import warc

✗ from warc3_wet import warc

Despite the package name `warc3-wet`, the primary module to import for functionality is `warc`, mirroring the original library's interface. Importing directly from `warc3_wet` is not the intended public API.

This quickstart demonstrates how to open and iterate through records in a WARC or WET file. It includes a minimal setup to create a dummy WARC/WET file for immediate execution and then processes it, printing the target URI and content length for each record.

import warc # Assuming 'test.warc.wet' is a valid WARC/WET file # For demonstration, we'll create a dummy file if it doesn't exist # In a real scenario, you would have an actual WARC/WET file. import os if not os.path.exists("test.warc.wet"): with open("test.warc.wet", "w") as f: f.write("WARC/1.0\r\n") f.write("WARC-Type: warcinfo\r\n") f.write("WARC-Date: 2023-01-01T12:00:00Z\r\n") f.write("WARC-Record-ID: <urn:uuid:00000000-0000-0000-0000-000000000001>\r\n") f.write("Content-Length: 0\r\n") f.write("\r\n") f.write("\r\n") f.write("WARC/1.0\r\n") f.write("WARC-Type: response\r\n") f.write("WARC-Target-URI: http://example.com/\r\n") f.write("WARC-Date: 2023-01-01T12:00:01Z\r\n") f.write("WARC-Record-ID: <urn:uuid:00000000-0000-0000-0000-000000000002>\r\n") f.write("Content-Length: 33\r\n") f.write("Content-Type: text/plain\r\n") f.write("\r\n") f.write("HTTP/1.1 200 OK\r\n") f.write("Content-Length: 9\r\n") f.write("\r\n") f.write("Hello World\r\n") with warc.open("test.warc.wet") as f: for record in f: if 'WARC-Target-URI' in record and 'Content-Length' in record: print(f"URI: {record['WARC-Target-URI']}, Length: {record['Content-Length']}") # Clean up the dummy file os.remove("test.warc.wet")

warc3-wet --version

Debug

Known issues

gotchaDespite the package name `warc3-wet`, the module to import is `warc`. Users accustomed to the PyPI package name might incorrectly attempt `import warc3_wet` or `from warc3_wet import warc`, which is not the correct public API usage.

fix

Always use `import warc` to access the library's functionality, consistent with the original `warc` library interface.

affects: 0.2.x and earlier

breakingThis library is a Python 3 port and fork of an older, 'now dead' Python 2 `warc` library. While the interface is largely unchanged, direct compatibility with Python 2 applications using the original `warc` library is not guaranteed, and migration efforts will be required for Python 2 codebases.

fix

Ensure your environment is Python 3. Review and update any code expecting Python 2-specific behaviors or dependencies when migrating from the original `warc` library.

affects: All versions (compared to original Python 2 warc library)

gotchaThe official documentation for `warc3-wet` points to `http://warc.readthedocs.org/`, which is the documentation for the *original* `warc` library. While the interface is stated to be largely unchanged, be aware that any installation instructions on that external documentation may not apply to `warc3-wet`.

fix

Always follow the installation instructions provided on the `warc3-wet` PyPI page or GitHub README (`pip install warc3-wet`). Consult the `warc.readthedocs.org` for API usage, but verify installation and package-specific details against `warc3-wet`'s own distribution.

affects: All versions

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'warc3_wet'

The `warc3-wet` library has not been installed in your Python environment or is not accessible in the current Python path.

fix

Run `pip install warc3-wet` in your terminal to install the library.

zlib.error: Error -3 while decompressing: invalid stored block lengths

This error indicates that the library encountered a corrupted or malformed gzipped WARC record that the underlying `zlib` compression library failed to decompress. This is common with imperfect web archive datasets that `warc3-wet` aims to handle.

fix

Wrap the record processing logic in a `try-except zlib.error` block to gracefully skip problematic or unreadable records. Ensure your WARC files are as intact as possible.

AttributeError: 'Record' object has no attribute 'payload'

In `warc3-wet`, the raw byte content of a WARC record's payload is not directly accessible via a `record.payload` attribute. Instead, it must be explicitly read from the record's associated reader.

fix

Use `record.reader.read()` to retrieve the byte content of the record's payload. For example, `payload_bytes = record.reader.read()`.

EOFError: Compressed file ended before the end-of-stream marker was reached

This error occurs when a WARC file (especially gzipped ones) is truncated or incomplete, causing the library to unexpectedly reach the end of the file while still expecting more data.

fix

Implement error handling by wrapping your WARC record iteration or reading loop in a `try-except EOFError` block to gracefully manage truncated files and continue processing or exit cleanly.

Upgrade

Version history

0.2.5latest on PyPI · released Jul 17, 2024

Audit

Dependencies

No dependency data recorded yet.

Agent activity

42 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/Willian-Zhang/warc3 ↗

packagehttps://pypi.org/project/warc3-wet/ ↗