PyORC is a Python module designed for efficiently reading and writing data in the Apache ORC (Optimized Row Columnar) file format. It provides high-performance access to ORC files, commonly used in big data ecosystems like Apache Hive, Spark, and Flink. The current version is 0.11.0, and the library maintains an active development schedule with several releases per year.
pip install pyorcVerified import paths — ran on the pinned version, not inferred.
This quickstart demonstrates how to define an ORC schema, write data into an ORC file, and then read the data back using `pyorc.Writer` and `pyorc.Reader`. It includes examples for `int`, `string`, `decimal`, and `timestamp` types, ensuring timezone awareness for `datetime` objects.
If you are opening a file-like object (e.g., from `open()`, `io.BytesIO`), use `pyorc.Reader(file_like_object=f_obj)`. For a file path, use `pyorc.Reader('path/to/file.orc')`.Update imports from `from pyorc.enums import TypeKind` to `from pyorc import TypeKind`. Review `pyorc.Column` usage if relying on default `tzinfo` or other schema defaults.
Ensure all `datetime` objects passed to `pyorc.Writer` have an explicit timezone, preferably UTC. Example: `datetime.datetime(YYYY, M, D, H, M, S, tzinfo=datetime.timezone.utc)`.
Carefully define your `TypeDescription` to match the data you intend to write. Ensure `decimal` values have correct precision/scale. Validate your data against the schema before writing.
Use the `file_like_object` keyword argument for file-like objects: `reader = pyorc.Reader(file_like_object=f_obj)`.
Review your `TypeDescription` and the data being written. Ensure column order, data types, and (for decimals) precision/scale are correctly aligned. For example, if schema expects `string`, don't pass `int`.
Verify the integrity of the ORC file. Ensure it's not truncated. If writing, try different `compression` or `stripe_size` options. If reading, ensure the file is indeed an ORC file.