Deltalake is an open-source Python library providing native Delta Lake bindings based on the `delta-rs` Rust library, offering efficient and robust interaction with Delta Lake tables without requiring Apache Spark or JVM dependencies. It includes seamless integration with data manipulation libraries like Pandas, Polars, and PyArrow. The library is actively developed, with its current version being 1.5.0, and receives frequent updates to enhance performance and features.
Install & Compatibility
Where this runs
tested against v1.6.0 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.925 runs
installs and imports cleanly · install 0.0s · import 0.126s · 457.4MB
glibcpy 3.10–3.925 runs
installs and imports cleanly · install 11.3s · import 0.110s · 425MB
453MB installed
● package 453MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
DeltaTable
✓ from deltalake import DeltaTable
write_deltalake
✓ from deltalake import write_deltalake
This quickstart demonstrates how to create, append data to, and read different versions (time travel) of a Delta Lake table using `deltalake` and Pandas. It first creates an initial table, then appends new records, and finally shows how to access a previous state of the table by specifying a version.
import pandas as pd
from deltalake import write_deltalake, DeltaTable
import os
# Define a Delta Lake table path
table_path = "./tmp_delta_table"
# Ensure the directory exists or is cleaned up for a fresh start
if os.path.exists(table_path):
import shutil
shutil.rmtree(table_path)
# 1. Create a Pandas DataFrame
df = pd.DataFrame({"id": [1, 2], "value": ["A", "B"]})
# 2. Write the DataFrame to a Delta Lake table
write_deltalake(table_path, df)
print(f"Initial Delta table created at: {table_path}")
# 3. Load the Delta table
dt = DeltaTable(table_path)
print(f"Current table version: {dt.version()}")
print("Current table data:")
print(dt.to_pandas().to_markdown(index=False))
# 4. Append new data to the table
new_df = pd.DataFrame({"id": [3, 4], "value": ["C", "D"]})
write_deltalake(table_path, new_df, mode="append")
print("\nData appended. New table version:")
dt_updated = DeltaTable(table_path)
print(f"Current table version: {dt_updated.version()}")
print("Updated table data:")
print(dt_updated.to_pandas().to_markdown(index=False))
# 5. Read an older version of the table (Time Travel)
dt_v0 = DeltaTable(table_path, version=0)
print("\nData from version 0 (time travel):")
print(dt_v0.to_pandas().to_markdown(index=False))
# Clean up temporary files (optional)
# shutil.rmtree(table_path)
deltalake --version
Debug
Known issues
breakingIn `deltalake` v1.5.0, the `get_add_actions` method now returns an `ArrowTable` instead of an `ArrowRecordBatch`. Code relying on the specific `ArrowRecordBatch` type or its API will break.fixUpdate code to expect and handle `pyarrow.Table` objects from `get_add_actions`. Adjust API calls accordingly (e.g., `to_pandas()` might still work on `ArrowTable`).
affects: >=1.5.0
breakingCheckpoint schema changes between `deltalake` versions, notably around `0.25.5` and `1.0.2`, can lead to `DeltaError: Failed to parse parquet: Arrow: Incompatible type` when attempting to read or create checkpoints from older tables, especially if `nullable` properties for fields like `path`, `size`, `modificationTime` changed from `True` to `False`.fixThere is currently no direct migration path provided in the library for this specific checkpoint schema change. Users might need to recreate tables or use the version of `deltalake` that created the original checkpoints for continued compatibility.
affects: >=1.0.2 when interacting with tables created by <=0.25.5
gotchaThe `deltalake` Python library is a native implementation distinct from `delta-spark`. While both interact with Delta Lake, `deltalake` does not require Apache Spark or a JVM. Ensure you are using the correct library for your ecosystem, as `delta-spark` imports (e.g., `from delta.tables import DeltaTable`) are not compatible with `deltalake`.fixIf integrating with Spark, use `delta-spark`. For Python-native operations without Spark, use `deltalake`. Avoid mixing imports or expectations from the two libraries.
affects: All versions
gotchaConcurrent write operations (e.g., multiple processes appending or updating a table simultaneously) can lead to `ConcurrentAppendException`, `ConcurrentDeleteReadException`, or `ConcurrentModificationException` due to optimistic concurrency control. While Delta Lake guarantees ACID properties, conflicts require handling.fixImplement retry logic with exponential backoff and jitter for write operations. Consider partitioning tables strategically to minimize file-level conflicts.
affects: All versions
gotchaOperations like `DeltaTable.delete()` or `write_deltalake(mode="overwrite")` only mark files for deletion in the Delta transaction log. The physical files are not immediately removed from storage. This can lead to increased storage costs if not managed.fixRegularly run `DeltaTable.vacuum()` on your tables to physically remove stale data files. Be aware that `vacuum()` can break time travel beyond its retention period (default 7 days).
affects: All versions
gotchaSome functionalities, especially around `MERGE` operations, might require configuring disk spilling for large datasets to avoid out-of-memory errors.fixUtilize disk spilling configuration options available for `MERGE` operations in `deltalake` v1.5.0+ when dealing with large datasets.
affects: >=1.5.0 for disk spilling feature, older versions prone to OOM for large merges
gotchaWhen using `dt.to_pandas().to_markdown()` to display table data, `pandas` requires the optional `tabulate` library to be installed. Without it, an `ImportError: Missing optional dependency 'tabulate'` will occur.fixEnsure the `tabulate` library is installed in your environment (e.g., `pip install tabulate`) if you intend to use `pandas.DataFrame.to_markdown()`.
affects: All versions
gotchaWhen converting Delta tables to pandas DataFrames and then attempting to use display methods like `to_markdown()`, `to_latex()`, or `to_html()`, you might encounter `ImportError` for packages like `tabulate`, `jinja2`, or `xhtml2pdf`. These are optional dependencies for pandas display functionalities, not direct dependencies of `deltalake`.fixEnsure all necessary optional dependencies for pandas display methods are installed in your environment (e.g., `pip install tabulate`, `pip install jinja2`). Check pandas documentation for specific requirements of each display function.
affects: All versions
Audit
Dependencies
pandasrequiredRequired for DataFrame integration and `write_deltalake` functionality.
pyarrowrequiredUnderpins data handling, especially for DataFrame conversions and Arrow-native operations.
tabulateoptionalNeeded for pretty-printing DataFrames in some quickstart examples.