dlt is an open-source python-first scalable data loading library that does not require any backend to run. It simplifies data ingestion from various sources to analytical destinations, handling schema evolution, retries, and state management. dlt releases new versions frequently, approximately monthly, often including breaking changes.
pip install dltVerified import paths — ran on the pinned version, not inferred.
This quickstart demonstrates how to create a simple dlt pipeline to extract GitHub issues and load them into a DuckDB destination. Ensure you have `duckdb` installed (`pip install dlt[duckdb]`) and set a `GITHUB_TOKEN` environment variable if you hit rate limits.
Upgrade Pydantic to version 2 (`pip install 'pydantic>=2'`). Review your code for any Pydantic v1 specific patterns or configurations that may need updating.
The `dlt pipeline show` command no longer functions. Users should adapt to using the new Marimo-based interactive widgets (introduced in 1.22.2) or use `dlt inspect` for command-line pipeline details.
If your code reads incremental metrics from dlt's trace object, you will need to update it to access the new table-based structure and location of these metrics.
If you used `data_type: freeze` and expected to change properties like `nullable`, `precision`, or `scale` on existing columns, these will now be treated as breaking changes. Review your schema definitions and data type contracts.
While `dlt` still supports the full `sources.<section>.<name>.<key>` path, the new compact lookup may cause unexpected configuration resolution or overrides if not carefully managed. Review your configuration files and ensure keys are resolved as intended.
Instead of `github_issues_source(token=...)(repo_owner, repo_name)`, you should initialize the source and then use it with `dlt.run` or select specific resources. For example, `pipeline = dlt.pipeline(pipeline_name='my_pipeline', destination='duckdb', dataset_name='github_data'); pipeline.run(github_issues_source(token=os.environ.get('GITHUB_TOKEN', '')).with_resources('issues'), write_disposition='merge')` or similar patterns depending on your exact use case.Update source invocation. If your code uses a pattern like `my_source_factory(config_args)(execution_arg1, execution_arg2)`, you must now pass all execution-related arguments directly to the source factory as keyword arguments (e.g., `my_source_factory(execution_arg1='value1', execution_arg2='value2', **config_args)`). Review the specific source's documentation for its updated invocation signature.
In Databricks, if you intend to use the open-source `dlt` library, you need to use an init script to remove or rename the conflicting Databricks DLT module. A common fix involves running a shell script like `removedbdlt.sh` that unsets `PYTHONPATH` or moves the conflicting module. For interactive notebooks, you might also mock the `dlt` class to pass syntax checks. If you are *not* in Databricks, simply `pip install dlt`.
Delta Live Tables functionality in Databricks is meant to be run through the DLT pipeline interface, not directly in interactive notebooks or Spark Connect clusters. To resolve this, ensure your code is deployed and executed as a DLT pipeline within the Databricks environment, where the necessary runtime and dependencies are managed automatically.
Ensure all required configuration fields are defined in your `.dlt/config.toml` or `.dlt/secrets.toml` files, or as environment variables. If using profile-specific configurations (e.g., `dev.config.toml`), create a `.dlt/.workspace` file in your project's root to enable profile loading.
Pin the version of `deltalake` to a compatible version with your `dlt` installation. For `dlt` versions that had issues with `deltalake` 1.0, pinning `deltalake` to `0.25.0` or an earlier compatible version often resolves the issue. Always check the `dlt` documentation or release notes for recommended dependency versions.
Ensure that the `initial_value` for your incremental cursor has the exact same data type as the field in your source data that `dlt` is tracking for incremental loading. If type conversion is needed, apply a transformation (e.g., using `add_map`) to convert the field's type *before* incremental tracking.
No dependency data recorded yet.