Install & Compatibility
Where this runs
tested against v0.0.10 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.915 runs
installs and imports cleanly · install 0.0s · import 0.000s · 57.5MB
glibcpy 3.10–3.915 runs
installs and imports cleanly · install 4.9s · import 0.000s · 59MB
62MB installed
● package 62MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
DataflowPipeline
✓ from dlt_meta.src import DataflowPipeline
✗ from dlt_meta.src import DataflowPipeline
This quickstart demonstrates how to programmatically invoke the `dlt-meta` framework within a Databricks environment (typically a notebook or job). It assumes `dlt-meta` is installed and metadata has been onboarded. The `DataflowPipeline.invoke_dlt_pipeline` method orchestrates the creation and execution of DLT pipelines based on the provided layer and environment, reading from pre-configured metadata.
# This code typically runs within a Databricks Notebook or job after metadata onboarding.
# Ensure 'dlt-meta' is installed via %pip install dlt-meta in the notebook or as a cluster library.
import dlt
from dlt_meta import DataflowPipeline
import os
# These parameters would typically be passed as job parameters in Databricks
# For local testing, you might set environment variables or hardcode.
layer = os.environ.get('DLT_META_LAYER', 'bronze').lower() # e.g., 'bronze' or 'silver'
env = os.environ.get('DLT_META_ENV', 'dev').lower() # e.g., 'dev', 'qa', 'prod'
# In a Databricks environment, 'spark' session is implicitly available.
# For local testing outside Databricks, you would need to initialize a SparkSession.
# Example placeholder for local SparkSession (not typically done in DLT-META's primary use-case):
# from pyspark.sql import SparkSession
# spark = SparkSession.builder.appName("dlt-meta-local").getOrCreate()
try:
print(f"Attempting to invoke DLT-META for layer: {layer} (env: {env}).")
# The 'spark' object is expected to be the Databricks SparkSession
DataflowPipeline.invoke_dlt_pipeline(spark=spark, layer=layer, env=env)
print(f"DLT-META successfully invoked for layer: {layer} (env: {env}).")
except ImportError:
print("ERROR: Could not import DataflowPipeline from dlt_meta. Ensure the 'dlt-meta' library is installed and available.")
raise
except Exception as e:
print(f"ERROR: An exception occurred during DLT-META pipeline invocation for layer '{layer}' in env '{env}': {e}")
raise
Upgrade
Version history
0.0.10latest on PyPI · released Sep 16, 2025
Audit
Dependencies
pythonrequiredRequired runtime environment.
databricks-clirequiredRequired for CLI interactions and deployment to Databricks workspace (v0.213 or later).
PyYAMLoptionalUsed for parsing YAML metadata files.
setuptoolsoptionalCore dependency for Python package management.
databricks-sdkoptionalUsed for interacting with Databricks APIs.