Registry / data / airbyte-source-declarative-manifest

airbyte-source-declarative-manifest

JSON →
library6.5.2pypiunverified

This library serves as the base source implementation for low-code Airbyte connectors, allowing users to define data sources using a YAML-based declarative manifest. It interprets these manifests to run data synchronization processes without requiring extensive Python coding. The current version is 6.5.2, and it follows the Airbyte platform's rapid release cadence, with major platform updates typically monthly, influencing the underlying CDK and manifest capabilities.

pip install airbyte-source-declarative-manifest
INSTALL
IMPORT
SIG · AIRBYTE-SOURCE-DEC
A
airbyte-source-declarative-manifest
dataenv6.5.2
Install
18.8s avg
Import
3801ms
Disk
165MB
Pass rate
4/ 10
Env Coverage4 / 10
glibc
3.93.13
musl
3.93.13
Install & Compatibility
Where this runs
tested against v6.5.2 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
musl
glibc
py 3.10
✕ build_error
✓ 21.95s
py 3.11
✕ build_error
✓ 20.45s
py 3.12
✕ build_error
✕ build_error
py 3.13
✕ build_error
✕ build_error
py 3.9
✓ —
✓ 13.85s
165MB installed
● package 165MB
Code
Verified usage

Verified import paths — ran on the pinned version, not inferred.

DeclarativeSource
from airbyte_cdk.sources.declarative.declarative_source import DeclarativeSource
from airbyte_source_declarative_manifest import DeclarativeSource
The `airbyte-source-declarative-manifest` package provides the underlying engine; users interact with it primarily through the `airbyte-cdk`'s `DeclarativeSource` class to define and run connectors.

This quickstart demonstrates how to programmatically use the `DeclarativeSource` from `airbyte-cdk` to load and interact with a declarative manifest. This is how the `airbyte-source-declarative-manifest` library is implicitly used. It defines a minimal YAML manifest, saves it, and then runs a 'check' operation against it. In a real Airbyte deployment, the platform would handle loading the manifest and invoking the source.

import json import os from pathlib import Path from airbyte_cdk.sources.declarative.declarative_source import DeclarativeSource # Create a dummy manifest file for demonstration manifest_content = """ version: "0.3.0" resolver: type: "inline" schemas: streams: - name: "example_stream" retriever: type: "simple" requester: url_base: "https://jsonplaceholder.typicode.com" path: "/todos/1" http_method: "GET" authenticator: type: "no_auth" record_selector: field_path: [] stream_mapper: type: "simple" primary_key: [] transformations: [] """ # Save the manifest to a temporary file manifest_path = Path("temp_manifest.yaml") manifest_path.write_text(manifest_content) # Define the config for the source (can be empty if not needed by manifest) config = {} # Initialize the declarative source source = DeclarativeSource(str(manifest_path)) # Run a check operation (or other CDK operations like discover, read) # This is how Airbyte internally invokes the source. status = source.check(None, config) print(f"Source check status: {status.status.value}") # Clean up the temporary manifest file manifest_path.unlink()
Debug
Known issues
breakingDeclarative OAuth 2.0 was introduced in Airbyte 1.5.0. Existing declarative connectors using older OAuth configurations may require manifest updates to align with the new, standardized OAuth 2.0 flow, particularly within the connector builder context.
fix
Consult Airbyte's official documentation for Declarative OAuth 2.0 to update your connector manifest's authentication section. Ensure `airbyte-cdk` and `airbyte-source-declarative-manifest` are updated to compatible versions.
affects: Airbyte platform versions < 1.5.0 when upgrading to 1.5.0+
gotchaThe schema for declarative manifests evolves. A manifest created for an older `airbyte-cdk` or `airbyte-source-declarative-manifest` version might become incompatible with newer versions due to changes in component definitions, available features, or required configurations.
fix
Regularly check Airbyte's release notes and declarative connector documentation when upgrading `airbyte-cdk` or `airbyte-source-declarative-manifest`. Validate your manifest against the latest schema, especially for breaking changes in components or data types.
affects: All versions across major updates.
gotchaDebugging declarative sources can be more challenging than programmatic ones. Errors often manifest as cryptic YAML parsing issues, misconfigurations of declarative components, or unexpected behavior due to implicit logic, rather than explicit Python stack traces.
fix
Thoroughly validate your YAML manifest using a linter. Utilize Airbyte's connector builder for initial development and testing. Increase logging verbosity and examine Airbyte job logs carefully for clues related to manifest parsing or component execution.
affects: All versions
Errors
Common errors & fixes
yaml.parser.ParserError: while parsing a block mapping
The YAML manifest file has incorrect syntax, often due to indentation errors, missing colons, or improper key-value structure.
fix
Use a YAML linter (e.g., `yamllint`) to identify syntax errors. Pay close attention to indentation, which is crucial in YAML. Ensure all keys are followed by a colon and a space.
airbyte_cdk.sources.declarative.exceptions.ComponentNotFoundException: Component 'MyCustomComponent' not found in registry.
The declarative manifest references a component (e.g., a retriever, authenticator, or transformation) that is either misspelled or not available in the installed version of `airbyte-source-declarative-manifest` or `airbyte-cdk`.
fix
Double-check the component name against Airbyte's official documentation for declarative components. Ensure your `airbyte-cdk` and `airbyte-source-declarative-manifest` packages are up to date and compatible with the components you are trying to use.
Failed to authenticate with OAuth 2.0 client credentials.
The OAuth 2.0 configuration in the declarative manifest is incorrect, possibly due to outdated fields or misconfigured parameters after the introduction of Declarative OAuth 2.0.
fix
Review the Airbyte documentation for Declarative OAuth 2.0 for the correct manifest structure and required parameters. Verify your client ID, client secret, and refresh token (if applicable) are correctly configured in the manifest or environment variables.
Upgrade
Version history
6.5.2latest on PyPI · released Nov 8, 2024
Audit
Dependencies
airbyte-cdkrequiredThis library is a specific implementation used by the Airbyte CDK to run declarative connectors, requiring the CDK's core functionalities.
Agent activity
55 hits · last 30 days
node
48
OpenAI (training)
1
Resources
airbyte-source-declarative-manifest — pip install airbyte-source-declarative-manifest · libregistry