Registry / devops / apache-airflow-microsoft-fabric-plugin

apache-airflow-microsoft-fabric-plugin

library1.0.3pypypi✓ verified 25d ago

The `apache-airflow-microsoft-fabric-plugin` provides an operator and hook to interact with Microsoft Fabric items, such as Lakehouse, Notebook, Datafactory, and Datawarehouse, directly from Apache Airflow. It enables running Spark jobs (like notebooks) within Fabric. The current version is 1.0.3, and it is a community-driven project updated as needed.

devops workflow data azure

pip install apache-airflow-microsoft-fabric-plugin

github ↗package ↗

INSTALL

IMPORT

SIG · APACHE-AIRFLOW-MIC

apache-airflow-microsoft-fabric-plugin

devopspythonv1.0.3

Install

1.5s avg

Import

—

Disk

16MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v1.0.3 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.95 runs

installs and imports cleanly · install 0.0s · import 0.000s · 17.9MB

glibc

py 3.10–3.95 runs

installs and imports cleanly · install 1.5s · import 0.000s · 18MB

16MB installed

● package 16MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

FabricRunSparkJobOperator

✓ from apache_airflow_microsoft_fabric_plugin.operators.fabric import FabricRunSparkJobOperator

✗ from apache_airflow_microsoft_fabric_plugin import FabricRunSparkJobOperator

This example DAG demonstrates how to use the `FabricRunSparkJobOperator` to execute a Microsoft Fabric Spark Notebook. Before running, configure an Airflow Connection with `Conn Id: azure_fabric_default` (or your custom ID), `Conn Type: Microsoft Fabric`, and necessary authentication details (e.g., `tenant_id`, `client_id`, `client_secret` for service principal, or `managed_identity_client_id` for Managed Identity) in its `Extra` JSON field. Replace placeholder IDs with your actual Fabric Workspace, Lakehouse (if applicable), and Notebook IDs.

from __future__ import annotations import os import pendulum from airflow.models.dag import DAG from apache_airflow_microsoft_fabric_plugin.operators.fabric import FabricRunSparkJobOperator # Configure your Airflow connection 'azure_fabric_default' with type 'Microsoft Fabric' # and extra fields like {"tenant_id": "...", "client_id": "...", "client_secret": "..."} # You can use environment variables for sensitive data in real scenarios. with DAG( dag_id="microsoft_fabric_notebook_execution_dag", schedule=None, start_date=pendulum.datetime(2023, 10, 26, tz="UTC"), catchup=False, tags=["microsoft_fabric", "notebook"], ) as dag: run_spark_job = FabricRunSparkJobOperator( task_id="run_spark_notebook_task", fabric_conn_id="azure_fabric_default", # Ensure this matches your Airflow connection ID workspace_id=os.environ.get("FABRIC_WORKSPACE_ID", "your_fabric_workspace_id"), lakehouse_id=os.environ.get("FABRIC_LAKEHOUSE_ID", "your_fabric_lakehouse_id"), # Optional, if notebook interacts with a specific lakehouse notebook_id=os.environ.get("FABRIC_NOTEBOOK_ID", "your_fabric_notebook_id"), job_parameters={ "param1": "airflow_run", "dag_run_id": "{{ dag_run.run_id }}" } # Optional: pass parameters to your notebook )

Debug

Known issues

gotchaThe Airflow Connection must be configured with 'Microsoft Fabric' as the 'Conn Type'. Selecting a generic or incorrect connection type will lead to authentication and API errors.

fix

When creating or editing the Airflow Connection for Microsoft Fabric, ensure you select 'Microsoft Fabric' from the 'Conn Type' dropdown in the Airflow UI.

affects: All versions

gotchaAuthentication details (tenant_id, client_id, client_secret for service principal, or managed_identity_client_id for Managed Identity) must be correctly specified in the 'Extra' JSON field of the Airflow Connection.

fix

Verify that your 'Extra' field JSON is valid and contains the correct credentials. For example: `{"tenant_id": "YOUR_TENANT_ID", "client_id": "YOUR_CLIENT_ID", "client_secret": "YOUR_CLIENT_SECRET"}`.

affects: All versions

gotchaThis plugin requires Apache Airflow version 2.4.0 or higher. Older Airflow versions may not support the necessary provider interfaces or features.

fix

Ensure your Airflow environment is running version 2.4.0 or newer. Upgrade Airflow if necessary.

affects: <2.4.0

gotchaThe 'pendulum' Python package is a required dependency and must be installed in the environment for the library or its Airflow provider to function correctly.

fix

Install the 'pendulum' package in your environment: `pip install pendulum`.

affects: All versions

breakingThe required Python package 'pendulum' is not found in the environment. This is a core dependency typically installed with Apache Airflow.

fix

Ensure that Apache Airflow and its relevant provider package (e.g., `apache-airflow-providers-microsoft-fabric`) are correctly installed in your Python environment. For example, `pip install apache-airflow[microsoft.fabric]`.

affects: All versions

Upgrade

Version history

1.0.3latest on PyPI · released Sep 9, 2024

Audit

Dependencies

apache-airflowrequiredRequires Apache Airflow to function, specifically version 2.4.0 or higher.

msalrequiredUsed for Microsoft Authentication Library (MSAL) integration to authenticate with Azure/Fabric.

Agent activity

64 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/ambika-garg/apache-airflow-microsoft-fabric-plugin.git ↗

packagehttps://pypi.org/project/apache-airflow-microsoft-fabric-plugin/ ↗