Registry / workflow / apache-airflow-providers-apache-pig

apache-airflow-providers-apache-pig

library4.8.4pypypi✓ verified 51d ago

Apache Airflow provider for Apache Pig. Version 4.8.4 requires Airflow >=2.9.0 and Python >=3.10. This provider allows Airflow to run Pig jobs via the PigOperator. Release cadence follows Airflow's provider release cycle (monthly).

workflow

pip install apache-airflow-providers-apache-pig

github ↗package ↗docs ↗changelog ↗

Install & Compatibility

Where this runs

tested against v4.8.5 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.95 runs

installs and imports cleanly · install 0.0s · import 5.232s · 250.7MB

glibc

py 3.10–3.95 runs

installs and imports cleanly · install 23.3s · import 4.816s · 248MB

250MB installed

● package 250MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

PigOperator

✓ from airflow.providers.apache.pig.operators.pig import PigOperator

Correct import path after provider restructuring (Airflow 2.0+).

Basic DAG using PigOperator. Note: 'pig_cli_conn_id' must be set to a connection with your Pig CLI environment.

from datetime import datetime from airflow import DAG from airflow.providers.apache.pig.operators.pig import PigOperator with DAG(dag_id='example_pig', start_date=datetime(2025,1,1), schedule='@once', catchup=False) as dag: run_pig = PigOperator( task_id='run_pig', pig='ls /user/hadoop;', pig_opts='-x local', pig_cli_conn_id='pig_default' ) run_pig

airflow --version

Debug

Known issues

gotchaThe PigOperator expects the 'pig' argument to be a script string or file path. If using a file, set 'pig' to the file path and ensure the file is accessible on all worker nodes.

fix

For inline Pig Latin, use a multiline string. For a file, set pig='/path/to/script.pig'.

affects: all

deprecatedThe 'pig_conn_id' parameter was renamed to 'pig_cli_conn_id' in provider version 4.0.0 (Airflow 2.3+). Using the old name will fail.

fix

Use 'pig_cli_conn_id' instead of 'pig_conn_id'.
Also ensure the connection type is 'pig'.

affects: >=4.0.0

gotchaThe PigOperator runs the pig command via subprocess. Ensure the 'pig' CLI is installed and available in PATH on all worker nodes where tasks are executed.

fix

Install Apache Pig and verify with 'which pig'. Set the executable path in the connection or via 'pig_cli_conn_id'.

affects: all

Errors

Common errors & fixes

airflow.exceptions.AirflowException: The conn_id `pig_default` isn't defined

Connection 'pig_default' is not set up in Airflow.

fix

Create a connection in Airflow UI (Admin -> Connections) with Conn Id: pig_default, Conn Type: Pig, Host: localhost (or appropriate).

TypeError: __init__() got an unexpected keyword argument 'pig_conn_id'

Using deprecated parameter 'pig_conn_id' in provider 4.0.0+.

fix

Replace 'pig_conn_id' with 'pig_cli_conn_id'.

Upgrade

Version history

4.8.5latest on PyPI

Audit

Dependencies

apache-airflowrequiredRequired: Airflow >=2.9.0 is needed for this provider version.

Agent activity

71 hits · last 30 days

node

ahrefsbot

Amazon

amazonbot

Resources

githubhttps://github.com/apache/airflow ↗

packagehttps://pypi.org/project/apache-airflow-providers-apache-pig/ ↗

docshttps://airflow.apache.org/docs/apache-airflow-providers-apache-pig/4.8.4 ↗

changeloghttps://airflow.apache.org/docs/apache-airflow-providers-apache-pig/4.8.4/changelog.html ↗