Registry / workflow / apache-airflow-providers-amazon

apache-airflow-providers-amazon

library9.35.0pypypi✓ verified 25d ago

The `apache-airflow-providers-amazon` package extends Apache Airflow by providing operators, hooks, and sensors for seamless integration with various Amazon Web Services (AWS). It enables orchestrating workflows involving services like Amazon S3, Redshift, EMR, SQS, and more. The provider is actively maintained by the Apache Airflow community with frequent releases, typically every few weeks.

workflow aws devops data

pip install apache-airflow-providers-amazon

github ↗package ↗docs ↗changelog ↗

INSTALL

IMPORT

SIG · APACHE-AIRFLOW-PRO

apache-airflow-providers-amazon

workflowpythonv9.35.0

Install

—

Import

—

Disk

—

Pass rate

0/ 10

Env Coverage0 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v9.35.0 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

glibc

py 3.10

1/2 runs

py 3.11

1/2 runs

py 3.12

1/2 runs

py 3.13

1/2 runs

py 3.9

1/2 runs

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

S3Hook

✓ from airflow.providers.amazon.aws.hooks.s3 import S3Hook

S3CopyObjectOperator

✓ from airflow.providers.amazon.aws.operators.s3 import S3CopyObjectOperator

S3KeySensor

✓ from airflow.providers.amazon.aws.sensors.s3 import S3KeySensor

EcsOperator

✓ from airflow.providers.amazon.aws.operators.ecs import EcsOperator

✗ from airflow.providers.amazon.aws.operators.ecs_operator import EcsOperator

The `ecs_operator` module was refactored, and components moved to `ecs` module in some versions.

This example demonstrates a basic Airflow DAG that uses the `S3CopyObjectOperator` from the Amazon provider to copy an object between two S3 buckets. Ensure you have an Airflow connection named `aws_default` configured with appropriate AWS credentials and permissions, or set AWS environment variables.

from __future__ import annotations import os import pendulum from airflow.models.dag import DAG from airflow.providers.amazon.aws.operators.s3 import S3CopyObjectOperator # Ensure AWS credentials are configured in Airflow Connection 'aws_default' # or via environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION) # For local testing, you can mock this or use dummy values if S3 interaction isn't critical. with DAG( dag_id="example_s3_copy_dag", start_date=pendulum.datetime(2023, 1, 1, tz="UTC"), catchup=False, schedule=None, tags=["s3", "aws", "example"], ) as dag: copy_s3_object = S3CopyObjectOperator( task_id="copy_s3_object_task", source_bucket_key=f"s3://{os.environ.get('SOURCE_BUCKET', 'my-source-bucket')}/source_file.txt", dest_bucket_key=f"s3://{os.environ.get('DEST_BUCKET', 'my-dest-bucket')}/dest_file.txt", aws_conn_id="aws_default", # Assumes 'aws_default' connection is configured in Airflow )

airflow --version

Debug

Known issues

breakingProvider versions 9.x and above (including 9.24.0) require Apache Airflow version 2.11.0 or greater. Attempting to install on older Airflow versions will lead to compatibility issues or automatic Airflow upgrades.

fix

Upgrade your Apache Airflow environment to at least version 2.11.0. Consult the Airflow documentation for upgrade paths.

affects: 9.0.0+

breakingFor provider versions 3.0.0 and above, the `CloudFormationCreateStackOperator` and `CloudFormationDeleteStackOperator` had their `params` argument renamed to `cloudformation_parameters`. This was to avoid a clash with Airflow 2.2+'s DAG `params` field.

fix

Update existing DAGs to use `cloudformation_parameters` instead of `params` for these operators.

affects: 3.0.0+

breakingIn provider version 9.24.0, the deprecated `delegate_to` parameter has been removed from `GCSToS3Operator`, `GlacierToGCSOperator`, and `GoogleApiToS3Operator`.

fix

Use the `impersonation_chain` parameter for impersonation in affected operators.

affects: 9.24.0

deprecatedThe use of `aws_access_key_id` and `aws_secret_access_key` directly in AWS connection `extra` fields is being deprecated in favor of `endpoint_url` for some connections.

fix

Refer to the latest Airflow AWS provider documentation for recommended connection configurations, preferring `endpoint_url` in `extra` where applicable.

affects: 4.1.0+

gotchaWhen running on Amazon MWAA with Airflow v2.7.2+, your `requirements.txt` file should include a `--constraint` statement. If omitted, MWAA will apply a default constraint file, which might not align with desired provider versions.

fix

Always explicitly include a `--constraint` statement in your `requirements.txt` for MWAA environments.

affects: Airflow 2.7.2+

breakingInstallation or execution of this library on Python 3.13 environments (e.g., python:3.13-alpine) may result in timeouts due to potential compatibility issues with the new Python version.

fix

It is recommended to use a Python version officially supported by the library, typically Python 3.8-3.12, until full Python 3.13 compatibility is confirmed.

affects: Python 3.13

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'airflow.providers.amazon.aws.operators.s3'

The module path has changed in recent versions of the 'apache-airflow-providers-amazon' package.

fix

Update the import statement to: 'from airflow.providers.amazon.aws.transfers.s3_to_s3 import S3ToS3Operator'

AttributeError: module 'airflow.providers.amazon.aws.hooks.s3' has no attribute 'S3Hook'

The 'S3Hook' class has been moved to a different module in the 'apache-airflow-providers-amazon' package.

fix

Change the import to: 'from airflow.providers.amazon.aws.hooks.s3 import S3Hook'

ImportError: cannot import name 'EmrCreateJobFlowOperator' from 'airflow.providers.amazon.aws.operators.emr'

The 'EmrCreateJobFlowOperator' has been relocated in the 'apache-airflow-providers-amazon' package.

fix

Use the correct import: 'from airflow.providers.amazon.aws.operators.emr_create_job_flow import EmrCreateJobFlowOperator'

ImportError: cannot import name 'RedshiftSQLOperator' from 'airflow.providers.amazon.aws.operators.redshift'

The 'RedshiftSQLOperator' has been moved to the 'common-sql' provider package.

fix

Install the 'apache-airflow-providers-common-sql' package and import using: 'from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator'

ImportError: cannot import name 'S3ToRedshiftOperator' from 'airflow.providers.amazon.aws.transfers.s3_to_redshift'

The 'S3ToRedshiftOperator' has been moved to the 'common-sql' provider package.

fix

Install the 'apache-airflow-providers-common-sql' package and import using: 'from airflow.providers.common.sql.transfers.s3_to_sql import S3ToSqlOperator'

Upgrade

Version history

9.35.0latest on PyPI · released Aug 23, 2026

Audit

Dependencies

apache-airflowrequiredCore Apache Airflow library; required for provider functionality.

boto3requiredAWS SDK for Python; typically a transitive dependency but essential for AWS interactions.

pandasoptionalRequired by some operators (e.g., SQL to S3) for data manipulation, can be installed via `[pandas]` extra.

Agent activity

42 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/apache/airflow ↗

packagehttps://pypi.org/project/apache-airflow-providers-amazon/ ↗

docshttps://airflow.apache.org/docs/apache-airflow-providers-amazon/9.28.0 ↗

changeloghttps://airflow.apache.org/docs/apache-airflow-providers-amazon/9.28.0/changelog.html ↗