Install & Compatibility
Where this runs
tested against v2021.12.30 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.940 runs
installs and imports cleanly · install 0.0s · import 0.000s · 18.3MB
glibcpy 3.10–3.940 runs
installs and imports cleanly · install 16.3s · import 0.000s · 19MB
16MB installed
● package 16MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
GlueContext
✓ from awsglue.context import GlueContext
✗ from awsglue import GlueContext
This quickstart demonstrates the foundational boilerplate for an AWS Glue ETL script, initializing the SparkContext, GlueContext, and Job objects. While `awsglue-dev` provides the interfaces locally, actual ETL execution often requires a Glue environment (e.g., Docker container or AWS Glue service) to run successfully with real data sources. The `getResolvedOptions` function is used to handle job parameters, which are central to Glue job execution.
import sys
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue.utils import getResolvedOptions
# These parameters are typically passed by AWS Glue service
# For local development, you might set dummy values or omit if not testing getResolvedOptions
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext()
gLueContext = GlueContext(sc)
spark = gLueContext.spark_session
job = Job(gLueContext)
job.init(args['JOB_NAME'], args)
# Your Glue ETL script logic would go here
# For example, to create a DynamicFrame:
# from awsglue.dynamicframe import DynamicFrame
# dynamic_frame = gLueContext.create_dynamic_frame.from_options(
# connection_type='s3',
# connection_options={'paths': ['s3://your-bucket/your-data/'], 'recurse': True},
# format='json'
# )
print(f"Initialized GlueContext and SparkSession for job: {args['JOB_NAME']}")
# Don't forget job.commit() in a real Glue job
# job.commit()
Debug
Known issues
breakingThe `awsglue-dev` package primarily offers Python interfaces for local development (e.g., IDE auto-completion, static analysis). Actual AWS Glue ETL scripts built with these interfaces *must be executed within the AWS Glue service* or a compatible local Docker environment that includes the Glue Spark runtime JARs.fixDo not expect scripts using `awsglue-dev` to run fully functional locally without a full AWS Glue runtime setup (e.g., official AWS Glue Docker images) or deployment to the AWS Glue service. Use it for development and testing logic, not full local execution.
affects: All versions
gotchaFor local development with `awsglue-dev`, the `pyspark` library is a mandatory peer dependency and must be installed separately. Without it, core components like `SparkContext` and `GlueContext` will not function.fixEnsure `pip install pyspark` is run in your development environment alongside `pip install awsglue-dev`.
affects: All versions
gotchaAWS Glue job scripts, by default, may not automatically partition output data when writing to target data sources. This can lead to poor performance on large datasets.fixExplicitly implement partitioning in your ETL job scripts, especially when writing `DynamicFrame` or `DataFrame` outputs, to leverage performance optimizations like partition pruning. Refer to AWS Glue documentation for `repartition` or `write` options.
affects: All versions (especially for Glue-generated scripts)
gotchaAWS Glue environments (which `awsglue-dev` mirrors) often ship with a set of pre-installed Python packages, some of which may contain known vulnerabilities or be outdated. Relying solely on these default versions can pose security risks.fixFor production Glue jobs, explicitly manage dependencies using `requirements.txt` and ensure packages are updated to secure versions. Regularly audit your dependencies using security scanning tools.
affects: All versions of AWS Glue, reflected in local development environments
breakingMigrating AWS Glue jobs between major Glue versions (e.g., from Glue 2.0/3.0 to 4.0/5.0) can introduce breaking changes due to underlying Spark version upgrades, changes in supported Python versions, or deprecation of certain libraries/APIs. Scripts developed with `awsglue-dev` might need adjustments.fixConsult the official AWS Glue migration guides for each specific version upgrade. Test your Glue scripts thoroughly against the target Glue version in a development environment before deploying to production. Pay attention to Spark and Python compatibility.
affects: When transitioning between AWS Glue major service versions (e.g., 2.0, 3.0, 4.0, 5.0)
Upgrade
Version history
2021.12.30latest on PyPI · released Dec 30, 2021
Audit
Dependencies
pysparkrequiredRequired for local development and to successfully import and use GlueContext and SparkContext, as awsglue-dev extends PySpark.