Registry / aws / awsglue3-local

awsglue3-local

library1.0.0pypiunverified

The `awsglue3-local` package is a Python utility for facilitating local development of AWS Glue 3.0 jobs. It aims to simplify the setup of a local PySpark environment that mimics the Glue 3.0 runtime, allowing developers to test Glue scripts outside of the AWS cloud. As of its latest release, it's at version 1.0.0. The release cadence is irregular, typically tied to the need for Glue version compatibility.

aws data

pip install awsglue3-local

INSTALL

IMPORT

SIG · AWSGLUE3-LOCAL

awsglue3-local

awsenv1.0.0

Install

1.8s avg

Import

—

Disk

16MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v1.0.0 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.910 runs

installs and imports cleanly · install 0.0s · import 0.000s · 18.4MB

glibc

py 3.10–3.910 runs

installs and imports cleanly · install 1.8s · import 0.000s · 19MB

16MB installed

● package 16MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

GlueContext

✓ from awsglue.context import GlueContext

✗ from glue.context import GlueContext

Standard import for GlueContext in a Glue environment.

SparkSession

✓ from pyspark.sql import SparkSession

Standard PySpark SparkSession import. awsglue3-local configures the environment.

getResolvedOptions

✓ from awsglue.utils import getResolvedOptions

✗ from glue.utils import getResolvedOptions

Used to parse job arguments from the Glue environment.

DynamicFrame

✓ from awsglue.dynamicframe import DynamicFrame

Core Glue data structure for ETL operations.

A basic AWS Glue job script demonstrating the initialization of `GlueContext`, `SparkSession`, and parsing job arguments using `getResolvedOptions`. This code assumes `awsglue3-local` has correctly set up the environment for these imports to resolve. Note that full `DynamicFrame` functionality often requires additional Glue libraries and configurations, which `awsglue3-local` aims to facilitate but might require more complex setup for specific connectors.

import sys from awsglue.utils import getResolvedOptions from pyspark.context import SparkContext from awsglue.context import GlueContext from awsglue.job import Job # This part mimics how Glue passes arguments # In local development, you might set these via command line or hardcode them # For quickstart, we use an empty dict if not provided. args = getResolvedOptions(sys.argv, ['JOB_NAME']) sc = SparkContext() glucueContext = GlueContext(sc) spark = glucueContext.spark_session job = Job(glucueContext) job.init(args['JOB_NAME'], args) # Example: Create a simple Spark DataFrame data = [("Alice", 1), ("Bob", 2)] df = spark.createDataFrame(data, ["Name", "Id"]) df.show() # Example: Use Glue DynamicFrame (requires more setup for actual data sources) # try: # from awsglue.dynamicframe import DynamicFrame # # This part would typically involve reading from S3, JDBC, etc. # # For a truly local test, you might convert a Spark DataFrame to DynamicFrame # dynamic_frame = DynamicFrame.fromDF(df, glucueContext, "example_df") # dynamic_frame.printSchema() # except ImportError: # print("awsglue.dynamicframe not fully functional in this minimal local setup without full Glue libs.") print("Glue job finished locally.") job.commit()

Debug

Known issues

gotchaThe `awsglue` module is typically part of the AWS Glue runtime and is not fully pip-installable as a complete, standalone library providing all native Glue functionalities. `awsglue3-local` aims to provide the necessary environment and stub modules to allow standard Glue job scripts to run locally, but some features (e.g., direct S3/JDBC connectors without specific Hadoop/Spark configurations) might still require additional setup or behave differently.

fix

Be aware that local execution with `awsglue3-local` provides an approximation of the Glue runtime. For full fidelity, consider using official AWS Glue Docker images or `spark-submit` with `aws-glue-libs` on the classpath.

affects: All versions

breakingLocal Glue development environments, including those set up with `awsglue3-local`, can exhibit behavioral differences compared to the actual AWS Glue cloud environment. These discrepancies can stem from differences in Spark configuration, underlying libraries, resource management, or specific Glue service integrations not fully replicated locally.

fix

Always perform final validation and testing on the actual AWS Glue environment. Ensure local testing focuses on logic and syntax, rather than subtle performance or integration nuances.

affects: All versions

gotchaWhen using `getResolvedOptions`, if job arguments are not provided (e.g., when running a script directly without emulating `spark-submit --conf 'spark.driver.args="--JOB_NAME myjob"'`), it will raise an error indicating required arguments are missing. This is a common pitfall in local development.

fix

For local testing, ensure you either pass dummy arguments (e.g., `sys.argv.extend(['--JOB_NAME', 'my_local_job'])`) or handle missing arguments gracefully in your script, e.g., by providing default values or checking for argument existence.

affects: All versions

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'awsglue.context'

The Python environment does not have the `awsglue` module accessible on its `sys.path` or `awsglue3-local` failed to properly configure the environment.

fix

Ensure `awsglue3-local` is correctly installed. If running with PySpark, confirm that the Glue libraries (often `aws-glue-libs.jar`) are included in your Spark classpath. For `awsglue3-local`, this should ideally be handled, but manual intervention might be needed for complex setups.

java.lang.ClassNotFoundException: org.apache.hadoop.fs.s3a.S3AFileSystem

This typically indicates that the necessary Hadoop AWS S3 connector JARs are missing from your Spark classpath, which are required for interacting with S3 buckets from Spark/Glue.

fix

Ensure your local Spark environment (or the environment configured by `awsglue3-local`) includes the correct Hadoop-AWS JARs. For `pyspark` directly, this often involves `spark-submit --packages org.apache.hadoop:hadoop-aws:x.y.z ...`.

Py4JJavaError: An error occurred while calling o72.getDynamicFrame.fromDF.

This error often occurs when `DynamicFrame` operations are attempted without a fully initialized Glue context or when there are underlying Spark/JVM issues with the Glue extensions. It can also happen if the `awsglue` libraries are not properly linked.

fix

Verify that `GlueContext` is correctly initialized (`GlueContext(sc)`), and `job.init()` is called. Ensure your local environment is robust enough to handle the Glue-specific Spark extensions. Sometimes, restarting the Spark session helps.

Upgrade

Version history

1.0.0latest on PyPI · released Sep 13, 2021

Audit

Dependencies

No dependency data recorded yet.

Agent activity

28 hits · last 30 days

node

OpenAI (training)

Resources

No resource links recorded.