Registry /
gcp / google-cloud-dataproc-metastore
Install & Compatibility
Where this runs
tested against v1.23.0 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.95 runs
installs and imports cleanly · install 0.0s · import 1.908s · 75.2MB
glibcpy 3.10–3.95 runs
installs and imports cleanly · install 5.8s · import 1.320s · 73MB
74MB installed
● package 74MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
DataprocMetastoreClient
✓ from google.cloud.metastore_v1.services.dataproc_metastore import DataprocMetastoreClient
MetastoreService
✓ from google.cloud.metastore_v1.types import MetastoreService
Used for type hints and creating request/response objects.
ListServicesRequest
✓ from google.cloud.metastore_v1.types import ListServicesRequest
Specific request object for listing services.
This quickstart demonstrates how to initialize the Dataproc Metastore client and list existing Metastore services within a specified Google Cloud project and location. Ensure your environment is authenticated (e.g., via `gcloud auth application-default login`) and the Dataproc Metastore API is enabled for your project.
import os
from google.cloud.metastore_v1.services.dataproc_metastore import DataprocMetastoreClient
from google.cloud.metastore_v1.types import ListServicesRequest
def list_metastore_services(project_id: str, location: str) -> None:
"""Lists Dataproc Metastore services in a given project and location.
Args:
project_id: Your Google Cloud project ID.
location: The Google Cloud location (e.g., 'us-central1').
"""
# Instantiates a client
client = DataprocMetastoreClient()
# The resource name of the location where the services are located.
# Example: "projects/my-project/locations/us-central1"
parent = f"projects/{project_id}/locations/{location}"
# Construct the request
request = ListServicesRequest(parent=parent)
# Call the API
try:
page_result = client.list_services(request=request)
print(f"Dataproc Metastore services in {parent}:")
found_services = False
for service in page_result:
print(f"- {service.name} (State: {service.state.name})")
found_services = True
if not found_services:
print(" No Dataproc Metastore services found.")
except Exception as e:
print(f"Error listing services: {e}")
print("Ensure the API is enabled, credentials are set, and the location is valid.")
# To run this quickstart:
# 1. Ensure `gcloud auth application-default login` has been run or `GOOGLE_APPLICATION_CREDENTIALS` is set.
# 2. Set the `GOOGLE_CLOUD_PROJECT` environment variable to your project ID.
# 3. Set the `GOOGLE_CLOUD_LOCATION` environment variable to your desired location (e.g., "us-central1").
# Example usage:
# GOOGLE_CLOUD_PROJECT='your-project-id' GOOGLE_CLOUD_LOCATION='us-central1' python your_script_name.py
if __name__ == "__main__":
project_id = os.environ.get("GOOGLE_CLOUD_PROJECT", "")
location = os.environ.get("GOOGLE_CLOUD_LOCATION", "")
if not project_id:
print("Please set the GOOGLE_CLOUD_PROJECT environment variable.")
elif not location:
print("Please set the GOOGLE_CLOUD_LOCATION environment variable.")
else:
list_metastore_services(project_id, location)
Debug
Known issues
gotchaDataproc Metastore offers two service versions: Dataproc Metastore 1 and Dataproc Metastore 2. Version 2 provides horizontal scalability and has a different pricing model. When creating or configuring services, ensure you are aware of which version you intend to use as it impacts features and cost.fixReview the Dataproc Metastore documentation on 'Dataproc Metastore versions' and 'features and benefits' to understand the differences and choose the appropriate service version during creation.
affects: All
breakingIncompatible Dataproc or Hive Metastore versions can lead to issues. Specifically, Dataproc 3.x versions are incompatible with Dataproc Metastore. Using Dataproc 1.5 with Dataproc Metastore 3.1.2 may also result in backward compatibility problems.fixAlways check the 'Dataproc Metastore version support' documentation for supported Hive patch versions and Dataproc compatibility. For Dataproc 1.5 with Metastore 3.1.2, consider using the auxiliary versions feature.
affects: All versions when integrating with Dataproc/Hive
gotchaDataproc Metastore services can expose either Apache Thrift or gRPC endpoints. While Thrift is widely used, gRPC is often recommended for integration with newer Google Cloud services like Dataplex. The chosen endpoint protocol must match how clients connect to the service.fixSpecify the `--endpoint-protocol` flag (e.g., `grpc` or `thrift`) when creating your Dataproc Metastore service via `gcloud` or equivalent client library methods. Ensure your connecting applications are configured to use the corresponding protocol.
affects: All
gotchaProper authentication is critical for connecting to Google Cloud services. A common footgun is forgetting to set up Application Default Credentials or providing appropriate IAM roles for the service account/user.fixEnsure you have authenticated via `gcloud auth application-default login` locally, or that the `GOOGLE_APPLICATION_CREDENTIALS` environment variable points to a valid service account key file in production. Grant the necessary IAM roles (e.g., `roles/metastore.editor` or `roles/metastore.admin`) to your principal.
affects: All
gotchaGoogle Cloud client libraries often require the `GOOGLE_CLOUD_PROJECT` environment variable to be set, or the project ID to be explicitly provided in code, to identify which project operations should be performed against. Failure to set it can result in errors when interacting with services like Dataproc Metastore.fixEnsure the `GOOGLE_CLOUD_PROJECT` environment variable is set to your Google Cloud project ID, or explicitly pass the project ID when initializing client libraries or making API calls.
affects: All
gotchaWhen interacting with Google Cloud services, the `GOOGLE_CLOUD_PROJECT` environment variable must be set, or the project ID must be explicitly provided in your application code. This variable specifies which Google Cloud project the client library should operate on.fixSet the `GOOGLE_CLOUD_PROJECT` environment variable to your Google Cloud Project ID (e.g., `export GOOGLE_CLOUD_PROJECT='your-project-id'`) before running your application, or ensure your client library initialization explicitly specifies the project ID.
affects: All
Errors
Common errors & fixes
ModuleNotFoundError: No module named 'google.cloud.dataproc_metastore'
The `google-cloud-dataproc-metastore` library is not installed in your Python environment, or there is an issue with your Python path.
fixInstall the library using pip: `pip install google-cloud-dataproc-metastore`
google.api_core.exceptions.NotFound: 404 Not Found: projects/PROJECT_ID/locations/LOCATION/services/SERVICE_ID not found.
The specified Google Cloud Dataproc Metastore service, or another related resource like a backup or import source, could not be found. This often indicates an incorrect project ID, location, service ID, or that the resource does not exist.
fixVerify that the `project_id`, `location`, and `service_id` in your code or gcloud command accurately match an existing Dataproc Metastore service. Ensure the resource has been created successfully and is in the correct region.
The Dataproc Metastore service agent [SERVICE_AGENT] does not have sufficient IAM permissions to access the network [NETWORK].
The Dataproc Metastore service agent (a Google-managed service account) or the caller's identity lacks the necessary Identity and Access Management (IAM) permissions to perform the requested operation, often related to network access or Cloud Storage buckets.
fixGrant the Dataproc Metastore service agent (e.g., `service-PROJECT_NUMBER@gcp-sa-metastore.iam.gserviceaccount.com`) or the user/service account running the operation the required IAM roles, such as `roles/metastore.serviceAgent` in the project or `roles/datametastore.user` on the Metastore instance, and appropriate Cloud Storage permissions if accessing GCS buckets.
Unable to connect to Hive Metastore (or Connection refused, Host unreachable, Timeout errors)
This error typically occurs when a Dataproc cluster or another client cannot establish a network connection to the Dataproc Metastore service. Common reasons include misconfigured VPC Network Peering, incorrect firewall rules, or an improperly specified Metastore endpoint URI.
fixVerify the VPC Network Peering connection between your Dataproc workload's network and the service producer network is active. Check firewall rules to ensure outbound traffic from your Dataproc workload to Metastore's port (default 9083) is allowed. Confirm the `hive.metastore.uris` Spark property or equivalent configuration uses the correct Dataproc Metastore endpoint URI.
Current state of resource [RESOURCE_NAME] is not a valid state for this operation. Valid state(s) are [RESOURCE_STATE].
You are attempting to perform an operation (e.g., update, import, export, backup, restore) on a Dataproc Metastore service, backup, or import that is not in the required state (e.g., `ACTIVE`).
fixWait for the Dataproc Metastore service or associated resource to reach the `ACTIVE` or appropriate valid state before attempting the operation. You can check the resource's state in the Google Cloud Console or via the gcloud CLI.
Upgrade
Version history
1.23.0latest on PyPI · released Jun 3, 2026
Audit
Dependencies
No dependency data recorded yet.