Registry /
gcp / google-cloud-dataflow-client
The `google-cloud-dataflow-client` is the Python client library for interacting with the Google Cloud Dataflow API. It allows users to programmatically manage Dataflow jobs, such as listing existing jobs, getting job details, or submitting new ones based on templates. This library provides an interface to the Dataflow service API, distinct from the `apache-beam` SDK, which is used for defining Dataflow pipelines themselves. It is part of the broader `google-cloud-python` ecosystem and generally follows its release cadence, receiving regular updates for bug fixes and new features.
Install & Compatibility
Where this runs
tested against v0.14.0 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
JobsClient
✓ from google.cloud.dataflow_v1beta3 import JobsClient
✗ from google.cloud.dataflow_v1beta3 import JobsClient
This quickstart demonstrates how to initialize the Dataflow Jobs client and list existing Dataflow jobs in a specified Google Cloud project and region. Ensure your `GOOGLE_CLOUD_PROJECT` environment variable is set and you have authenticated via `gcloud auth application-default login`.
import os
from google.cloud.dataflow_v1beta3 import JobsClient
from google.cloud.dataflow_v1beta3.types import ListJobsRequest
# Set your Google Cloud Project ID (e.g., via GOOGLE_CLOUD_PROJECT environment variable)
project_id = os.environ.get("GOOGLE_CLOUD_PROJECT", "your-gcp-project-id")
region = "us-central1" # Dataflow jobs are regional, specify the region where your jobs run
if project_id == "your-gcp-project-id":
print("WARNING: Please set the GOOGLE_CLOUD_PROJECT environment variable or replace 'your-gcp-project-id'.")
exit()
try:
# Initialize the client (will use Application Default Credentials by default)
client = JobsClient()
# Create a request to list jobs in a specific project and region
request = ListJobsRequest(project_id=project_id, location=region)
print(f"Listing Dataflow jobs for project '{project_id}' in region '{region}':")
response = client.list_jobs(request=request)
jobs_found = False
for job in response.jobs:
print(f" Job ID: {job.id}, Name: {job.name}, Type: {job.type}, State: {job.current_state}")
jobs_found = True
if not jobs_found:
print(" No Dataflow jobs found.")
except Exception as e:
print(f"An error occurred: {e}")
print("Ensure you have authenticated with `gcloud auth application-default login` and enabled the Dataflow API in your project.")
Debug
Known issues
gotchaThis library (`google-cloud-dataflow-client`) is for *managing* Dataflow jobs and interacting with the Dataflow service. It is not for *defining* data processing pipelines. Pipeline definition is done using the Apache Beam SDK (`apache-beam` library).fixUse `apache-beam` for writing and defining your Dataflow pipelines. Use `google-cloud-dataflow-client` for programmatic control over the Dataflow service itself (e.g., listing, launching, or updating jobs).
affects: All versions
gotchaGoogle Cloud client libraries often expose API versions (e.g., `v1beta3`) directly in their import paths. Using an incorrect or deprecated API version in your import statement (e.g., `from google.cloud.dataflow_v1` instead of `dataflow_v1beta3`) can lead to `ImportError` or unexpected API behavior.fixAlways refer to the official documentation or the library's source code for the exact and recommended import paths for the specific API version you intend to use. For `google-cloud-dataflow-client` version `0.11.0`, the primary API surface is `dataflow_v1beta3`.
affects: All versions
gotchaDataflow operations are regional. When interacting with the Dataflow service, it is crucial to specify the correct `location` (region) for listing or creating jobs. Failing to do so may result in not finding expected jobs or encountering errors.fixEnsure that the `location` parameter is explicitly provided in your client calls (e.g., `client.list_jobs(project_id=project_id, location='us-central1')`) and matches the region where your Dataflow jobs are deployed or expected to run.
affects: All versions
deprecatedThe Dataflow API `v1beta3` is a beta release and, as such, its methods and types are subject to backward-incompatible changes without a long deprecation period. While generally stable, rely on `beta` components with caution in production environments.fixMonitor official Google Cloud Dataflow documentation and release notes for updates to the API and client library. Be prepared to adapt your code if `v1beta3` features are modified or a new, more stable API version becomes available and is recommended.
affects: All versions of `google-cloud-dataflow-client` that expose `v1beta3` clients (e.g., 0.11.0)
Audit
Dependencies
google-api-corerequiredCore library for Google API clients, providing common functionality like transport, authentication, and retry logic.