BigQuery DataFrames (bigframes) provides a scalable Python DataFrame and machine learning (ML) API powered by the BigQuery engine. It offers a pandas-like interface for analyzing and manipulating data directly within BigQuery, enabling efficient processing of terabytes of data and seamless integration with BigQuery ML and Vertex AI. The library is actively maintained, currently at version 2.39.0, with a rapid release cadence introducing new features and improvements.
pip install bigframesVerified import paths — ran on the pinned version, not inferred.
This quickstart demonstrates how to initialize BigQuery DataFrames with your GCP project ID and load data from a public BigQuery table. It then performs a basic operation (`head()`) to trigger query execution and display results. Ensure you have authenticated to Google Cloud and enabled the BigQuery API for your project.
Set `bpd.options.bigquery.allow_large_results = True` or pass `allow_large_results=True` directly to the method, e.g., `df.to_pandas(allow_large_results=True)`.
Understand that BigQuery DataFrames operations build a query plan. Use methods like `head()`, `to_pandas()`, or printing the object to trigger execution and retrieve results.
Avoid converting large DataFrames to pandas locally. Perform aggregations, filtering, and transformations using BigQuery DataFrames APIs first. Only use `to_pandas()` on small, already reduced datasets, or when absolutely necessary.
Be aware of potential BigQuery storage costs. For long-running or frequently used temporary results, consider managing them explicitly. You can close sessions using `bpd.close_session()` to potentially clean up temporary resources faster, though tables persist for 7 days.
Set `bpd.options.bigquery.location = "YOUR_REGION"` (e.g., "EU", "asia-east1") before calling `read_gbq()` if your data resides outside the 'US' multi-region.
Downgrade the `bigframes` library to a version prior to 1.0 (e.g., `pip install bigframes<1.0.0`) or update the `google-cloud-vertexai` SDK to its latest version, which should have compatibility fixes for the newer `bigframes` API.
Refactor the code to use vectorized `bigframes` DataFrame and Series APIs instead of `apply(axis=1)`. For instance, use direct column operations, `df.assign()`, or built-in string/numeric methods. If `apply(axis=1)` is critical and data types are supported, consider leveraging BigQuery Remote Functions (a preview feature) which enables scalar Python functions to run at BigQuery scale.
Explicitly define an index using `df.set_index('your_column')` or ensure the DataFrame is ordered using `df.sort_values('your_column')` before performing the operation. If partial ordering mode is causing issues, consider if it's necessary or adjust `bpd.options.bigquery.ordering_mode` (though this may have performance implications).Ensure that a consistent session is used across related DataFrame operations. Set the global project ID and location using `bigframes.pandas.options.bigquery.project = 'your-project-id'` and `bigframes.pandas.options.bigquery.location = 'your-location'`. If you need to change these settings, call `bigframes.pandas.close_session()` first to reset the current session. Alternatively, manage sessions explicitly by creating `bigframes.Session` objects with `bigframes.BigQueryOptions(project=..., location=...)`.
Upgrade your `numpy` package to the latest version (`pip install --upgrade numpy`) to ensure compatibility with `bigframes`. If the issue persists, consult the `bigframes` documentation or GitHub issues for known compatible `numpy` versions and potentially try a specific `numpy` version that is known to work with your `bigframes` version.