Registry / gcp / bigquery-schema-generator

bigquery-schema-generator

library1.6.1pypypi✓ verified 22d ago

bigquery-schema-generator is a Python library that generates BigQuery schemas from newline-delimited JSON or CSV data. Unlike BigQuery's native auto-detection which typically samples only the first 500 records, this tool processes all input data to create a more comprehensive and accurate schema. Currently at version 1.6.1, the library maintains an active release cadence, providing regular updates and bug fixes.

gcp data serialization database

pip install bigquery-schema-generator

github ↗package ↗

INSTALL

IMPORT

SIG · BIGQUERY-SCHEMA-GE

bigquery-schema-generator

gcppythonv1.6.1

Install

2.4s avg

Import

—

Disk

17MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v1.6.1 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.95 runs

installs and imports cleanly · install 0.0s · import 0.000s · 19.3MB

glibc

py 3.10–3.95 runs

installs and imports cleanly · install 2.4s · import 0.000s · 20MB

17MB installed

● package 17MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

SchemaGenerator

✓ from bigquery_schema_generator import SchemaGenerator

✗ from bigquery_schema_generator import SchemaGenerator

This example demonstrates how to use `SchemaGenerator` as a library to deduce a BigQuery schema from a list of Python dictionaries. The `deduce_schema_from_dict` method processes the records, and `flatten_schema` converts the internal representation into a BigQuery-compatible JSON schema format.

import json from bigquery_schema_generator.schema_generator import SchemaGenerator # Example data as a list of dictionaries data = [ { "id": "rec1", "name": "Alice", "values": [10, 20] }, { "id": "rec2", "name": "Bob", "values": [30] }, { "id": "rec3", "name": None, # Will be NULLABLE "values": [] # Will be REPEATED (empty array) } ] # Initialize the schema generator generator = SchemaGenerator() # Deduce schema from a list of dictionaries schema_map = generator.deduce_schema_from_dict(data) schema = generator.flatten_schema(schema_map) # Print the generated BigQuery schema in JSON format print(json.dumps(schema, indent=2))

generate-schema --version

Debug

Known issues

gotchaPrior to version 1.6.1, repeated type mismatches for a single field in the input data could cause the schema generator to 'forget' the field's type, leading to multiple warnings and potentially an unstable schema deduction. Ensure you are using version 1.6.1 or newer for robust type inference with inconsistent data.

fix

Upgrade to `bigquery-schema-generator` version 1.6.1 or later (`pip install --upgrade bigquery-schema-generator`).

affects: <1.6.1

gotchaAs of version 1.6.0, `null` fields are now allowed to convert to `REPEATED` (e.g., an empty list `[]`) to align with how `bq load` interprets null values for array-like fields. This changes the previous behavior where `null` fields would typically be omitted or result in `NULLABLE`. Be aware of this change if your schema generation logic relied on a different interpretation for nulls in potentially repeated fields.

fix

Review generated schemas if your data contains nulls in fields that might infer as `REPEATED` types. Use the `--keep_nulls` flag or similar options if specific handling of nulls is required to prevent their conversion to `REPEATED` in some contexts.

affects: >=1.6.0

gotchaWhen using `SchemaGenerator` with existing BigQuery tables, be mindful of BigQuery's strict rules regarding schema evolution. Specifically, you cannot add `REQUIRED` columns to an existing BigQuery table; new columns must be `NULLABLE` or `REPEATED`. While the library helps generate a schema, attempting to apply a schema with newly introduced `REQUIRED` fields to an existing table will result in an error. The `--infer_mode` flag, when used with CSV, can infer `REQUIRED` fields if all values are non-null.

fix

If updating an existing table, ensure new fields in the generated schema are `NULLABLE` or `REPEATED`. If `REQUIRED` fields are critical, consider creating a new table or using BigQuery's `ALTER TABLE` statements cautiously to modify modes where permitted.

affects: All versions

Upgrade

Version history

1.6.1latest on PyPI · released Jan 13, 2024

Audit

Dependencies

No dependency data recorded yet.

Agent activity

53 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/bxparks/bigquery-schema-generator ↗

packagehttps://pypi.org/project/bigquery-schema-generator/ ↗