Registry / data / pipelinewise-singer-python

pipelinewise-singer-python

library3.0.2pypypi✓ verified 22d ago

This library is a fork of Singer's singer-python, specifically tailored for PipelineWise compatibility. It provides utilities for implementing the Singer.io data replication specification, enabling taps (data extractors) and targets (data loaders) to communicate using a standard JSON-based message format over stdout. The current version is 2.0.1, with releases occurring infrequently, typically driven by critical bug fixes or significant feature enhancements like performance improvements.

data workflow

pip install pipelinewise-singer-python

github ↗package ↗

INSTALL

IMPORT

SIG · PIPELINEWISE-SINGE

pipelinewise-singer-python

datapythonv3.0.2

Install

3.2s avg

Import

443ms

Disk

25MB

Pass rate

4/ 10

Env Coverage4 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v2.0.1 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

glibc

py 3.10

✕ build_error

✓ 3s

py 3.11

✕ build_error

py 3.12

✓ —

✓ 3.2s

py 3.13

✕ build_error

py 3.9

✕ build_error

✓ 3.4s

25MB installed

● package 25MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

singer

✓ import singer

The entire library's functionality is typically accessed via the top-level 'singer' module after import.

This quickstart demonstrates basic usage of the `pipelinewise-singer-python` library to emit Singer.io compliant messages (schema, record, state) to standard output. These messages can then be consumed by a Singer.io target. The example defines a simple schema and writes three records, followed by a state message.

import singer import sys import json # Define a simple schema for demonstration schema = { 'properties': { 'id': {'type': 'integer', 'key': True}, 'name': {'type': 'string'}, 'value': {'type': 'number'} } } # Write the schema message singer.write_schema('my_stream', schema, ['id']) # Write some record messages records = [ {'id': 1, 'name': 'Item A', 'value': 100.5}, {'id': 2, 'name': 'Item B', 'value': 200.0}, {'id': 3, 'name': 'Item C', 'value': 150.75} ] for record in records: singer.write_record('my_stream', record) # Write a state message (optional, but good practice for incremental processing) singer.write_state({'last_processed_id': records[-1]['id']}) print("\n--- Output captured (simulated stdout) ---") # For demonstration, manually capture output to show what 'singer' writes # In a real Singer pipeline, this output goes to stdout.

Debug

Known issues

breakingVersion 2.0.0 replaced the standard `json` library with `orjson` for improved performance. While generally a drop-in replacement, applications with custom JSON handling or those relying on specific `json` library behaviors not supported by `orjson` (e.g., certain `json.dumps` parameters) may experience unexpected issues.

fix

Review custom JSON serialization/deserialization logic in your application. Test thoroughly after upgrading to ensure compatibility with `orjson`. Consult `orjson` documentation for any behavioral differences.

affects: >=2.0.0

gotchaThe library does not provide a default logging configuration. Users are responsible for setting up their own logging using standard Python `logging` module practices. If no logging is configured, log messages from the library might not appear or might go to stderr without proper formatting. An environment variable `LOGGING_CONF_FILE` can be used to point to a logging configuration file.

fix

Implement a logging configuration in your application that utilizes `pipelinewise-singer-python`. Refer to Python's `logging` module documentation or set the `LOGGING_CONF_FILE` environment variable to a valid logging configuration file path.

affects: All versions

gotchaThe `BATCH` message type, introduced in `v1.2.0` and enhanced in `v1.3.0` (with `time_extracted`), allows for more efficient data transfer. However, older taps or targets in a Singer.io pipeline might not fully support this message type, leading to compatibility issues or data processing failures if not all components are updated.

fix

Ensure all taps and targets in your Singer.io pipeline are updated to versions that explicitly support the `BATCH` message type, especially if you intend to leverage its performance benefits. Check component documentation for `BATCH` support.

affects: <1.2.0 (for pipelines not supporting BATCH)

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'singer'

The `pipelinewise-singer-python` library is installed as `pipelinewise-singer-python` via pip, but its core module for import is simply `singer`. This error occurs when a Python script tries to import `pipelinewise_singer_python` directly instead of `singer` or if the virtual environment where it's installed isn't active.

fix

Ensure the virtual environment is activated and use `import singer` in your Python code. If running a PipelineWise component, ensure PipelineWise is correctly installed and configured, as it manages the virtual environments for taps and targets.

AttributeError: 'NoneType' object has no attribute 'get'

This error often indicates that a configuration object (like `properties` or `state` in a Singer tap or target) is `None` when the code expects it to be a dictionary and tries to call the `.get()` method on it. This typically happens when a JSON configuration file (e.g., `config.json` or `state.json`) is missing, empty, or malformed, causing `utils.load_json()` to return `None`.

fix

Verify that your configuration and state JSON files exist, are valid, and contain the expected data structure. Ensure they are correctly passed to the Singer tap or target. For PipelineWise, use `pipelinewise import --dir .` to generate the necessary JSON files from your YAML configurations.

KeyError: 'type'

This `KeyError` typically arises when the `pipelinewise-singer-python` library or a component (like a target) attempts to access the 'type' key within a dictionary representing a schema property or a record, but the key is missing from that dictionary. This often points to an invalid or incomplete Singer schema being processed, especially when dealing with nested structures or when schema properties are empty.

fix

Review the generated Singer `catalog.json` and `schema` messages for the affected stream to ensure that all expected properties, especially 'type', are correctly defined. If using a custom tap, verify that it's emitting valid Singer schema messages. Ensure that all components (taps and targets) are compatible with the Singer specification they are expected to implement.

Log messages from the library might not appear or might go to stderr without proper formatting.

By default, `pipelinewise-singer-python` does not set up a predefined logging configuration. If the calling application or PipelineWise itself doesn't configure Python's `logging` module, then messages from the library will not be formatted or directed to specific outputs, often resulting in them being silently dropped or printed unformatted to standard error.

fix

Implement a logging configuration in your application using Python's `logging` module. Alternatively, set the `LOGGING_CONF_FILE` environment variable to the path of a valid logging configuration file (e.g., a `.conf` file).

Upgrade

Version history

3.0.2latest on PyPI · released Apr 22, 2026

Audit

Dependencies

orjsonrequiredSwitched to orjson in v2.0.0 for faster JSON serialization/deserialization.

backoffrequiredUsed for retry logic, bumped to 1.11.1 in v2.0.0.

pytzrequiredUsed for timezone handling, bumped to latest in v2.0.0.

Agent activity

22 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/transferwise/pipelinewise-singer-python ↗

packagehttps://pypi.org/project/pipelinewise-singer-python/ ↗