Registry / observability / fluent-logger

fluent-logger

library0.11.1pypypi✓ verified 26d ago

fluent-logger-python is a Python library used to record events from Python applications to Fluentd or Fluent Bit. It provides both an event-based interface (`FluentSender`) and a standard Python `logging.Handler` (`FluentHandler`) for structured logging. The current version is 0.11.1 and it is actively maintained with regular releases.

observability http-networking

pip install fluent-logger

github ↗package ↗

INSTALL

IMPORT

SIG · FLUENT-LOGGER

fluent-logger

observabilitypythonv0.11.1

Install

1.7s avg

Import

20ms

Disk

17MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.11.1 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.95 runs

installs and imports cleanly · install 0.0s · import 0.020s · 18.9MB

glibc

py 3.10–3.95 runs

installs and imports cleanly · install 1.7s · import 0.020s · 20MB

17MB installed

● package 17MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

FluentSender

✓ from fluent import sender logger = sender.FluentSender('app_tag', host='localhost', port=24224)

✗ from fluent_logger import sender

The top-level package for imports is `fluent`, not `fluent_logger` or `fluent-logger`.

FluentHandler

✓ from fluent import handler fluent_handler = handler.FluentHandler('app.tag')

✗ from fluent_logger import handler

The top-level package for imports is `fluent`, not `fluent_logger` or `fluent-logger`.

FluentRecordFormatter

✓ from fluent import handler formatter = handler.FluentRecordFormatter({'host': '%(hostname)s', 'where': '%(module)s.%(funcName)s'})

✗ from fluent_logger import handler

The top-level package for imports is `fluent`, not `fluent_logger` or `fluent-logger`.

This quickstart demonstrates how to configure the standard Python `logging` module to send structured logs to Fluentd using `FluentHandler` and `FluentRecordFormatter`. It includes capturing extra contextual data and exception tracebacks.

import logging from fluent import handler import os # Configure Fluentd to listen on 0.0.0.0:24224 (default) # For a quick test, you can run Fluentd with: # <source> # @type forward # port 24224 # bind 0.0.0.0 # </source> # <match app.**> # @type stdout # </match> # Set up a Python logger logger = logging.getLogger('my_app') logger.setLevel(logging.INFO) # Configure FluentHandler fluent_host = os.environ.get('FLUENTD_HOST', 'localhost') fluent_port = int(os.environ.get('FLUENTD_PORT', 24224)) # The tag 'app.follow' will be used in Fluentd configuration to route logs h = handler.FluentHandler('app.follow', host=fluent_host, port=fluent_port) # Optional: Configure a formatter for structured logs # The dictionary keys are the output field names in Fluentd custom_format = { 'host': '%(hostname)s', 'where': '%(module)s.%(funcName)s', 'type': '%(levelname)s', 'message': '%(message)s', 'stack_trace': '%(exc_text)s' } formatter = handler.FluentRecordFormatter(custom_format) h.setFormatter(formatter) # Add the FluentHandler to the logger logger.addHandler(h) try: logger.info('This is an info message from Python!', extra={'user_id': 123, 'action': 'login'}) raise ValueError("Something went wrong here!") except ValueError as e: logger.error('An error occurred.', exc_info=True, extra={'error_code': 500}) print(f"Logs sent to Fluentd at {fluent_host}:{fluent_port} with tag 'app.follow'") print("Check your Fluentd/Fluent Bit output.")

Debug

Known issues

breakingVersion 0.11.0 dropped official support for Python 3.5 and 3.6. The library now requires Python 3.7 or newer.

fix

Upgrade your Python environment to 3.7+ before upgrading fluent-logger to v0.11.0 or newer.

affects: >=0.11.0

gotchaThe `event.Event()` API does not provide a mechanism to check for success or failure of log delivery, unlike directly using `FluentSender.emit()` which returns a boolean.

fix

For critical logging paths requiring delivery confirmation, consider using `fluent.sender.FluentSender` directly and calling its `emit()` method, which returns `True` on success or `False` on error. You can then use `logger.last_error()` to retrieve error details.

affects: <0.11.1

breakingIn v0.11.0, `FluentSender` introduced a new `forward_packet_error` option. When set to `True` (default behavior in previous versions was to catch), the sender will no longer catch exceptions during event serialization (e.g., if `msgpack` fails), allowing errors to propagate.

fix

If you relied on silent error suppression during serialization, you might need to adjust your error handling logic or explicitly set `forward_packet_error=False` when initializing `FluentSender` or `FluentHandler` (if supported directly). It's generally recommended to handle these exceptions for robustness.

affects: >=0.11.0

gotchaWhen using `FluentRecordFormatter`, how Python's `extra` dictionary for log records is processed can be non-obvious. It might either be merged directly into the top-level record or nested under an 'extra' key, potentially leading to unexpected field names or data structure in Fluentd.

fix

Explicitly define all desired fields in your `FluentRecordFormatter`'s `fmt` dictionary. For dynamic `extra` fields, configure `FluentRecordFormatter` with `exclude_attrs` to control which default `LogRecord` attributes are logged, or handle the merging in a custom formatter or `buffer_overflow_handler` if the default behavior is not suitable. Ensure your Fluentd configuration expects the structure sent by the formatter.

affects: All

gotcha`FluentSender.setup()` creates a global singleton sender instance. While convenient for simple applications, this can be problematic in complex applications requiring multiple distinct Fluentd connections or different root tags, especially in multi-threaded environments or when testing.

fix

For more control, instantiate `FluentSender` objects directly. `FluentSender` now supports `None` as a root tag, allowing multiple root tags within a single connection. This provides greater flexibility and isolation between loggers.

affects: All

Errors

Common errors & fixes

ModuleNotFoundError: No module named 'fluent-logger'

The `fluent-logger` package is not installed in your Python environment or is not accessible within your project's `PYTHONPATH`.

fix

Install the library using pip: `pip install fluent-logger`

Connection refused

The Python application failed to establish a connection with the Fluentd or Fluent Bit server. This usually means the Fluentd/Fluent Bit service is not running, is listening on a different host/port, or a firewall is blocking the connection.

fix

Ensure Fluentd/Fluent Bit is running and configured to listen on the specified host and port (default 24224). Verify network connectivity and firewall rules. Example Fluentd configuration: `<source> @type forward port 24224 </source>`.

UnicodeDecodeError

This error occurs when `fluent-logger` attempts to decode a byte sequence into a string using an incorrect encoding, often when handling logs containing non-ASCII characters or improperly encoded JSON payloads.

fix

Ensure all strings sent to `fluent-logger` are consistently encoded (e.g., UTF-8). If using `FluentRecordFormatter`, verify that the incoming log records' encoding matches the expected decoding in Fluentd/Fluent Bit, and consider using appropriate `encoding` parameters or filtering in Fluentd/Fluent Bit if the issue persists downstream. For direct `emit` calls, ensure dictionary values are properly encoded Python strings.

AttributeError: 'Logger' object has no attribute 'flush'

This error can occur in specific `logging` configurations, particularly when using Python's `logging` module with multiprocessing, and a custom handler (like `FluentHandler`) or a stream wrapper is used without implementing the `flush` method expected by the logging system.

fix

When creating a custom stream-like object for logging, ensure it implements a `flush` method, even if it's a no-op. For standard `FluentHandler` usage, ensure it's initialized correctly as a logging handler without conflicts from other custom stream configurations that might implicitly require `flush`.

Upgrade

Version history

0.11.1latest on PyPI · released Jun 6, 2024

Audit

Dependencies

msgpackrequiredRequired for serializing logs into MessagePack format for Fluentd/Fluent Bit communication.

Agent activity

26 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/fluent/fluent-logger-python ↗

packagehttps://pypi.org/project/fluent-logger/ ↗