Registry / data / maggma

maggma

JSON →
library0.72.1pypypi✓ verified 85d ago

Maggma is a framework to build scientific data processing pipelines, handling data from diverse sources like databases, Azure Blobs, and local files, up to REST APIs. It provides core abstractions, `Store` and `Builder`, for modular ETL-like operations. The `Store` interface often mimics PyMongo syntax, enabling consistent data access across different backends. Actively developed by the Materials Project, it is currently at version 0.72.1 and requires Python 3.9+.

pip install maggma
INSTALL
IMPORT
SIG · MAGGMA
M
maggma
datapythonv0.72.1
Install
17.4s avg
Import
2160ms
Disk
259MB
Pass rate
10/ 10
Env Coverage10 / 10
glibc
3.93.13
musl
3.93.13
Install & Compatibility
Where this runs
tested against v0.72.1 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
musl
py 3.103.920 runs
installs and imports cleanly · install 0.0s · import 2.230s · 256MB
glibc
py 3.103.920 runs
installs and imports cleanly · install 17.4s · import 2.089s · 246MB
259MB installed
● package 259MB
Code
Verified usage

Verified import paths — ran on the pinned version, not inferred.

MemoryStore
from maggma.stores import MemoryStore
MongoStore
from maggma.stores import MongoStore
Builder
from maggma.builders import Builder
Store
from maggma.core import Store

This quickstart demonstrates the core concepts of Maggma: defining data as a list of dictionaries, creating a `Store` (using `MemoryStore` for simplicity), connecting to it, adding data using the `update` method, and querying data. It highlights the use of a `key` field for unique document identification. A commented-out example for `MongoStore` is included to illustrate persistent storage.

import os from maggma.stores import MemoryStore # Sample data turtles = [ {"name": "Leonardo", "color": "blue", "tool": "sword"}, {"name": "Donatello", "color": "purple", "tool": "staff"}, {"name": "Michelangelo", "color": "orange", "tool": "nunchuks"}, {"name": "Raphael", "color": "red", "tool": "sai"} ] # Create a MemoryStore (in-memory, data not persistent) # 'key' argument specifies the unique identifier for documents store = MemoryStore(key="name") # Connect to the store (for MemoryStore, this just initializes it) store.connect() # Add data to the store using update # upsert=True means insert if not found, update if found store.update(turtles, key_field='name', upsert=True) # Query the store print(f"Total documents: {store.count()}") print(f"Blue turtle: {store.query(criteria={'color': 'blue'}).current()}") # Find distinct values print(f"Distinct colors: {list(store.distinct(field='color'))}") # Close the store connection (important for persistent stores) store.close() # Example of using a persistent store (e.g., MongoStore) # Requires a MongoDB instance running and pymongo installed. # uri = os.environ.get('MONGO_URI', 'mongodb://localhost:27017/test_db') # from maggma.stores import MongoStore # mongo_store = MongoStore(collection_name='my_collection', database_name='test_db', host=uri, key='name') # try: # mongo_store.connect() # mongo_store.update(turtles, key_field='name', upsert=True) # print(f"MongoStore count: {mongo_store.count()}") # finally: # mongo_store.close()
maggma --version
Debug
Known issues
breakingThe `maggma.api` module has been deprecated and will be migrated. This could significantly impact projects relying on Maggma's built-in API functionalities.
fix
Review the Changelog and documentation for `v0.72.0` for migration details. Projects should update their API implementations to align with the new recommended patterns.
affects: v0.72.0 and later
gotchaMaggma's `Store` classes provide a unified interface that resembles PyMongo. However, not all `Store` implementations (e.g., FileStore, S3Store) support the full breadth of PyMongo's query capabilities or advanced features like aggregation pipelines. Over-reliance on PyMongo-specific syntax with non-Mongo backends can lead to unexpected behavior or unsupported operations.
fix
Consult the specific `Store` class documentation for its supported query features. Stick to basic `query`, `count`, `distinct` operations for maximum compatibility across different `Store` types. For advanced queries, consider processing data after retrieval or using a `MongoStore`.
affects: All versions
gotchaUsing `MemoryStore` is suitable for testing and quick examples, but it is not persistent. Any data added to a `MemoryStore` will be lost when the Python interpreter closes or the `Store` object is garbage collected.
fix
For persistent storage, use a dedicated `Store` implementation like `MongoStore`, `FileStore`, `GridFSStore`, or `S3Store`. Ensure proper connection and disconnection for persistent stores.
affects: All versions
gotchaDocuments added to a `Store` must have a unique identifier, specified by the `key` argument during `Store` initialization (defaulting to `task_id`). If duplicates are inserted with the same key and `upsert=True`, the old document will be overwritten. If `upsert=False`, it may lead to errors depending on the store implementation.
fix
Always ensure your data has a robust, unique identifier for the `key` field. When performing `update` operations, be mindful of the `key_field` and `upsert` parameters to avoid unintended data overwrites or errors.
affects: All versions
breakingMaggma, particularly components like `OpenDataStore`, has reported compatibility issues with `numpy` version 2.0. This can lead to unexpected errors or broken functionality.
fix
Pin your `numpy` version to `<2.0` (e.g., `numpy<2.0`) in your project's dependencies until official `maggma` compatibility with `numpy` 2.0 is confirmed and released.
affects: Reported with `numpy` 2.0 (maggma v0.72.1).
Errors
Common errors & fixes
StoreError: No field 'last_updated' in store document.
A Store is configured with a `last_updated_field` (defaulting to 'last_updated') but the documents being processed do not contain this field, which is essential for incremental building and tracking updates.
fix
Ensure all documents in your source Store have a field named 'last_updated' (or the custom `last_updated_field` you've specified) containing a datetime object, or set `store.last_updated_field = None` if incremental updates based on time are not needed.
AttributeError: 'MongoStore' object has no attribute 'find_one'
The `maggma.Store` interface, while mimicking PyMongo syntax, provides its own methods like `query_one` for querying single documents, rather than directly exposing the `find_one` method from the underlying PyMongo collection.
fix
Replace `find_one` with `query_one` when attempting to retrieve a single document from a `maggma.Store` object.
pymongo.errors.ConfigurationError: Server at localhost:27017 reports wire version X, but this version of PyMongo requires at least Y (MongoDB Z.0).
The version of PyMongo installed (which `maggma` uses for `MongoStore`) is incompatible with the version of the MongoDB server you are trying to connect to.
fix
Upgrade your MongoDB server to a version compatible with your PyMongo client (e.g., MongoDB Z.0 or newer) or downgrade your PyMongo library to a version that supports your MongoDB server.
ModuleNotFoundError: No module named 'maggma.stores.some_non_existent_module'
The user is attempting to import a specific `Store` or `Builder` class from an incorrect or non-existent module path within the `maggma` library.
fix
Consult the `maggma` documentation to find the correct import path for the desired class, e.g., `from maggma.stores import MongoStore` or `from maggma.builders import MapBuilder`.
Upgrade
Version history
0.72.1latest on PyPI · released Feb 11, 2026
Audit
Dependencies
pydanticrequiredData validation and settings management.
pymongorequiredPrimary MongoDB interaction, often used as a backend for 'Store' classes.
montyrequiredUtility functions for materials science, a common dependency in the Materials Project ecosystem.
pandasrequiredData manipulation and analysis, used in some 'Store' and 'Builder' implementations.
numpyrequiredNumerical operations, a fundamental data science library.
boto3optionalAWS SDK for Python, enabling S3 and Azure Blob 'Store' functionality.
sshtunneloptionalSSH tunneling capabilities, often used for secure database connections.
Agent activity
9 hits · last 30 days
node
8
OpenAI (training)
1
Resources
maggma — pip install maggma · libregistry