pygmars

library1.0.0pypypi✓ verified 82d ago

A library for crafting simple regex-based small language lexers and parsers. Build parsers from grammars and accept Pygments lexers as input. Derived from NLTK. Current version 1.0.0, released May 2024. Maintained by aboutcode-org, release cadence irregular.

data

pip install pygmars

github ↗package ↗

INSTALL

IMPORT

SIG · PYGMARS

pygmars

datapythonv1.0.0

Install

1.5s avg

Import

—

Disk

16MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v1.0.0 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.95 runs

installs and imports cleanly · install 0.0s · import 0.000s · 17.9MB

glibc

py 3.10–3.95 runs

installs and imports cleanly · install 1.5s · import 0.000s · 18MB

16MB installed

● package 16MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

Token

✓ from pygmars import Token

✗ from pygmars import Lexer

as_token_label

✓ from pygmars import as_token_label

convert_pygments_token_to_label

✓ from pygmars import convert_pygments_token_to_label

Create a lexer for arithmetic expressions, define a grammar, and parse a simple expression.

from pygmars import Lexer, Grammar, Parser # Define a simple grammar for arithmetic expressions lexer = Lexer() lexer.add_token('NUM', r'\d+') lexer.add_token('PLUS', r'\+') lexer.add_token('MINUS', r'-') lexer.add_token('TIMES', r'\*') lexer.add_token('DIVIDE', r'/') lexer.add_token('LPAREN', r'\(') lexer.add_token('RPAREN', r'\)') lexer.add_ignore(' ') grammar = Grammar() grammar.add_production('E', ['E', 'PLUS', 'T']) grammar.add_production('E', ['T']) grammar.add_production('T', ['T', 'TIMES', 'F']) grammar.add_production('T', ['F']) grammar.add_production('F', ['LPAREN', 'E', 'RPAREN']) grammar.add_production('F', ['NUM']) parser = Parser(grammar) tokens = lexer.tokenize('2+3*4') parse_tree = parser.parse(tokens) print(parse_tree.pformat())

Debug

Known issues

breakingIn version 1.0.0, the __str__ method of ParseString no longer formats the string, breaking code that relied on it (PR #14).

fix

If you parsed structured data and used str(ParseString(...)) expecting formatting, you now need to call .format() or access attributes directly.

affects: >=1.0.0

deprecatedThe library is derived from NLTK but has diverged. Do not mix imports; use pygmars exclusively for lexing/parsing tasks.

fix

Ensure you import only from pygmars, not from nltk.

affects: all

gotchaLexer token regexes are compiled in order of addition; overlapping patterns may lead to unexpected tokenization. Add more specific tokens first.

fix

Order token additions from most specific to least specific to avoid regex precedence issues.

affects: all

gotchaThe library does not support left-recursive grammars directly. Defining left-recursive productions (e.g., E -> E + T) will cause recursion depth error.

fix

Restructure grammars to be right-recursive or use a different parsing strategy.

affects: all

Upgrade

Version history

1.0.0latest on PyPI · released Jul 16, 2025

Audit

Dependencies

pygmentsoptionalPygmars accepts Pygments lexers as input

Agent activity

12 hits · last 30 days

node

Resources

githubhttps://github.com/aboutcode-org/pygmars ↗

packagehttps://pypi.org/project/pygmars/ ↗