Registry / data / mrjob
library0.7.4pypypiunverified

mrjob is a Python library that allows you to write MapReduce jobs and run them on Hadoop, Amazon EMR, or your local machine. Version 0.7.4 is the latest release; development appears slow with no recent releases.

pip install mrjob
INSTALL
IMPORT
SIG · MRJOB
M
mrjob
datapythonv0.7.4
Install
1.8s avg
Import
1100ms
Disk
22MB
Pass rate
10/ 10
Env Coverage10 / 10
glibc
3.93.13
musl
3.93.13
Install & Compatibility
Where this runs
tested against v0.7.4 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
musl
py 3.103.95 runs
installs and imports cleanly · install 0.0s · import 0.708s · 23.3MB
glibc
py 3.103.95 runs
installs and imports cleanly · install 1.8s · import 0.612s · 24MB
22MB installed
● package 22MB
Code
Verified usage

Verified import paths — ran on the pinned version, not inferred.

MRJob
from mrjob.job import MRJob
from mrjob.job import MRJob

Word count MapReduce job using mrjob. Run locally with: python word_count.py input.txt

from mrjob.job import MRJob import re WORD_RE = re.compile(r"[\w']+") class MRWordFreqCount(MRJob): def mapper(self, _, line): for word in WORD_RE.findall(line): yield (word.lower(), 1) def combiner(self, word, counts): yield (word, sum(counts)) def reducer(self, word, counts): yield (word, sum(counts)) if __name__ == '__main__': MRWordFreqCount.run()
mrjob --version
Debug
Known issues
gotchamrjob 0.7.4 requires `pyyaml` for configuration files (`.mrjob.conf`). Missing it will cause import errors.
fix
Install pyyaml: `pip install pyyaml`
affects: >=0.7.0
deprecatedSupport for Python 2 was dropped after version 0.5.0. Using Python 3 is required for 0.7.x.
fix
Use Python 3.6+.
affects: >=0.6.0
gotchaRunning on Amazon EMR requires `boto3` and proper AWS credentials. Jobs may fail silently if credentials are not configured.
fix
Install boto3: `pip install boto3`. Set environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
affects: all
Upgrade
Version history
0.7.4latest on PyPI · released Sep 17, 2020
Audit
Dependencies

No dependency data recorded yet.

Agent activity
20 hits · last 30 days
node
18
OpenAI (training)
1
Resources
mrjob — pip install mrjob · libregistry